Video analysis method based on visual language model

Abstract

The invention provides a video analysis method based on a visual language model. The method comprises the following steps: acquiring a video to be analyzed; traversing each video frame in the to-be-analyzed video, inputting a key frame and a first recognition result of each non-key frame corresponding to the key frame into a visual language model under the condition that the currently traversed video frame is the key frame to obtain a second recognition result of the key frame output by the visual language model, the first identification result is an identification result of the non-key frame obtained by inputting the non-key frame into an image identification model corresponding to an image type after the image type of the non-key frame is determined; the image recognition model is obtained by training a first sample image corresponding to the image type and a recognition result label of the first sample image; and based on the second identification result, determining a video analysis result of the to-be-analyzed video. The accuracy of the video analysis result of the visual language model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a video analysis method based on a visual language model. Background Art

[0002] With the rapid development of deep learning technology, the understanding and analysis of video content has become an important branch of artificial intelligence. In this field, the Visual Language Model (VLM) has attracted widespread attention due to its potential in video analysis. By combining visual information and language description, the Visual Language Model can provide a deeper understanding of video content.

[0003] In the prior art, each video frame in the video to be analyzed is directly input into the visual language model for analysis. However, the visual language model cannot directly identify the content in the video frame. For example, the visual language model cannot accurately identify the key objects or behaviors in the video frame, which will result in low accuracy of the analysis results obtained in the video analysis task. Summary of the invention

[0004] The present invention provides a video analysis method based on a visual language model, which is used to solve the defect that the accuracy of the video analysis results obtained when performing video analysis through a visual language model in the prior art is not high, and achieves the purpose of improving the accuracy of the video analysis results of the visual language model.

[0005] The present invention provides a video analysis method based on a visual language model, comprising: Get the video to be analyzed; Traversing each video frame in the video to be analyzed, when the currently traversed video frame is a key frame, inputting the first recognition results of the key frame and each non-key frame corresponding to the key frame into a visual language model, and obtaining a second recognition result of the key frame output by the visual language model, wherein the first recognition result is a recognition result of the non-key frame obtained by inputting the non-key frame into an image recognition model corresponding to the image type after determining the image type of the non-key frame, and the image recognition model is trained based on a first sample image corresponding to the image type and a recognition result label of the first sample image; Based on the second recognition result, a video analysis result of the video to be analyzed is determined.

[0006] According to a video analysis method based on a visual language model provided by the present invention, the method further includes: When the currently traversed video frame is a non-key frame, the video analysis result of the video to be analyzed is determined based on the second recognition result of the target key frame that is before the currently traversed non-key frame and closest to the currently traversed non-key frame, and the first recognition result of other non-key frames between the target key frame and the currently traversed non-key frame.

[0007] According to a video analysis method based on a visual language model provided by the present invention, the method further includes: For each video frame in the video to be analyzed, dividing the video frame to obtain at least two image blocks; Determining the optical flow value of each of the image blocks; In a case where the number of target image blocks included in the video frame is greater than a preset number, determining the video frame as a key frame, and the target image block is an image block with an optical flow value greater than a preset optical flow value; The other video frames in the video to be analyzed except the key frame are determined as non-key frames.

[0008] According to a video analysis method based on a visual language model provided by the present invention, the determining the image type of the non-key frame includes: The non-key frame is input into an image classification model to obtain an image type of the non-key frame output by the image classification model, wherein the image classification model is trained based on a second sample image and a type label of the second sample image.

[0009] According to a video analysis method based on a visual language model provided by the present invention, the method further includes: In a case where the image type of the non-key frame is a security inspection X-ray image type, inputting the non-key frame of the security inspection X-ray image type into a security inspection X-ray image recognition model to obtain a first target category, a first target position, and a first confidence level included in the non-key frame of the security inspection X-ray image type output by the security inspection X-ray image recognition model; The first target category, the first target position and the first confidence level are determined as a first recognition result of a non-key frame of the security inspection X-ray image type.

[0010] According to a video analysis method based on a visual language model provided by the present invention, the method further includes: In the case where the image type of the non-key frame is an infrared image type, inputting the non-key frame of the infrared image type into an infrared image recognition model to obtain a second target category, a second target position, and a second confidence level included in the non-key frame of the infrared image type output by the infrared image recognition model; The second target category, the second target position and the second confidence level are determined as a first recognition result of the non-key frame of the infrared image type.

[0011] According to a video analysis method based on a visual language model provided by the present invention, the method further includes: In a case where the image type of the non-key frame is a depth image type, inputting the non-key frame of the depth image type into a depth image recognition model to obtain a third target category, a third target position, and target depth information included in the non-key frame of the depth image type output by the depth image recognition model; The third target category, the third target position and the target depth information are determined as a first recognition result of the non-key frame of the depth of field image type.

[0012] According to a video analysis method based on a visual language model provided by the present invention, the method further includes: In a case where the image type of the non-key frame is a visible light image type, inputting the non-key frame of the visible light image type into a visible light image recognition model to obtain a fourth target category, a fourth target position, a third confidence level, and the number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model; The fourth target category, the fourth target position, the third confidence level, and the number of targets are determined as a first recognition result of the non-key frame of the visible light image type.

[0013] According to a video analysis method based on a visual language model provided by the present invention, the method further includes: Performing face recognition on a target corresponding to the fourth target position to obtain a target face; Matching the target face with each face in the face library; In the case that a face matching the target face exists in the face database, an identifier corresponding to the face matching the target face in the face database is determined as the identifier of the target.

[0014] According to a video analysis method based on a visual language model provided by the present invention, the first recognition result of the key frame and each non-key frame corresponding to the key frame is input into the visual language model to obtain a second recognition result of the key frame output by the visual language model, including: Obtaining identification requirement information input by the user; Constructing prompt information based on the recognition requirement information, the key frame, the image type of the key frame, and the first recognition results of each non-key frame corresponding to the key frame; The prompt information is input into the visual language model to obtain a second recognition result of the key frame output by the visual language model.

[0015] The present invention also provides a video analysis device based on a visual language model, comprising: An acquisition module is used to acquire the video to be analyzed; An input module, used for traversing each video frame in the video to be analyzed, and when the currently traversed video frame is a key frame, inputting a first recognition result of the key frame and each non-key frame corresponding to the key frame into a visual language model to obtain a second recognition result of the key frame output by the visual language model, wherein the first recognition result is a recognition result of the non-key frame obtained by inputting the non-key frame into an image recognition model corresponding to the image type after determining the image type of the non-key frame, and the image recognition model is trained based on a first sample image corresponding to the image type and a recognition result label of the first sample image; A determination module is used to determine a video analysis result of the video to be analyzed based on the second recognition result.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video analysis method based on the visual language model as described above is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video analysis method based on the visual language model as described in any one of the above is implemented.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video analysis method based on the visual language model as described above is implemented.

[0019] The video analysis method based on the visual language model provided by the present invention obtains the video to be analyzed and traverses each video frame in the video to be analyzed. When the currently traversed video frame is a key frame, the first recognition result of the key frame and each non-key frame corresponding to the key frame is input into the visual language model to obtain the second recognition result of the key frame output by the visual language model. The first recognition result is the recognition result of the non-key frame obtained by inputting the non-key frame into the image recognition model corresponding to the image type after determining the image type of the non-key frame. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image, so as to determine the video analysis result of the video to be analyzed based on the second recognition result. Since the non-key frame can be first input into the image recognition model for target recognition, and the first recognition result output by the image recognition model is used as the input of the visual language model, the visual language model can perform video analysis based on the target recognition result output by the image recognition model as auxiliary information, thereby improving the image recognition ability of the visual language model and improving the accuracy of the video analysis result. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 One of the flowcharts of the video analysis method based on the visual language model provided in an embodiment of the present invention.

[0022] Figure 2 A schematic diagram of the structure of an image classification model provided by an embodiment of the present invention.

[0023] Figure 3 for Figure 2 Schematic diagram of the structure of the SE module.

[0024] Figure 4 The second flowchart of the video analysis method based on the visual language model provided in the embodiment of the present invention.

[0025] Figure 5 A schematic diagram of the structure of a video analysis device based on a visual language model provided in an embodiment of the present invention.

[0026] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] At present, when performing video analysis through a visual language model, each video frame in the video to be analyzed is usually input into the visual language model, and a deeper understanding of the video content is obtained by combining visual information and language description. However, the visual language model cannot directly identify objects or behaviors in video frames. Therefore, the accuracy of the video analysis results obtained in the above manner is low. For example, the visual language model cannot accurately determine the number of people contained in the video frame, and cannot detect fine targets, such as being unable to detect contraband based on security X-ray images, and cannot directly perform face recognition.

[0029] In view of the above problems, an embodiment of the present invention provides a video analysis method based on a visual language model. In this method, it is considered that the small model based on the convolutional neural network is good at accurate target detection and recognition, especially for targets with smaller resolution, such as smoking detection, head detection, and identification of contraband in security X-ray machines, etc., and the detection accuracy is high, while the visual language model is good at inferring events with certain ambiguity. Therefore, the small model based on the convolutional neural network and the visual language model can be combined. The small model based on the convolutional neural network is first used for target detection, and the output result of the small model is used as the input of the visual language model. The visual language model performs video analysis based on the output result of the small model, thereby improving the image recognition ability of the visual language model and improving the accuracy of the video analysis results.

[0030] Combine the following Figures 1 to 4 The video analysis method based on the visual language model provided by the embodiment of the present invention is described. The execution subject of the method can be an electronic device such as a security inspection machine, a computer or a server, or a specially designed intelligent device, or a video analysis device based on the visual language model set in the electronic device or the intelligent device. The video analysis device based on the visual language model can be implemented by software, hardware or a combination of both. The method can be applied to various scenes such as bus stations, railway stations, subway stations or high-speed railway stations that require security inspections, headcounts or face detection.

[0031] Figure 1 One of the flow charts of the video analysis method based on the visual language model provided by the embodiment of the present invention is as follows: Figure 1 As shown, the method includes: Step 101: Obtain the video to be analyzed.

[0032] In this step, the video to be analyzed may be a video captured in real time by an image acquisition device, a video shot by a security X-ray machine, or a video obtained from a website or stored locally.

[0033] Step 102: traverse each video frame in the video to be analyzed. When the currently traversed video frame is a key frame, input the first recognition results of the key frame and each non-key frame corresponding to the key frame into the visual language model to obtain a second recognition result of the key frame output by the visual language model. The first recognition result is the recognition result of the non-key frame obtained by inputting the non-key frame into the image recognition model corresponding to the image type after determining the image type of the non-key frame. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image.

[0034] In this step, after the video to be analyzed is acquired, the video to be analyzed can be decomposed into a sequence of video frames, that is, an image sequence. It should be understood that when performing content analysis on the video to be analyzed, each video frame in the video to be analyzed is analyzed. Therefore, it is necessary to start from the first video frame and traverse each video frame in the video frame sequence. If the currently traversed video frame is a non-key frame, it is necessary to classify the non-key frame, that is, determine the image type of the non-key frame, wherein the image type may include, for example, security X-ray images, infrared images, depth of field images, or visible light images. Of course, other image types may also be included, such as thermal infrared images, microwave images, etc. After determining the image type of the currently traversed non-key frame, the image recognition model corresponding to the image type will be called, and the currently traversed non-key frame will be input into the image recognition model corresponding to the image type, thereby obtaining the first recognition result of the non-key frame output by the image recognition model.

[0035] For example, if the currently traversed non-key frame is a security X-ray image, the security X-ray image recognition model is called; if the currently traversed non-key frame is an infrared image, the infrared image recognition model is called; if the currently traversed non-key frame is a depth of field image, the depth of field image recognition model is called; if the currently traversed non-key frame is a visible light image, the visible light image recognition model is called.

[0036] Among them, the image recognition model is a small model based on a convolutional neural network, which is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image. Taking the security inspection X-ray image recognition model as an example, by obtaining multiple security inspection X-ray first sample images, and marking the recognition result label in each security inspection X-ray first sample image, the recognition result label may include contraband, for example. Multiple security inspection X-ray first sample images are input into the initial security inspection X-ray image recognition model to obtain the predicted recognition result output by the initial security inspection X-ray image recognition model. Based on the predicted recognition result and the recognition result label, the loss information can be determined. According to the loss information, the model parameters of the initial security inspection X-ray image recognition model are adjusted. By repeating the above process until the number of iterations meets the preset number or the loss is minimized, the final model is determined as the security inspection X-ray image recognition model.

[0037] If the currently traversed video frame is a key frame, the first recognition results of the key frame and the non-key frames corresponding to the key frame are input into the visual language model, and the visual language model uses the first recognition results of the non-key frames as auxiliary information to analyze and understand the content in the key frame, thereby obtaining the second recognition result of the key frame output by the visual language model. Among them, the non-key frames corresponding to the key frame can be understood as all non-key frames between the key frame and the previous key frame.

[0038] Step 103: Based on the second recognition result, determine the video analysis result of the video to be analyzed.

[0039] In this step, the second recognition result of the currently traversed key frame output by the visual language model can be directly determined as the current video analysis result of the video to be analyzed. When the key frame is an X-ray image, the video analysis result can include whether a suspicious object is contained and the location of the suspicious object. When the key frame is a visible light image, the video analysis result can include the number of people contained in the video or abnormal situations, such as whether there is a fight.

[0040] The video analysis method based on the visual language model provided by the embodiment of the present invention obtains the video to be analyzed and traverses each video frame in the video to be analyzed. When the currently traversed video frame is a key frame, the first recognition result of the key frame and each non-key frame corresponding to the key frame is input into the visual language model to obtain the second recognition result of the key frame output by the visual language model. The first recognition result is the recognition result of the non-key frame obtained by inputting the non-key frame into the image recognition model corresponding to the image type after determining the image type of the non-key frame. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image, so as to determine the video analysis result of the video to be analyzed based on the second recognition result. Since the non-key frame can be first input into the image recognition model for target recognition, and the first recognition result output by the image recognition model is used as the input of the visual language model, the visual language model can perform video analysis based on the target recognition result output by the image recognition model as auxiliary information, thereby improving the image recognition ability of the visual language model and improving the accuracy of the video analysis result.

[0041] In addition, since it is not necessary to input all video frames in the video to be analyzed into the visual language model, only the key frames need to be input into the visual language model, which can reduce the computing power requirements of the visual language model, improve the speed of image analysis and processing, and reduce the delay of video analysis.

[0042] Exemplarily, based on the above embodiment, when the currently traversed video frame is a non-key frame, the video analysis result of the video to be analyzed is determined based on the second recognition result of the target key frame that is before the currently traversed non-key frame and closest to the currently traversed non-key frame, and the first recognition result of other non-key frames between the target key frame and the currently traversed non-key frame.

[0043] Specifically, if the currently traversed video frame is a non-key frame, since the non-key frame is not input into the visual language model, when determining the video analysis result of the video to be analyzed, it is necessary to fuse the second recognition result of the target key frame that is before the currently traversed non-key frame and closest to the currently traversed non-key frame, and the first recognition results of other non-key frames between the target key frame and the currently traversed non-key frame. For example, the second recognition result of the target key frame and the first recognition result sequence of other non-key frames can be used as conditions to calculate the probability of a certain event occurring or a certain item appearing in the currently traversed non-key frame through a Bayesian network, that is, , where E n Indicates the fusion result of the nth non-key frame currently traversed, that is, the video analysis result of the video to be analyzed. Indicates the first recognition result of the nth non-key frame currently traversed. Indicates the first recognition result of the nkth non-key frame. Indicates the second recognition result of the target keyframe.

[0044] In this embodiment, when the currently traversed video frame is a non-key frame, the second recognition result of the key frame before and closest to the non-key frame and the first recognition results of other non-key frames can be combined to determine the video analysis result of the video to be analyzed through a fusion method, thereby avoiding the situation where the first recognition result of the currently traversed non-key frame recognized by the image recognition model is directly used as the video analysis result, so that the final determined video analysis result is more accurate.

[0045] Exemplarily, based on the above embodiments, when determining whether the currently traversed video frame is a key frame or a non-key frame, it can be done in the following way: for each video frame in the video to be analyzed, the video frame is divided to obtain at least two image blocks, and the optical flow value of each image block is determined. When the number of target image blocks included in the video frame is greater than a preset number, the video frame is determined as a key frame, the target image block is an image block whose optical flow value is greater than a preset optical flow value, and other video frames in the video to be analyzed except the key frames are determined as non-key frames.

[0046] Specifically, key frames are mainly used to determine whether the scene has changed, whether new objects have appeared, etc. Therefore, in an embodiment of the present invention, the optical flow method can be used to classify key frames and non-key frames. For the currently traversed video frame, the video frame can be divided into MxN image blocks. For example, each image block can be 11x11 in size, and the pixel center point of the image block is used as the feature point of the optical flow method. The optical flow calculation adopts the Lucas-Kanade (LK) pyramid algorithm. The LK pyramid algorithm adopts a pyramid method, that is, the window is fixed, and the image block is generated into a pyramid. The same size window is used for optical flow calculation on each layer of the pyramid. When the image block size is small, the window appears larger. At this time, the optical flow can track faster targets. In the original image, the optical flow window is relatively small, and the obtained optical flow is more accurate. In the above manner, MxN optical flows can be obtained, and the optical flow of each image block is used. Indicated by, where i and j represent the index of the image block, so that The optical flow value of each image block is determined in this way. By counting the number of target image blocks whose optical flow values ​​in the currently traversed video frame are greater than the preset optical flow value, when the number of target image blocks exceeds the preset number, the currently traversed video frame is determined as a key frame, otherwise, the currently traversed video frame is a non-key frame. The preset number can be set according to actual conditions or experience, for example, it can be set to 10%.

[0047] In this embodiment, by dividing the video frame and determining the optical flow value of each image block obtained after the division, whether the video frame is a key frame is determined based on the number of image blocks whose optical flow values ​​exceed the preset optical flow value. This method can avoid analyzing the content in the video frame, thereby improving the efficiency of classifying key frames and non-key frames.

[0048] Exemplarily, based on the above embodiments, when determining the image type of a non-key frame, the non-key frame can be input into an image classification model to obtain the image type of the non-key frame output by the image classification model, and the image classification model is trained based on the second sample image and the type label of the second sample image.

[0049] Specifically, EfficientNetV2 can be used to classify non-key frames. Figure 2 A schematic diagram of the structure of an image classification model provided by an embodiment of the present invention, Figure 3 for Figure 2 The structural diagram of the Squeeze-and-Excitation (SE) module in the figure is as follows: Figure 2 As shown in the figure, the image classification model can be Fused-MBConv, and adopts an improved progressive learning method, which can adaptively adjust the regularization factor according to the image size, such as dropout, data augmentation, etc. Among them, H, W and C represent the height, width and number of channels of the feature map respectively. Connv is a convolutional layer used to extract spatial features in the feature map. Among them, the structure of SE is as follows Figure 3 As shown in the figure, the global pooling layer can compress the spatial dimension of each channel into a single value, which contains the global information of the entire feature map. SiLU is the activation function, and the sigmoid activation function is used to compress the output of the convolutional layer to between 0 and 1.

[0050] By acquiring multiple second sample images and marking the image type in each second sample image, such as what type of image it belongs to, the type label of each second sample image is obtained. The second sample image is input into the initial image classification model to obtain the predicted image type output by the initial image classification model. According to the predicted image type and the pre-marked type label, the loss information can be determined, and the model parameters of the initial image classification model can be adjusted based on the loss information. The above steps are iteratively performed until the loss information is minimized or the number of iterations reaches a preset number, and the final model is determined as the image classification model.

[0051] By inputting non-key frames into a trained image classification model, the image type of the non-key frames output by the image classification model can be obtained. For example, non-key frames can be classified into security X-ray images, infrared images, depth of field images or visible light images, etc. In this way, the image type can be determined quickly and accurately, thereby improving the efficiency and accuracy of image type determination.

[0052] It should be understood that when the image types of non-key frames are different, different image recognition models will be called, thereby obtaining different first recognition results of non-key frames. Below, different types of non-key frames are analyzed to obtain different first recognition results, which will be described in detail.

[0053] Exemplarily, when the image type of the non-key frame is a security inspection X-ray image type, the non-key frame of the security inspection X-ray image type is input into a security inspection X-ray image recognition model, and the first target category, first target position and first confidence included in the non-key frame of the security inspection X-ray image type output by the security inspection X-ray image recognition model are obtained, and the first target category, first target position and first confidence are determined as the first recognition result of the non-key frame of the security inspection X-ray image type.

[0054] Specifically, when the image type of the non-key frame is a security inspection X-ray image type, the security inspection X-ray image recognition model trained using the security inspection X-ray sample image will be called. The security inspection X-ray image recognition model is usually a small model based on a convolutional neural network, which can detect the types of trained items in the security inspection X-ray image, such as firearms (including imitations, main components), ammunition, controlled devices, explosives, compressed and liquefied gases and their containers, heavy metals with high density, medicines, publications, etc. The non-key frame is input into the security inspection X-ray image recognition model, and the first target category, first target position and first confidence included in the non-key frame output by the security inspection X-ray image recognition model can be obtained, such as ,in, Indicates the category of the target included in the non-key frame, Indicates the location of the target. Indicates the confidence of the first target type.

[0055] Furthermore, the first target category, the first target position and the first confidence output by the security X-ray image recognition model can be used as the first recognition result of the non-key frame of the security X-ray image type, and the first recognition result can be subsequently used as the input of the visual language model, thereby assisting the visual language model to perform in-depth analysis of the content in the key frame image.

[0056] In this embodiment, the security inspection X-ray image recognition model can be used to accurately detect contraband in non-key frames, which provides a basis for subsequent contraband analysis through a visual language model and can greatly improve the accuracy of contraband identification.

[0057] Exemplarily, when the image type of the non-key frame is an infrared image type, the non-key frame of the infrared image type is input into the infrared image recognition model, and the second target category, second target position and second confidence included in the non-key frame of the infrared image type output by the infrared image recognition model are obtained, and the second target category, second target position and second confidence are determined as the first recognition result of the non-key frame of the infrared image type.

[0058] Specifically, when the image type of the non-key frame is an infrared image type, the infrared image recognition model trained with the infrared sample image will be called. The infrared image recognition model is usually a small model based on a convolutional neural network, which can detect targets such as people, vehicles, fireworks, and smoke in the infrared image. The non-key frame is input into the infrared image recognition model, and the second target category, second target position, and second confidence included in the non-key frame output by the infrared image recognition model can be obtained, such as ,in, Indicates the category of the target included in the non-key frame, Indicates the location of the target. Indicates the confidence of the second target type.

[0059] Furthermore, the second target category, the second target position and the second confidence output by the infrared image recognition model can be used as the first recognition result of the non-key frame of the infrared image type, and the first recognition result can be subsequently used as the input of the visual language model, thereby assisting the visual language model to perform in-depth analysis of the content in the key frame image.

[0060] In this embodiment, the infrared image recognition model can be used to accurately detect targets in non-key frames, which provides a basis for subsequent target analysis through a visual language model and can greatly improve the accuracy of target recognition.

[0061] Exemplarily, when the image type of a non-key frame is a depth of field image type, the non-key frame of the depth of field image type is input into a depth of field image recognition model to obtain a third target category, a third target position and target depth information included in the non-key frame of the depth of field image type output by the depth of field image recognition model, and the third target category, the third target position and the target depth information are determined as the first recognition result of the non-key frame of the depth of field image type.

[0062] Specifically, when the image type of the non-key frame is a depth image type, a depth image recognition model trained with a depth sample image will be called. The depth image recognition model is usually a small model based on a convolutional neural network, which can detect objects such as people and vehicles in the depth image and the depth of the object. The non-key frame is input into the depth image recognition model, and the third target category, third target position and target depth information included in the non-key frame output by the depth image recognition model can be obtained, such as ,in, Indicates the category of the target included in the non-key frame, Indicates the location of the target. Indicates the depth information of the detected target.

[0063] Furthermore, the third target category, third target position and target depth information output by the depth of field image recognition model can be used as the first recognition result of a non-key frame of the depth of field image type, and the first recognition result can subsequently be used as the input of the visual language model, thereby assisting the visual language model in performing in-depth analysis of the content in the key frame image.

[0064] In this embodiment, the depth of field image recognition model can be used to accurately detect targets and depth information in non-key frames, which provides a basis for subsequent target analysis through a visual language model and can greatly improve the accuracy of target recognition.

[0065] Exemplarily, when the image type of the non-key frame is a visible light image type, the non-key frame of the visible light image type is input into a visible light image recognition model to obtain a fourth target category, a fourth target position, a third confidence level and the number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model, and the fourth target category, the fourth target position, the third confidence level and the number of targets are determined as the first recognition result of the non-key frame of the visible light image type.

[0066] Specifically, when the image type of the non-key frame is a visible light image type, a visible light image recognition model trained using visible light sample images will be called. The visible light image recognition model is usually a small model based on a convolutional neural network, which can detect targets such as people, vehicles, and objects in visible light images. The non-key frame is input into the visible light image recognition model, and the fourth target category, fourth target position, and fourth confidence included in the non-key frame output by the visible light image recognition model can be obtained, such as ,in, Indicates the category of the target included in the non-key frame, Indicates the location of the target. Indicates the confidence of the fourth target type. In addition, after detecting the type and position of each target included in the non-key frame, the number of targets included in the non-key frame can be further determined. Assuming that the target is a pedestrian, the number of people in the current scene can be counted. The key point coordinates of each human body can also be determined through a human posture recognition algorithm, so as to analyze the pedestrian's movements based on these key point coordinates.

[0067] Furthermore, the fourth target category, fourth target position, third confidence level and the number of targets counted output by the visible light image recognition model can be used as a first recognition result of a non-key frame of the visible light image type. Subsequently, the first recognition result can be used as an input to a visual language model, thereby assisting the visual language model in performing in-depth analysis of the content in the key frame image.

[0068] In this embodiment, the visible light image recognition model can be used to accurately detect targets and the number of targets in non-key frames, which provides a basis for subsequent target analysis through the visual language model, can greatly improve the accuracy of target recognition, and can accurately count the number of targets in the current scene through the visual language model.

[0069] Exemplarily, based on the above embodiment, when the image type of the non-key frame is a visible light image type, face recognition can also be performed on the target corresponding to the fourth target position to obtain the target face, and the target face is matched with each face in the face library. When there is a face matching the target face in the face library, the identifier corresponding to the face matching the target face in the face library is determined as the identifier of the target.

[0070] Specifically, after the fourth target position where the target in the visible light image is located is detected by the visible light image recognition model, the target corresponding to the fourth target position is recognized by the face recognition algorithm to obtain the target face. The recognized target face is compared with each face in the pre-established face library to determine whether there is a face matching the target face in the face library. The face library includes multiple faces and an identifier corresponding to each face, and the identifier can be the name of the target, the ID number, or other content that can uniquely represent the target. When there is a face matching the target face in the face library, the identifier corresponding to the successfully matched face in the face library is determined as the identifier of the target detected in the visible light image.

[0071] In this embodiment, after the target face in the visible light image is determined by face recognition, the target face is compared with each face in the face library, so that the identity of the target in the visible light image can be determined. Further, after the identity of the target is input into the visual language model, the visual language model can perform in-depth analysis of the key frame image based on the identity of the target, thereby improving the ability of the visual language model to recognize faces.

[0072] Exemplarily, based on the above embodiments, when the first recognition results of the key frames and the non-key frames corresponding to the key frames are input into the visual language model to obtain the second recognition results of the key frames output by the visual language model, the recognition requirement information input by the user can be obtained, and prompt information can be constructed based on the recognition requirement information, the key frames, the image type of the key frames, and the first recognition results of the non-key frames corresponding to the key frames, and the prompt information is input into the visual language model to obtain the second recognition results of the key frames output by the visual language model.

[0073] Specifically, the user can input recognition requirement information through voice, text or other means. For example, for security X-ray images, the recognition requirement information may include "describe all suspicious items in the X-ray image"; for visible light images, the recognition requirement information may include "describe abnormal situations in the image", etc.

[0074] After obtaining the recognition requirement information input by the user, a prompt information may be constructed based on the recognition requirement information, the key frame, the image type of the key frame, and the first recognition result of each non-key frame corresponding to the key frame. The prompt information is prompt. For the X-ray image type, the constructed prompt information is as follows: Problem: Describe all suspicious objects in an X-ray image. Background: The image is an X-ray image, and target 1 (coordinates, confidence), target 2 (coordinates, confidence), ..., target N (coordinates, confidence) are detected in the previous nk to n-1 frames.

[0075] The above can be expressed as: If it is a visible light image type, the constructed prompt information can be as follows: Problem: Describe anomalies in visible light images; Background: The image is a visible light image, and in the first nk to n-1 frames there are pedestrian 1 (coordinates, confidence), pedestrian 2 (coordinates, confidence), ..., pedestrian N (coordinates, confidence), number of people number1, number2, ..., numberk, human key point 1, human key point 2, ..., human key point N, etc.

[0076] The above can be expressed as: Furthermore, the constructed prompt information can be input into a visual language model, and the visual language model combines the first recognition results of each non-key frame to perform an in-depth analysis on the content in the key frame, thereby obtaining a second recognition result of the key frame.

[0077] In this embodiment, by combining the first recognition result of the non-key frame to analyze the key frame, the visual language model can understand the video content more comprehensively, thereby improving the analysis accuracy of the video to be analyzed. In addition, prompt information can be constructed based on the recognition requirement information input by the user, so that the visual language model can perform video analysis according to different user requirements, thereby improving the flexibility of video analysis.

[0078] Figure 4 The second flow chart of the video analysis method based on the visual language model provided by the embodiment of the present invention is as follows: Figure 4 As shown, after the video to be analyzed is decomposed into a video frame sequence, each video frame in the video frame sequence will be traversed, and it will be determined whether the currently traversed video frame is a key frame. If it is not a key frame, the image type of the non-key frame will be further determined, that is, the non-key frame will be classified to determine whether the non-key frame belongs to a security inspection X-ray image, an infrared image, a depth of field image, or a visible light image, and the image recognition model corresponding to the image type will be called to identify the non-key frame. For example, if it is a security inspection X-ray image, the security inspection X-ray image recognition model will be called, if it is an infrared image, the infrared image recognition model will be called, if it is a depth of field image, the depth of field image recognition model will be called, and if it is a visible light image, the visible light image recognition model will be called, so that the recognition result of the non-key frame is output through the image recognition model corresponding to the image type. , where R n-k represents the recognition result of the nkth non-key frame. The recognition result of the target key frame that is before the currently traversed non-key frame and closest to the currently traversed non-key frame The video analysis results of the video to be analyzed are obtained by fusion through the Bayesian network.

[0079] If the nth video frame currently being traversed is a key frame, the recognition results of the key frame and the corresponding non-key frames are As the conditional input of the visual language model, the recognition result of the key frame output by the visual language model is obtained , and the recognition results of the key frames As the video analysis result of the video to be analyzed.

[0080] The video analysis method based on the visual language model provided by the embodiment of the present invention obtains the video to be analyzed and traverses each video frame in the video to be analyzed. When the currently traversed video frame is a key frame, the first recognition result of the key frame and each non-key frame corresponding to the key frame is input into the visual language model to obtain the second recognition result of the key frame output by the visual language model. The first recognition result is the recognition result of the non-key frame obtained by inputting the non-key frame into the image recognition model corresponding to the image type after determining the image type of the non-key frame. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image, so as to determine the video analysis result of the video to be analyzed based on the second recognition result. Since the non-key frame can be first input into the image recognition model for target recognition, and the first recognition result output by the image recognition model is used as the input of the visual language model, the visual language model can perform video analysis based on the target recognition result output by the image recognition model as auxiliary information, thereby improving the image recognition ability of the visual language model and improving the accuracy of the video analysis result.

[0081] In addition, since it is not necessary to input all video frames in the video to be analyzed into the visual language model, only the key frames need to be input into the visual language model, which can reduce the computing power requirements of the visual language model, improve the speed of image analysis and processing, and reduce the delay of video analysis.

[0082] The video analysis device based on the visual language model provided by the present invention is described below. The video analysis device based on the visual language model described below and the video analysis method based on the visual language model described above can be referenced to each other.

[0083] Figure 5 A schematic diagram of the structure of a video analysis device based on a visual language model provided by an embodiment of the present invention, referring to Figure 5 As shown, the video analysis device 500 based on the visual language model includes: An acquisition module 11 is used to acquire the video to be analyzed; An input module 12 is used to traverse each video frame in the video to be analyzed, and when the currently traversed video frame is a key frame, input the first recognition result of the key frame and each non-key frame corresponding to the key frame into a visual language model to obtain a second recognition result of the key frame output by the visual language model, wherein the first recognition result is a recognition result of the non-key frame obtained by inputting the non-key frame into an image recognition model corresponding to the image type after determining the image type of the non-key frame, and the image recognition model is trained based on a first sample image corresponding to the image type and a recognition result label of the first sample image; The determination module 13 is used to determine the video analysis result of the video to be analyzed based on the second recognition result.

[0084] In an exemplary embodiment, the determination module 13 is also used to determine the video analysis result of the video to be analyzed based on the second recognition result of the target key frame that is before the currently traversed non-key frame and closest to the currently traversed non-key frame, and the first recognition result of other non-key frames between the target key frame and the currently traversed non-key frame, when the currently traversed video frame is a non-key frame.

[0085] In an exemplary embodiment, the apparatus further comprises: a partitioning module, wherein: A division module, used for dividing each video frame in the video to be analyzed into at least two image blocks; The determination module 13 is further used to determine the optical flow value of each image block; The determination module 13 is further configured to determine the video frame as a key frame when the number of target image blocks included in the video frame is greater than a preset number, and the target image block is an image block with an optical flow value greater than a preset optical flow value; The determination module 13 is further configured to determine other video frames in the video to be analyzed except the key frame as non-key frames.

[0086] In an exemplary embodiment, the determination module 13 is specifically configured to: The non-key frame is input into an image classification model to obtain an image type of the non-key frame output by the image classification model, wherein the image classification model is trained based on a second sample image and a type label of the second sample image.

[0087] In an exemplary embodiment, the input module 12 is further configured to, when the image type of the non-key frame is a security inspection X-ray image type, input the non-key frame of the security inspection X-ray image type into a security inspection X-ray image recognition model to obtain a first target category, a first target position, and a first confidence level included in the non-key frame of the security inspection X-ray image type output by the security inspection X-ray image recognition model; The determination module 13 is further configured to determine the first target category, the first target position and the first confidence level as a first recognition result of the non-key frame of the security inspection X-ray image type.

[0088] In an exemplary embodiment, the input module 12 is further used to input the non-key frame of the infrared image type into the infrared image recognition model when the image type of the non-key frame is an infrared image type, and obtain the second target category, the second target position and the second confidence included in the non-key frame of the infrared image type output by the infrared image recognition model; The determination module 13 is further configured to determine the second target category, the second target position and the second confidence level as a first recognition result of the non-key frame of the infrared image type.

[0089] In an exemplary embodiment, the input module 12 is further configured to, when the image type of the non-key frame is a depth image type, input the non-key frame of the depth image type into a depth image recognition model to obtain a third target category, a third target position, and target depth information included in the non-key frame of the depth image type output by the depth image recognition model; The determination module 13 is further configured to determine the third target category, the third target position and the target depth information as a first recognition result of the non-key frame of the depth of field image type.

[0090] In an exemplary embodiment, the input module 12 is further configured to, when the image type of the non-key frame is a visible light image type, input the non-key frame of the visible light image type into a visible light image recognition model to obtain a fourth target category, a fourth target position, a third confidence level, and the number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model; The determination module 13 is further configured to determine the fourth target category, the fourth target position, the third confidence level, and the number of targets as a first recognition result of the non-key frame of the visible light image type.

[0091] In an exemplary embodiment, the apparatus further comprises: an identification module and a matching module, wherein: A recognition module, used for performing face recognition on a target corresponding to the fourth target position to obtain a target face; A matching module, used to match the target face with each face in the face library; The determination module 13 is further configured to determine, when there is a face matching the target face in the face database, an identifier corresponding to the face matching the target face in the face database as an identifier of the target.

[0092] In an exemplary embodiment, the input module 12 is specifically configured to: Obtaining identification requirement information input by the user; Constructing prompt information based on the recognition requirement information, the key frame, the image type of the key frame, and the first recognition results of each non-key frame corresponding to the key frame; The prompt information is input into the visual language model to obtain a second recognition result of the key frame output by the visual language model.

[0093] The video analysis device based on the visual language model of this embodiment can be used to execute the method of any embodiment in the video analysis method based on the visual language model side embodiment. Its specific implementation process and technical effects are similar to those in the video analysis method based on the visual language model side embodiment. For details, please refer to the detailed introduction in the video analysis method based on the visual language model side embodiment, which will not be repeated here.

[0094] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the video analysis method based on the visual language model, the method comprising: obtaining a video to be analyzed; traversing each video frame in the video to be analyzed, and when the currently traversed video frame is a key frame, inputting the first recognition result of the key frame and each non-key frame corresponding to the key frame into the visual language model to obtain a second recognition result of the key frame output by the visual language model, wherein the first recognition result is the recognition result of the non-key frame obtained by inputting the non-key frame into the image recognition model corresponding to the image type after determining the image type of the non-key frame, and the image recognition model is obtained by training based on the first sample image corresponding to the image type and the recognition result label of the first sample image; and determining the video analysis result of the video to be analyzed based on the second recognition result.

[0095] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0096] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video analysis method based on the visual language model provided by the above-mentioned methods, and the method includes: obtaining a video to be analyzed; traversing each video frame in the video to be analyzed, and when the currently traversed video frame is a key frame, inputting a first recognition result of the key frame and each non-key frame corresponding to the key frame into a visual language model to obtain a second recognition result of the key frame output by the visual language model, wherein the first recognition result is a recognition result of the non-key frame obtained by inputting the non-key frame into an image recognition model corresponding to the image type after determining the image type of the non-key frame, and the image recognition model is trained based on a first sample image corresponding to the image type and a recognition result label of the first sample image; based on the second recognition result, determining the video analysis result of the video to be analyzed.

[0097] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the video analysis method based on a visual language model provided by the above-mentioned methods, the method comprising: obtaining a video to be analyzed; traversing each video frame in the video to be analyzed, and when the currently traversed video frame is a key frame, inputting a first recognition result of the key frame and each non-key frame corresponding to the key frame into a visual language model to obtain a second recognition result of the key frame output by the visual language model, wherein the first recognition result is a recognition result of the non-key frame obtained by inputting the non-key frame into an image recognition model corresponding to the image type after determining the image type of the non-key frame, and the image recognition model is trained based on a first sample image corresponding to the image type and a recognition result label of the first sample image; based on the second recognition result, determining a video analysis result of the video to be analyzed.

[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video analysis method based on a visual language model, characterized in that: include: Get the video to be analyzed; Traversing each video frame in the video to be analyzed, when the currently traversed video frame is a key frame, inputting the first recognition results of the key frame and each non-key frame corresponding to the key frame into a visual language model, and obtaining a second recognition result of the key frame output by the visual language model, wherein the first recognition result is a recognition result of the non-key frame obtained by inputting the non-key frame into an image recognition model corresponding to the image type after determining the image type of the non-key frame, and the image recognition model is trained based on a first sample image corresponding to the image type and a recognition result label of the first sample image; Based on the second recognition result, a video analysis result of the video to be analyzed is determined.

2. The video analysis method based on the visual language model according to claim 1 is characterized in that: The method further comprises: When the currently traversed video frame is a non-key frame, the video analysis result of the video to be analyzed is determined based on the second recognition result of the target key frame that is before the currently traversed non-key frame and closest to the currently traversed non-key frame, and the first recognition result of other non-key frames between the target key frame and the currently traversed non-key frame.

3. The video analysis method based on the visual language model according to claim 1 is characterized in that: The method further comprises: For each video frame in the video to be analyzed, dividing the video frame to obtain at least two image blocks; Determining the optical flow value of each of the image blocks; In a case where the number of target image blocks included in the video frame is greater than a preset number, determining the video frame as a key frame, and the target image block is an image block with an optical flow value greater than a preset optical flow value; The other video frames in the video to be analyzed except the key frame are determined as non-key frames.

4. The video analysis method based on the visual language model according to claim 1 is characterized in that: The determining the image type of the non-key frame comprises: The non-key frame is input into an image classification model to obtain an image type of the non-key frame output by the image classification model, wherein the image classification model is trained based on a second sample image and a type label of the second sample image.

5. The video analysis method based on the visual language model according to claim 1 is characterized in that: The method further comprises: In a case where the image type of the non-key frame is a security inspection X-ray image type, inputting the non-key frame of the security inspection X-ray image type into a security inspection X-ray image recognition model to obtain a first target category, a first target position, and a first confidence level included in the non-key frame of the security inspection X-ray image type output by the security inspection X-ray image recognition model; The first target category, the first target position and the first confidence level are determined as a first recognition result of a non-key frame of the security inspection X-ray image type.

6. The video analysis method based on the visual language model according to claim 1, characterized in that: The method further comprises: In the case where the image type of the non-key frame is an infrared image type, inputting the non-key frame of the infrared image type into an infrared image recognition model to obtain a second target category, a second target position, and a second confidence level included in the non-key frame of the infrared image type output by the infrared image recognition model; The second target category, the second target position and the second confidence level are determined as a first recognition result of the non-key frame of the infrared image type.

7. The video analysis method based on the visual language model according to claim 1 is characterized in that: The method further comprises: In a case where the image type of the non-key frame is a depth image type, inputting the non-key frame of the depth image type into a depth image recognition model to obtain a third target category, a third target position, and target depth information included in the non-key frame of the depth image type output by the depth image recognition model; The third target category, the third target position and the target depth information are determined as a first recognition result of the non-key frame of the depth of field image type.

8. The video analysis method based on the visual language model according to claim 1, characterized in that: The method further comprises: In a case where the image type of the non-key frame is a visible light image type, inputting the non-key frame of the visible light image type into a visible light image recognition model to obtain a fourth target category, a fourth target position, a third confidence level, and the number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model; The fourth target category, the fourth target position, the third confidence level, and the number of targets are determined as a first recognition result of the non-key frame of the visible light image type.

9. The video analysis method based on the visual language model according to claim 8, characterized in that: The method further comprises: Performing face recognition on a target corresponding to the fourth target position to obtain a target face; Matching the target face with each face in the face library; In the case that a face matching the target face exists in the face database, an identifier corresponding to the face matching the target face in the face database is determined as the identifier of the target.

10. The video analysis method based on the visual language model according to any one of claims 1 to 9, characterized in that: The step of inputting the first recognition result of the key frame and each non-key frame corresponding to the key frame into a visual language model to obtain a second recognition result of the key frame output by the visual language model comprises: Obtaining identification requirement information input by the user; Constructing prompt information based on the recognition requirement information, the key frame, the image type of the key frame, and the first recognition results of each non-key frame corresponding to the key frame; The prompt information is input into the visual language model to obtain a second recognition result of the key frame output by the visual language model.

Images