Video analysis method based on visual language model

Abstract

The application provides a video analysis method based on a visual language model, which comprises the following steps: obtaining a video to be analyzed; traversing each video frame in the video to be analyzed; in the case that the current traversed video frame is a key frame, inputting the key frame and the first identification result of each non-key frame corresponding to the key frame into a visual language model to obtain the second identification result of the key frame output by the visual language model; the first identification result is the identification result of the non-key frame obtained by inputting the non-key frame into an image identification model corresponding to the image type after determining the image type of the non-key frame; the image identification model is obtained based on the first sample image corresponding to the image type and the identification result label of the first sample image; and determining the video analysis result of the video to be analyzed based on the second identification result. The application can improve the accuracy of the video analysis result of the visual language model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a video analysis method based on a visual language model. Background Technology

[0002] With the rapid development of deep learning technology, video content understanding and analysis has become an important branch of artificial intelligence. Within this field, Visual Language Models (VLMs) have attracted widespread attention due to their potential in video analysis. By combining visual information and linguistic descriptions, VLMs can achieve a deeper understanding of video content.

[0003] In existing technologies, each video frame in the video to be analyzed is directly input into a visual language model for analysis. However, visual language models cannot directly identify the content in video frames. For example, they cannot accurately identify key objects or behaviors in video frames, resulting in low accuracy of the analysis results obtained in video analysis tasks. Summary of the Invention

[0004] This invention provides a video analysis method based on a visual language model to address the shortcomings of existing technologies that result in low accuracy of video analysis results obtained through visual language models, thereby improving the accuracy of video analysis results obtained through visual language models.

[0005] This invention provides a video analysis method based on a visual language model, comprising:

[0006] Obtain the video to be analyzed;

[0007] The process iterates through each video frame in the video to be analyzed. If the currently iterated video frame is a keyframe, the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe are input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model. The first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into the image recognition model corresponding to the image type after determining the image type of the non-keyframe. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image.

[0008] Based on the second identification result, the video analysis result of the video to be analyzed is determined.

[0009] According to a video analysis method based on a visual language model provided by the present invention, the method further includes:

[0010] If the currently traversed video frame is a non-critical frame, the video analysis result of the video to be analyzed is determined based on the second identification result of the target critical frame that is before and closest to the currently traversed non-critical frame, and the first identification result of other non-critical frames between the target critical frame and the currently traversed non-critical frame.

[0011] According to a video analysis method based on a visual language model provided by the present invention, the method further includes:

[0012] For each video frame in the video to be analyzed, the video frame is divided to obtain at least two image blocks;

[0013] Determine the optical flow value of each image block;

[0014] If the number of target image blocks in the video frame is greater than a preset number, the video frame is determined as a key frame, and the target image block is an image block whose optical flow value is greater than a preset optical flow value.

[0015] All video frames in the video to be analyzed, except for the keyframes, are identified as non-keyframes.

[0016] According to a video analysis method based on a visual language model provided by the present invention, determining the image type of the non-keyframe includes:

[0017] The non-keyframe is input into the image classification model to obtain the image type of the non-keyframe output by the image classification model. The image classification model is trained based on the second sample image and the type label of the second sample image.

[0018] According to a video analysis method based on a visual language model provided by the present invention, the method further includes:

[0019] When the image type of the non-key frame is a security X-ray image, the non-key frame of the security X-ray image type is input into the security X-ray image recognition model to obtain the first target category, first target location and first confidence level included in the non-key frame of the security X-ray image type output by the security X-ray image recognition model;

[0020] The first target category, the first target location, and the first confidence level are determined as the first identification result of the non-key frame of the security inspection X-ray image type.

[0021] According to a video analysis method based on a visual language model provided by the present invention, the method further includes:

[0022] When the image type of the non-key frame is an infrared image, the non-key frame of the infrared image type is input into the infrared image recognition model to obtain the second target category, second target location and second confidence level included in the non-key frame of the infrared image type output by the infrared image recognition model;

[0023] The second target category, the second target location, and the second confidence level are determined as the first identification result of the non-key frame of the infrared image type.

[0024] According to a video analysis method based on a visual language model provided by the present invention, the method further includes:

[0025] When the image type of the non-key frame is a depth image type, the non-key frame of the depth image type is input into the depth image recognition model to obtain the third target category, third target position and target depth information included in the non-key frame of the depth image type output by the depth image recognition model;

[0026] The third target category, the third target location, and the target depth information are determined as the first recognition result of the non-key frame of the depth image type.

[0027] According to a video analysis method based on a visual language model provided by the present invention, the method further includes:

[0028] When the image type of the non-key frame is a visible light image type, the non-key frame of the visible light image type is input into the visible light image recognition model to obtain the fourth target category, fourth target location, third confidence level and number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model;

[0029] The fourth target category, the fourth target location, the third confidence level, and the number of targets are determined as the first recognition result of the non-key frame of the visible light image type.

[0030] According to a video analysis method based on a visual language model provided by the present invention, the method further includes:

[0031] Perform face recognition on the target corresponding to the fourth target location to obtain the target face;

[0032] The target face is matched with faces in the face database;

[0033] If a face matching the target face exists in the face database, the identifier corresponding to the face matching the target face in the face database is determined as the identifier of the target.

[0034] According to a video analysis method based on a visual language model provided by the present invention, the step of inputting the first recognition results of the key frame and each non-key frame corresponding to the key frame into a visual language model to obtain the second recognition result of the key frame output by the visual language model includes:

[0035] Obtain user-inputted recognition requirements;

[0036] Based on the recognition requirement information, the key frame, the image type of the key frame, and the first recognition result of each non-key frame corresponding to the key frame, a prompt message is constructed;

[0037] The prompt information is input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model.

[0038] The present invention also provides a video analysis device based on a visual language model, comprising:

[0039] The acquisition module is used to acquire the video to be analyzed.

[0040] The input module is used to traverse each video frame in the video to be analyzed. When the currently traversed video frame is a keyframe, the first recognition result of the keyframe and each non-keyframe corresponding to the keyframe is input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model. The first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into the image recognition model corresponding to the image type after determining the image type of the non-keyframe. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image.

[0041] The determining module is used to determine the video analysis result of the video to be analyzed based on the second identification result.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video analysis method based on the visual language model as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video analysis method based on the visual language model as described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video analysis method based on the visual language model as described above.

[0045] The video analysis method based on a visual language model provided by this invention acquires the video to be analyzed and traverses each video frame in the video. When the currently traversed video frame is a keyframe, the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe are input into a visual language model to obtain the second recognition result of the keyframe output by the visual language model. The first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into an image recognition model corresponding to the image type after determining the image type of the non-keyframe. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result labels of the first sample image. Therefore, based on the second recognition result, the video analysis result of the video to be analyzed is determined. Since non-keyframes can be first input into the image recognition model for target recognition, and the first recognition result output by the image recognition model is used as the input of the visual language model, the visual language model can use the target recognition result output by the image recognition model as auxiliary information for video analysis, thereby improving the image recognition capability of the visual language model and improving the accuracy of the video analysis results. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is one of the flowcharts illustrating the video analysis method based on a visual language model provided in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the structure of the image classification model provided in an embodiment of the present invention.

[0049] Figure 3 for Figure 2 A schematic diagram of the SE module in the middle.

[0050] Figure 4 This is the second flowchart illustrating the video analysis method based on a visual language model provided in this embodiment of the invention.

[0051] Figure 5 This is a schematic diagram of the structure of a video analysis device based on a visual language model provided in an embodiment of the present invention.

[0052] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] Currently, when performing video analysis using visual language models, each video frame is typically input into the model. By combining visual information and verbal descriptions, a deeper understanding of the video content is achieved. However, visual language models cannot directly identify objects or their behaviors within video frames. Therefore, the accuracy of video analysis results obtained through this method is relatively low. For example, visual language models cannot accurately determine the number of people in a video frame, cannot detect fine targets (such as detecting contraband based on security X-ray images), and cannot directly perform facial recognition.

[0055] To address the aforementioned problems, this invention provides a video analysis method based on a visual language model. In this method, considering that current convolutional neural network-based small models excel at accurate target detection and recognition, especially for low-resolution targets such as smoking detection, head detection, and contraband identification in security X-ray machines, the accuracy of detection is relatively high. Visual language models, on the other hand, excel at inferring events with a certain degree of ambiguity. Therefore, the convolutional neural network-based small model and the visual language model can be combined. First, the convolutional neural network-based small model is used for target detection, and its output is used as input to the visual language model. The visual language model then performs video analysis based on the output of the small model, thereby improving the image recognition capability of the visual language model and increasing the accuracy of the video analysis results.

[0056] The following is combined with Figures 1 to 4 The video analysis method based on a visual language model provided in this invention is described below. The execution subject of this method can be an electronic device such as a security inspection machine, computer, or server, or a specially designed intelligent device. It can also be a video analysis device based on a visual language model installed in the electronic or intelligent device, which can be implemented through software, hardware, or a combination of both. This method can be applied to various scenarios requiring security checks, people counting, or face detection, such as bus stations, train stations, subway stations, or high-speed rail stations.

[0057] Figure 1 This is one of the flowcharts illustrating the video analysis method based on a visual language model provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0058] Step 101: Obtain the video to be analyzed.

[0059] In this step, the video to be analyzed can be a video captured in real time by an image acquisition device, a video taken by a security X-ray machine, or a video obtained from a website or stored locally.

[0060] Step 102: Traverse each video frame in the video to be analyzed. If the currently traversed video frame is a keyframe, input the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe into the visual language model to obtain the second recognition result of the keyframe output by the visual language model. The first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into the image recognition model corresponding to the image type after determining the image type of the non-keyframe. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image.

[0061] In this step, after acquiring the video to be analyzed, it can be decomposed into a sequence of video frames, i.e., an image sequence. It should be understood that content analysis of the video involves analyzing each video frame. Therefore, starting from the first video frame, each video frame in the sequence is traversed. If the currently traversed frame is a non-critical frame, it needs to be classified, i.e., its image type needs to be determined. Image types can include, for example, security X-ray images, infrared images, depth-of-field images, or visible light images. Other image types can also be included, such as thermal infrared images and microwave images. After determining the image type of the currently traversed non-critical frame, the corresponding image recognition model is invoked, and the currently traversed non-critical frame is input into the image recognition model, thus obtaining the first recognition result of the non-critical frame output by the image recognition model.

[0062] For example, if the currently traversed non-key frame is a security X-ray image, then the security X-ray image recognition model is called; if the currently traversed non-key frame is an infrared image, then the infrared image recognition model is called; if the currently traversed non-key frame is a depth image, then the depth image recognition model is called; if the currently traversed non-key frame is a visible light image, then the visible light image recognition model is called.

[0063] The image recognition model is a small model based on a convolutional neural network, trained using a first sample image corresponding to the image type and its recognition result label. Taking a security X-ray image recognition model as an example, multiple security X-ray first sample images are acquired, and each image is labeled with a recognition result label, which may include, for example, prohibited items. These multiple security X-ray first sample images are input into the initial security X-ray image recognition model to obtain the predicted recognition result output by the model. Based on this predicted recognition result and the recognition result label, loss information can be determined. The model parameters of the initial security X-ray image recognition model are adjusted according to this loss information. This process is repeated until the number of iterations meets a preset number or the loss is minimized. The final model obtained is then determined as the security X-ray image recognition model.

[0064] If the currently traversed video frame is a keyframe, the first recognition results of the keyframe and its corresponding non-keyframes are input into the visual language model. The visual language model uses the first recognition results of the non-keyframes as auxiliary information to analyze and understand the content of the keyframe, thereby obtaining the second recognition result of the keyframe output by the visual language model. Here, the non-keyframes corresponding to the keyframe can be understood as all non-keyframes between the keyframe and the previous keyframe.

[0065] Step 103: Based on the second recognition result, determine the video analysis result of the video to be analyzed.

[0066] In this step, the second recognition result of the currently traversed keyframe, output by the visual language model, can be directly determined as the current video analysis result of the video to be analyzed. Specifically, when the keyframe is an X-ray image, the video analysis result can include whether it contains suspicious items and their location; when the keyframe is a visible light image, the video analysis result can include the number of people or unusual situations in the video, such as whether there is fighting.

[0067] The video analysis method based on a visual language model provided in this invention acquires the video to be analyzed and traverses each video frame in the video. When the currently traversed video frame is a keyframe, the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe are input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model. The first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into the image recognition model corresponding to the image type after determining the image type of the non-keyframe. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image. Therefore, based on the second recognition result, the video analysis result of the video to be analyzed is determined. Since the non-keyframes can be first input into the image recognition model for target recognition, and the first recognition result output by the image recognition model is used as the input of the visual language model, the visual language model can use the target recognition result output by the image recognition model as auxiliary information for video analysis, thereby improving the image recognition capability of the visual language model and improving the accuracy of the video analysis result.

[0068] In addition, since it is not necessary to input all video frames of the video to be analyzed into the visual language model, only the key frames need to be input into the visual language model, thereby reducing the computational requirements of the visual language model, improving the speed of image analysis and processing, and reducing the latency of video analysis.

[0069] For example, based on the above embodiments, when the currently traversed video frame is a non-key frame, the video analysis result of the video to be analyzed is determined based on the second identification result of the target key frame that is before and closest to the currently traversed non-key frame, and the first identification result of other non-key frames between the target key frame and the currently traversed non-key frame.

[0070] Specifically, if the currently traversed video frame is a non-key frame, since non-key frames are not input into the visual language model, when determining the video analysis result of the video to be analyzed, it is necessary to fuse the second recognition result of the target key frame that is before and closest to the currently traversed non-key frame, and the first recognition results of other non-key frames between the target key frame and the currently traversed non-key frame. For example, the second recognition result of the target key frame and the sequence of first recognition results of other non-key frames can be used as conditions to calculate the probability of a certain event or item occurring in the currently traversed non-key frame using a Bayesian network, i.e. , of which E n This represents the fusion result of the non-keyframe in the currently traversed nth frame, which is also the video analysis result of the video to be analyzed. This represents the first recognition result of the nth non-key frame in the current iteration. This represents the first identification result of the non-key frame in frame nk. This represents the second recognition result of the target keyframe.

[0071] In this embodiment, when the currently traversed video frame is a non-key frame, the second recognition result of the key frame that is closest to the non-key frame and the first recognition result of other non-key frames can be combined to determine the video analysis result of the video to be analyzed through fusion. This avoids the situation where the first recognition result of the currently traversed non-key frame is directly used as the video analysis result by the image recognition model, so that the accuracy of the final determined video analysis result is higher.

[0072] For example, based on the above embodiments, when determining whether the currently traversed video frame is a key frame or a non-key frame, it can be done in the following way: For each video frame in the video to be analyzed, the video frame is divided into at least two image blocks, the optical flow value of each image block is determined, and if the number of target image blocks in the video frame is greater than a preset number, the video frame is determined as a key frame, the target image block is the image block with an optical flow value greater than the preset optical flow value, and the other video frames in the video to be analyzed other than the key frames are determined as non-key frames.

[0073] Specifically, keyframes are mainly used to determine whether the scene has changed or whether new objects have appeared. Therefore, in this embodiment of the invention, optical flow can be used to classify keyframes and non-keyframes. For the currently traversed video frame, the video frame can be divided into MxN image blocks, for example, each image block can be 11x11 in size, and the pixel center point of the image block is used as the feature point of the optical flow method. The optical flow calculation adopts the Lucas-Kanade (LK) pyramid algorithm. The LK pyramid algorithm uses a pyramid method, that is, the window is fixed, the image blocks are used to generate a pyramid, and the same size window is used for optical flow calculation on each layer of the pyramid. When the image block size is small, the window appears larger, and the optical flow can track faster targets. In the original image, the optical flow window is relatively small, and the obtained optical flow is more accurate. In the above way, MxN optical flows can be obtained, and the optical flow of each image block is used... This is represented as, where i and j represent the indices of the image patch, thus allowing us to determine the order based on the image patch index. The optical flow value of each image block is determined using the following method. By counting the number of target image blocks in the currently traversed video frames whose optical flow value is greater than a preset value, if the number of target image blocks exceeds the preset number, the currently traversed video frame is determined as a keyframe; otherwise, the currently traversed video frame is a non-keyframe. The preset number can be set according to actual conditions or experience; for example, it can be set to 10%.

[0074] In this embodiment, by dividing the video frame and determining the optical flow value of each image block obtained after division, the number of image blocks whose optical flow value exceeds the preset optical flow value determines whether the video frame is a key frame. This method can avoid analyzing the content in the video frame, thereby improving the efficiency of key frame and non-key frame classification.

[0075] For example, based on the above embodiments, when determining the image type of a non-key frame, the non-key frame can be input into the image classification model to obtain the image type of the non-key frame output by the image classification model. The image classification model is trained based on the type labels of the second sample image and the second sample image.

[0076] Specifically, EfficientNetV2 can be used to classify non-key frames. Figure 2 This is a schematic diagram of the structure of the image classification model provided in an embodiment of the present invention. Figure 3 for Figure 2 A schematic diagram of the Squeeze-and-Excitation (SE) module is shown below. Figure 2 As shown, the image classification model can be Fused-MBConv, employing an improved progressive learning approach that adaptively adjusts the regularization factor based on image size, such as dropout and data augmentation. Here, H, W, and C represent the height, width, and number of channels of the feature map, respectively. connv is a convolutional layer used to extract spatial features from the feature map. The structure of SE is as follows... Figure 3 As shown, the global pooling layer can compress the spatial dimension of each channel into a single value, which contains global information of the entire feature map. SiLU is the activation function, and the sigmoid activation function is used to compress the output of the convolutional layer to between 0 and 1.

[0077] By acquiring multiple second sample images and labeling each image with its type (e.g., what type of image it is), a type label is obtained for each second sample image. These second sample images are then input into the initial image classification model, yielding the predicted image type output by the model. Based on the predicted image type and the pre-labeled type, the loss information is determined, and the model parameters of the initial image classification model are adjusted accordingly. This process is repeated iteratively until the loss information is minimized or the preset number of iterations is reached. The final model obtained is then identified as the image classification model.

[0078] By inputting non-keyframes into a trained image classification model, the image type of the non-keyframes output by the model can be obtained. For example, non-keyframes can be classified as security X-ray images, infrared images, depth images, or visible light images. In this way, the image type can be determined quickly and accurately, improving the efficiency and accuracy of image type determination.

[0079] It should be understood that different image recognition models will be invoked when the image type of non-keyframes is different, resulting in different first recognition results for the non-keyframes. The following will analyze different types of non-keyframes and describe in detail the different first recognition results obtained.

[0080] For example, when the image type of the non-key frame is a security X-ray image, the non-key frame of the security X-ray image type is input into the security X-ray image recognition model to obtain the first target category, the first target location and the first confidence level included in the non-key frame of the security X-ray image type output by the security X-ray image recognition model. The first target category, the first target location and the first confidence level are determined as the first recognition result of the non-key frame of the security X-ray image type.

[0081] Specifically, when the image type of the non-keyframe is a security X-ray image, the security X-ray image recognition model trained using security X-ray sample images will be invoked. This security X-ray image recognition model is typically a small model based on a convolutional neural network, capable of detecting trained item types in security X-ray images, such as firearms (including imitations and major components), ammunition, controlled implements, explosives, compressed and liquefied gases and their containers, high-density heavy metals, pharmaceuticals, publications, etc. Inputting the non-keyframe into the security X-ray image recognition model yields the first target category, first target location, and first confidence level included in the non-keyframe output by the security X-ray image recognition model, such as... ,in, Indicates the category of targets included in non-keyframes. Indicates the location of the target. This indicates the confidence level for the first target type.

[0082] Furthermore, the first target category, first target location, and first confidence level output by the security X-ray image recognition model can be used as the first recognition result of the non-key frame of the security X-ray image type. Subsequently, this first recognition result can be used as the input of the visual language model to assist the visual language model in performing in-depth analysis of the content in the key frame image.

[0083] In this embodiment, the X-ray image recognition model can accurately detect contraband in non-critical frames, providing a foundation for subsequent analysis of contraband using a visual language model and greatly improving the accuracy of contraband identification.

[0084] For example, when the image type of the non-key frame is an infrared image, the non-key frame of the infrared image type is input into the infrared image recognition model to obtain the second target category, the second target position and the second confidence level included in the non-key frame of the infrared image type output by the infrared image recognition model. The second target category, the second target position and the second confidence level are determined as the first recognition result of the non-key frame of the infrared image type.

[0085] Specifically, when the image type of the non-keyframe is an infrared image, an infrared image recognition model trained using infrared sample images will be invoked. This infrared image recognition model is typically a small model based on a convolutional neural network, capable of detecting targets such as people, vehicles, smoke, and fumes in the infrared image. Inputting the non-keyframe into the infrared image recognition model yields the second target category, second target location, and second confidence level included in the non-keyframe output by the infrared image recognition model, such as... ,in, Indicates the category of targets included in non-keyframes. Indicates the location of the target. This indicates the confidence level for the second objective type.

[0086] Furthermore, the second target category, second target location, and second confidence level output by the infrared image recognition model can be used as the first recognition result of the non-keyframe of the infrared image type. Subsequently, this first recognition result can be used as the input of the visual language model to assist the visual language model in performing in-depth analysis of the content in the keyframe image.

[0087] In this embodiment, the infrared image recognition model can accurately detect targets in non-key frames, providing a foundation for subsequent target analysis using a visual language model and greatly improving the accuracy of target recognition.

[0088] For example, when the image type of the non-key frame is a depth image type, the non-key frame of the depth image type is input into the depth image recognition model to obtain the third target category, third target position and target depth information included in the non-key frame of the depth image type output by the depth image recognition model. The third target category, third target position and target depth information are determined as the first recognition result of the non-key frame of the depth image type.

[0089] Specifically, when the image type of the non-keyframe is a depth-of-field image, a depth-of-field image recognition model trained using depth-of-field sample images will be invoked. This model is typically a small convolutional neural network-based model capable of detecting targets such as people and vehicles in the depth-of-field image, as well as the depth of these targets. Inputting the non-keyframe into the depth-of-field image recognition model yields the third target category, third target location, and target depth information included in the non-keyframe output by the model, such as... ,in, Indicates the category of targets included in non-keyframes. Indicates the location of the target. This indicates the depth information of the detected target.

[0090] Furthermore, the third target category, third target location, and target depth information output by the depth image recognition model can be used as the first recognition result of the non-keyframe of the depth image type. Subsequently, this first recognition result can be used as the input of the visual language model to assist the visual language model in performing depth analysis of the content in the keyframe image.

[0091] In this embodiment, the target and depth information in non-key frames can be precisely detected by the depth image recognition model, which provides a foundation for subsequent target analysis by the visual language model and can greatly improve the accuracy of target recognition.

[0092] For example, when the image type of the non-key frame is a visible light image, the non-key frame of the visible light image type is input into the visible light image recognition model to obtain the fourth target category, fourth target location, third confidence level and number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model. The fourth target category, fourth target location, third confidence level and number of targets are determined as the first recognition result of the non-key frame of the visible light image type.

[0093] Specifically, when the image type of the non-keyframe is a visible light image, a visible light image recognition model trained using visible light sample images will be invoked. This visible light image recognition model is typically a small model based on a convolutional neural network, capable of detecting targets such as people, vehicles, and objects in the visible light image. Inputting the non-keyframe into the visible light image recognition model yields the fourth target category, fourth target location, and fourth confidence score included in the non-keyframe output by the visible light image recognition model, such as... ,in, Indicates the category of targets included in non-keyframes. Indicates the location of the target. This represents the confidence level for the fourth target type. Furthermore, after detecting the type and location of each target included in the non-keyframes, the number of targets included in the non-keyframes can be further determined. Assuming the target is a pedestrian, the number of people in the current scene can be counted. Human pose recognition algorithms can also be used to determine the key point coordinates of each person, thereby enabling analysis of pedestrian movements based on these key point coordinates.

[0094] Furthermore, the fourth target category, fourth target location, third confidence level, and number of targets output by the visible light image recognition model can be used as the first recognition result of the non-keyframe of the visible light image type. Subsequently, this first recognition result can be used as the input of the visual language model to assist the visual language model in performing in-depth analysis of the content in the keyframe image.

[0095] In this embodiment, the visible light image recognition model can accurately detect targets and their number in non-key frames, providing a foundation for subsequent target analysis using a visual language model. This greatly improves the accuracy of target recognition, and the visual language model can accurately count the number of targets in the current scene.

[0096] For example, based on the above embodiments, when the image type of the non-keyframe is a visible light image type, face recognition can also be performed on the target corresponding to the fourth target position to obtain the target face. The target face is then matched with each face in the face database. If a face in the face database matches the target face, the identifier corresponding to the face in the face database that matches the target face is determined as the identifier of the target.

[0097] Specifically, after detecting the fourth target location in the visible light image using a visible light image recognition model, a face recognition algorithm is used to perform face recognition on the target corresponding to the fourth target location, thereby obtaining the target face. The identified target face is compared with faces in a pre-established face database to determine whether a face matching the target face exists in the database. The face database includes multiple faces and their corresponding identifiers, which can be the target's name, ID number, or other unique identifiers. When a face matching the target face exists in the face database, the identifier of the successfully matched face is used as the identifier of the target detected in the visible light image.

[0098] In this embodiment, after identifying the target face in the visible light image through face recognition, the target face is compared with faces in the face database to determine the target's identifier in the visible light image. Furthermore, after inputting the target's identifier into a visual language model, the visual language model can perform in-depth analysis of keyframe images based on the target's identifier, improving the visual language model's face recognition capabilities.

[0099] For example, based on the above embodiments, when the first recognition results of keyframes and corresponding non-keyframes are input into the visual language model to obtain the second recognition result of the keyframes output by the visual language model, the recognition requirement information input by the user can be obtained. Based on the recognition requirement information, keyframes, image types of keyframes, and the first recognition results of corresponding non-keyframes, prompt information is constructed and input into the visual language model to obtain the second recognition result of the keyframes output by the visual language model.

[0100] Specifically, users can input recognition requirements via voice, text, or other means. For example, for security X-ray images, recognition requirements may include "describe all suspicious items in the X-ray image," and for visible light images, recognition requirements may include "describe any abnormalities in the image."

[0101] After obtaining the user's input recognition requirement information, a prompt message can be constructed based on the recognition requirement information, keyframes, the image type of the keyframes, and the first recognition results of each non-keyframe corresponding to the keyframes. For X-ray images, the constructed prompt message is as follows:

[0102] Question: Describe all suspicious items in the X-ray images.

[0103] Background: The image is an X-ray image, and target 1 (coordinates, confidence), target 2 (coordinates, confidence), ..., target N (coordinates, confidence) were detected in the preceding frames nk to n-1.

[0104] The above content can be represented as:

[0105]

[0106] If the image type is visible light, the prompt message can be as follows:

[0107] Question: Describe anomalies in visible light images;

[0108] Background: The image is a visible light image, and in the preceding frames nk to n-1, there are pedestrian 1 (coordinates, confidence), pedestrian 2 (coordinates, confidence), ..., pedestrian N (coordinates, confidence), number of people number1, number2, ..., numberk, human body keypoint 1, human body keypoint 2, ..., human body keypoint N, etc.

[0109] The above content can be represented as:

[0110]

[0111] Furthermore, the constructed prompt information can be input into the visual language model. The visual language model combines the first recognition results of each non-key frame with the content in the key frame to conduct in-depth analysis, thereby obtaining the second recognition result of the key frame.

[0112] In this embodiment, by combining the first recognition results of non-keyframes to analyze keyframes, the visual language model can gain a more comprehensive understanding of the video content, improving the accuracy of the video analysis. Furthermore, prompts can be constructed based on user-inputted recognition requirements, enabling the visual language model to perform video analysis tailored to different user needs, thus enhancing the flexibility of video analysis.

[0113] Figure 4 This is the second flowchart illustrating the video analysis method based on a visual language model provided in this embodiment of the invention. Figure 4 As shown, after decomposing the video to be analyzed into a sequence of video frames, each video frame in the sequence is traversed, and it is determined whether the currently traversed video frame is a keyframe. If it is not a keyframe, the image type of the non-keyframe is further determined, that is, the non-keyframe is classified to determine whether it belongs to a security X-ray image, infrared image, depth image, or visible light image, and the image recognition model corresponding to the image type is called to identify the non-keyframe. For example, if it is a security X-ray image, the security X-ray image recognition model is called; if it is an infrared image, the infrared image recognition model is called; if it is a depth image, the depth image recognition model is called; if it is a visible light image, the visible light image recognition model is called. Thus, the recognition result of the non-keyframe is output through the image recognition model corresponding to the image type. , where R n-k This represents the recognition result of the nk-th non-keyframe. Furthermore, the recognition results of each non-keyframe can be... The identification result of the target key frame that is the closest to the currently traversed non-key frame and precedes it. By fusing the data using a Bayesian network, the video analysis results of the video to be analyzed are obtained.

[0114] If the nth video frame being traversed is a keyframe, then the identification results of the keyframe and its corresponding non-keyframes will be used. As a conditional input to the visual language model, the recognition results of the keyframes output by the visual language model are obtained. And the recognition results of the keyframes The video analysis results are used as the basis for analyzing the video.

[0115] The video analysis method based on a visual language model provided in this invention acquires the video to be analyzed and traverses each video frame in the video. When the currently traversed video frame is a keyframe, the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe are input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model. The first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into the image recognition model corresponding to the image type after determining the image type of the non-keyframe. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image. Therefore, based on the second recognition result, the video analysis result of the video to be analyzed is determined. Since the non-keyframes can be first input into the image recognition model for target recognition, and the first recognition result output by the image recognition model is used as the input of the visual language model, the visual language model can use the target recognition result output by the image recognition model as auxiliary information for video analysis, thereby improving the image recognition capability of the visual language model and improving the accuracy of the video analysis result.

[0116] In addition, since it is not necessary to input all video frames of the video to be analyzed into the visual language model, only the key frames need to be input into the visual language model, thereby reducing the computational requirements of the visual language model, improving the speed of image analysis and processing, and reducing the latency of video analysis.

[0117] The video analysis device based on a visual language model provided by the present invention is described below. The video analysis device based on a visual language model described below and the video analysis method based on a visual language model described above can be referred to and correspond to each other.

[0118] Figure 5 This is a schematic diagram of the structure of a video analysis device based on a visual language model provided in an embodiment of the present invention, with reference to... Figure 5 As shown, the video analysis device 500 based on a visual language model includes:

[0119] Module 11 is used to acquire the video to be analyzed;

[0120] Input module 12 is used to traverse each video frame in the video to be analyzed. When the currently traversed video frame is a key frame, the first recognition result of the key frame and each non-key frame corresponding to the key frame is input into the visual language model to obtain the second recognition result of the key frame output by the visual language model. The first recognition result is the recognition result of the non-key frame obtained by inputting the non-key frame into the image recognition model corresponding to the image type after determining the image type of the non-key frame. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image.

[0121] The determining module 13 is used to determine the video analysis result of the video to be analyzed based on the second recognition result.

[0122] In one example embodiment, the determining module 13 is further configured to determine the video analysis result of the video to be analyzed based on the second identification result of the target key frame that is before the currently traversed non-key frame and is closest to the currently traversed non-key frame, and the first identification result of other non-key frames between the target key frame and the currently traversed non-key frame, when the currently traversed video frame is a non-key frame.

[0123] In one example embodiment, the apparatus further includes: a partitioning module, wherein:

[0124] The segmentation module is used to divide the video frames in the video to be analyzed into at least two image blocks.

[0125] The determining module 13 is also used to determine the optical flow value of each of the image blocks;

[0126] The determining module 13 is further configured to determine the video frame as a key frame when the number of target image blocks in the video frame is greater than a preset number, wherein the target image block is an image block with an optical flow value greater than a preset optical flow value;

[0127] The determination module 13 is also used to determine other video frames in the video to be analyzed, excluding the key frames, as non-key frames.

[0128] In one example embodiment, the determining module 13 is specifically used for:

[0129] The non-keyframe is input into the image classification model to obtain the image type of the non-keyframe output by the image classification model. The image classification model is trained based on the second sample image and the type label of the second sample image.

[0130] In one example embodiment, the input module 12 is further configured to, when the image type of the non-key frame is a security X-ray image type, input the non-key frame of the security X-ray image type into the security X-ray image recognition model to obtain the first target category, first target location and first confidence level included in the non-key frame of the security X-ray image type output by the security X-ray image recognition model;

[0131] The determining module 13 is further configured to determine the first target category, the first target location, and the first confidence level as the first recognition result of the non-key frame of the security inspection X-ray image type.

[0132] In one example embodiment, the input module 12 is further configured to input the non-key frame of the infrared image type into the infrared image recognition model when the image type of the non-key frame is an infrared image type, so as to obtain the second target category, second target position and second confidence level included in the non-key frame of the infrared image type output by the infrared image recognition model;

[0133] The determining module 13 is further configured to determine the second target category, the second target location, and the second confidence level as the first recognition result of the non-key frame of the infrared image type.

[0134] In one example embodiment, the input module 12 is further configured to input the non-key frame of the non-key frame type into the depth image recognition model when the image type of the non-key frame is a depth image type, so as to obtain the third target category, third target position and target depth information included in the non-key frame of the depth image type output by the depth image recognition model;

[0135] The determining module 13 is further configured to determine the third target category, the third target location, and the target depth information as the first recognition result of the non-key frame of the depth image type.

[0136] In one example embodiment, the input module 12 is further configured to, when the image type of the non-key frame is a visible light image type, input the non-key frame of the visible light image type into the visible light image recognition model to obtain the fourth target category, fourth target location, third confidence level and number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model;

[0137] The determining module 13 is further configured to determine the fourth target category, the fourth target location, the third confidence level, and the number of targets as the first recognition result of the non-key frame of the visible light image type.

[0138] In one example embodiment, the device further includes: an identification module and a matching module, wherein:

[0139] The recognition module is used to perform face recognition on the target corresponding to the fourth target location to obtain the target face;

[0140] The matching module is used to match the target face with faces in the face database;

[0141] The determining module 13 is further configured to, when a face matching the target face exists in the face database, determine the identifier corresponding to the face matching the target face in the face database as the identifier of the target.

[0142] In one example embodiment, the input module 12 is specifically used for:

[0143] Obtain user-inputted recognition requirements;

[0144] Based on the recognition requirement information, the key frame, the image type of the key frame, and the first recognition result of each non-key frame corresponding to the key frame, a prompt message is constructed;

[0145] The prompt information is input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model.

[0146] The video analysis device based on the visual language model in this embodiment can be used to execute the method of any embodiment in the video analysis method based on the visual language model. Its specific implementation process and technical effects are similar to those in the video analysis method based on the visual language model. For details, please refer to the detailed description in the video analysis method based on the visual language model, which will not be repeated here.

[0147] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a video analysis method based on a visual language model. The method includes: acquiring a video to be analyzed; traversing each video frame in the video to be analyzed; if the currently traversed video frame is a keyframe, inputting the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe into a visual language model to obtain a second recognition result of the keyframe output by the visual language model; the first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into an image recognition model corresponding to the image type after determining the image type of the non-keyframe; the image recognition model is trained based on a first sample image corresponding to the image type and the recognition result labels of the first sample image; and determining the video analysis result of the video to be analyzed based on the second recognition result.

[0148] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video analysis method based on the visual language model provided by the above methods. The method includes: acquiring a video to be analyzed; traversing each video frame in the video to be analyzed; if the currently traversed video frame is a keyframe, inputting the first recognition result of the keyframe and each non-keyframe corresponding to the keyframe into a visual language model to obtain a second recognition result of the keyframe output by the visual language model, wherein the first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into an image recognition model corresponding to the image type after determining the image type of the non-keyframe, wherein the image recognition model is trained based on a first sample image corresponding to the image type and the recognition result label of the first sample image; and determining the video analysis result of the video to be analyzed based on the second recognition result.

[0150] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the video analysis method based on the visual language model provided by the above methods. The method includes: acquiring a video to be analyzed; traversing each video frame in the video to be analyzed; if the currently traversed video frame is a keyframe, inputting the first recognition result of the keyframe and each non-keyframe corresponding to the keyframe into a visual language model to obtain a second recognition result of the keyframe output by the visual language model, wherein the first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into an image recognition model corresponding to the image type after determining the image type of the non-keyframe, wherein the image recognition model is trained based on a first sample image corresponding to the image type and the recognition result label of the first sample image; and determining the video analysis result of the video to be analyzed based on the second recognition result.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video analysis method based on a visual language model, characterized in that, include: Obtain the video to be analyzed; The process iterates through each video frame in the video to be analyzed. If the currently iterated video frame is a keyframe, the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe are input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model. The first recognition result is the recognition result of the non-keyframe obtained by inputting the non-keyframe into the image recognition model corresponding to the image type after determining the image type of the non-keyframe. The image recognition model is trained based on the first sample image corresponding to the image type and the recognition result label of the first sample image. Based on the second identification result, the video analysis result of the video to be analyzed is determined; The method further includes: When the currently traversed video frame is a non-key frame, the video analysis result of the video to be analyzed is determined based on the second identification result of the target key frame that is before and closest to the currently traversed non-key frame, and the first identification result of other non-key frames between the target key frame and the currently traversed non-key frame. The step of inputting the first recognition results of the keyframe and each non-keyframe corresponding to the keyframe into the visual language model to obtain the second recognition result of the keyframe output by the visual language model includes: Obtain user-inputted recognition requirements; Based on the recognition requirement information, the key frame, the image type of the key frame, and the first recognition result of each non-key frame corresponding to the key frame, a prompt message is constructed; The prompt information is input into the visual language model to obtain the second recognition result of the keyframe output by the visual language model.

2. The video analysis method based on a visual language model according to claim 1, characterized in that, The method further includes: For each video frame in the video to be analyzed, the video frame is divided to obtain at least two image blocks; Determine the optical flow value of each image block; If the number of target image blocks in the video frame is greater than a preset number, the video frame is determined as a key frame, and the target image block is an image block whose optical flow value is greater than a preset optical flow value. All video frames in the video to be analyzed, except for the keyframes, are identified as non-keyframes.

3. The video analysis method based on a visual language model according to claim 1, characterized in that, Determining the image type of the non-key frame includes: The non-keyframe is input into the image classification model to obtain the image type of the non-keyframe output by the image classification model. The image classification model is trained based on the second sample image and the type label of the second sample image.

4. The video analysis method based on a visual language model according to claim 1, characterized in that, The method further includes: When the image type of the non-key frame is a security X-ray image, the non-key frame of the security X-ray image type is input into the security X-ray image recognition model to obtain the first target category, first target location and first confidence level included in the non-key frame of the security X-ray image type output by the security X-ray image recognition model; The first target category, the first target location, and the first confidence level are determined as the first identification result of the non-key frame of the security inspection X-ray image type.

5. The video analysis method based on a visual language model according to claim 1, characterized in that, The method further includes: When the image type of the non-key frame is an infrared image, the non-key frame of the infrared image type is input into the infrared image recognition model to obtain the second target category, second target location and second confidence level included in the non-key frame of the infrared image type output by the infrared image recognition model; The second target category, the second target location, and the second confidence level are determined as the first identification result of the non-key frame of the infrared image type.

6. The video analysis method based on a visual language model according to claim 1, characterized in that, The method further includes: When the image type of the non-key frame is a depth image type, the non-key frame of the depth image type is input into the depth image recognition model to obtain the third target category, third target position and target depth information included in the non-key frame of the depth image type output by the depth image recognition model; The third target category, the third target location, and the target depth information are determined as the first recognition result of the non-key frame of the depth image type.

7. The video analysis method based on a visual language model according to claim 1, characterized in that, The method further includes: When the image type of the non-key frame is a visible light image type, the non-key frame of the visible light image type is input into the visible light image recognition model to obtain the fourth target category, fourth target location, third confidence level and number of targets included in the non-key frame of the visible light image type output by the visible light image recognition model; The fourth target category, the fourth target location, the third confidence level, and the number of targets are determined as the first recognition result of the non-key frame of the visible light image type.

8. The video analysis method based on a visual language model according to claim 7, characterized in that, The method further includes: Perform face recognition on the target corresponding to the fourth target location to obtain the target face; The target face is matched with faces in the face database; If a face matching the target face exists in the face database, the identifier corresponding to the face matching the target face in the face database is determined as the identifier of the target.

Images