Video object searching method based on multi-modal model

Abstract

The invention discloses a method for searching a video object based on a multi-modal model, which is characterized in that a whole set of personnel searching process from video data preprocessing, feature extraction to multi-modal fusion and matching is constructed based on a self-developed multi-modal model, and a target object can be quickly retrieved from massive pictures (videos) according to text input; according to the method, comprehensive analysis and understanding of video contents are realized by using an advanced multi-mode technology, automatic personnel identification and positioning capabilities are provided by combining natural language description and video rapid positioning, a target person can be rapidly and accurately found in massive videos, the search time is remarkably shortened, manual intervention is reduced, and the search efficiency is improved. And the efficiency and the accuracy are greatly improved.

Description

Technical Field

[0001] The invention relates to computer science technology and is applied to video object search. Background Art

[0002] In the field of video personnel search, traditional methods usually rely on manual review and frame-by-frame analysis to find relevant personnel, which is not only time-consuming and costly, but also prone to errors and difficult to meet the rapidly changing needs in a dynamic environment. With the advancement of technology and the explosive growth of video, this traditional method is stretched.

[0003] In the ever-changing monitoring environment and various application scenarios, the power of "artificial intelligence AI" can be used to optimize the workflow in this field. This optimization is not only the analysis and understanding of video content, but also the precise positioning and rapid retrieval of personnel information, strengthening the video processing process with intelligent capabilities, and realizing security and efficiency as the center, and continuously improving the efficiency and correctness of monitoring management through data and technology.

[0004] Other video personnel search methods can only filter based on pre-configured tags (for example: male, female), and cannot perform searches based on natural language. In order to truly solve the challenges in video personnel search, the present invention combines advanced multimodal models and successfully develops a set of video search methods based on language descriptions. This method can quickly identify, track and locate target personnel from massive video and image data, capture key features in complex scenes, and provide fast and accurate decision support for security monitoring and emergency response. The application of this technology not only significantly improves search efficiency, but also brings innovative and groundbreaking working methods to the security industry.

[0005] At the overall solution level, there is currently no similar process-based intelligent video personnel search solution in the industry. This invention is the first institution to apply multimodal model capabilities to video personnel search and realize commercialization, achieving efficient retrieval from text descriptions to video search results; Summary of the invention

[0006] Aiming at the shortcomings of existing detection methods, a method for video object search based on multimodal model is proposed, which includes the following steps: Step S1, pull the video stream data produced by the camera, detect and extract key object information in the video stream; Step S2, storing the timestamp, location coordinates, and image (video) information corresponding to the key object extracted in step S1, and displaying the stored key object information by retrieval in the subsequent step S6; Step S3: Use the multimodal model Qwen2VL to encode the key objects and generate a vector Vector object; The multimodal model refers to a machine learning model that can process and understand multiple types of data at the same time; the modalities may include text, images, audio, and video; the multimodal model Qwen2VL is a large multimodal model developed by Alibaba, which provides the ability to understand and generate text and visual interactions; Step S4: The vector Vector generated in step S3 object Store in a vector database; the vector database can index vectors and quickly find similar vectors in a large amount of vector data; Step S5: Encode the query information input by the user into a query vector Vector query , and retrieve the matching similar vectors from the vector database object ; Step S6: similar vector Vector retrieved from step S5 object , find the corresponding original picture (video) from the storage location of step S2 and return it; when returning the result, the system can provide relevant additional information, such as the time and location of the key object, other similar objects, etc., to help users understand and use the returned data more comprehensively.

[0007] The present invention combines advanced multimodal models to develop a video search method based on language description. This method can quickly identify, track and locate target personnel from massive video and image data, capture key features in complex scenes, and provide fast and accurate decision support for security monitoring and emergency response. The application of this technology not only significantly improves search efficiency, but also brings innovative and groundbreaking working methods to the security industry.

[0008] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0009] The present invention also discloses an electronic device, characterized in that it comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the above method when they are executed. Beneficial Effects

[0010] Based on the self-developed multimodal model, the present invention constructs a complete set of personnel search processes from video data preprocessing, feature extraction to multimodal fusion and matching. By combining natural language description, the video rapid positioning method provides automated personnel recognition and positioning capabilities, which can quickly and accurately find the target person in massive videos, significantly shortening the search time, reducing manual intervention, and greatly improving efficiency and accuracy. The application of the framework of the present invention marks the first implementation of multimodal technology in the field of security video analysis, bringing intelligent solutions to the security monitoring industry.

[0011] At the overall solution level, there is currently no similar process-based intelligent video personnel search solution in the industry. This invention is the first institution to apply multimodal model capabilities to video personnel search and realize commercialization, achieving efficient retrieval from text descriptions to video search results; BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The present invention is a flowchart of a method for searching video objects based on a multimodal model. DETAILED DESCRIPTION

[0013] The present invention relates to a method for searching video objects based on a multimodal model, which specifically comprises the following steps: Step S1, pull the video stream data produced by the camera, detect and extract key object information in the video stream; Specifically include: Step S11, pulling the video stream data produced by the camera; Step S12: Detect objects in the video stream and extract key objects, such as people, non-motor vehicles, and motor vehicles; Step S13: Record the timestamp and position coordinates of the key object in step S12 appearing in the video stream.

[0014] Step S2, storing the timestamp, location coordinates, and image (video) information corresponding to the key object extracted in step S1, and displaying the stored key object information by retrieval in the subsequent step S6; Step S3: Use the multimodal model Qwen2VL to encode the key objects and generate a vector Vector object ; Specifically: Use the multimodal model Qwen2VL to perform multimodal encoding on the extracted key objects. After multimodal encoding, an image (video) containing the key objects will generate a vector Vector object; The multimodal model refers to a machine learning model that can process and understand multiple types of data at the same time; the modality can include text, image, audio, and video; the multimodal model Qwen2VL is a large multimodal model developed by Alibaba, which provides the ability to understand and generate text and visual interactions; the vector Vector object It contains the detailed features and semantic information of the image.

[0015] Step S4: The vector Vector generated in step S3 object Stored in a vector database; the vector database can index vectors and quickly find similar vectors in a large amount of vector data; the key object encoded vector Vector extracted from the video stream data object The vector database can be updated regularly to reduce redundant data and improve query efficiency and data reliability.

[0016] Step S5: Encode the query information input by the user into a query vector Vector query , and retrieve the matching similar vectors from the vector database object ; Specifically include: Step S51: The user inputs query information, which may be text or a picture; Step S52: Use the multimodal model Qwen2VL to perform multimodal encoding on the query information and convert the query information into a query vector Vector query , the multimodal encoding method is the same as step S3; Step S53: Based on the query vector Vector query Search in the vector database to find similar vectors that match the query information. object .

[0017] Step S6: similar vector Vector retrieved from step S5 object , find the corresponding original picture (video) from the storage location of step S2 and return it; when returning the result, the system can provide relevant additional information, such as the time and location of the key object, other similar objects, etc., to help users understand and use the returned data more comprehensively.

[0018] Based on the self-developed multimodal model, the present invention constructs a complete set of personnel search processes from video data preprocessing, feature extraction to multimodal fusion and matching. By combining natural language description, the video rapid positioning method provides automated personnel recognition and positioning capabilities, which can quickly and accurately find the target person in massive videos, significantly shortening the search time, reducing manual intervention, and greatly improving efficiency and accuracy. The application of the framework of the present invention marks the first implementation of multimodal technology in the field of security video analysis, bringing intelligent solutions to the security monitoring industry.

[0019] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for video object search based on a multimodal model, characterized in that: The following steps are involved: Step S1, pull the video stream data produced by the camera, detect and extract key object information in the video stream; Step S2, storing the timestamp, location coordinates, and video image information corresponding to the key object information extracted in step S1, and the stored content will be retrieved in step S6 to display the stored key object information; Step S3: Use the multimodal model Qwen2VL to encode the key objects and generate a vector Vector object ; The multimodal model refers to a machine learning model that can process and understand multiple types of data at the same time; the modalities include text, images, audio, and video; the multimodal model Qwen2VL provides the ability to understand and generate text and visual interactions through the multimodal large model developed by Alibaba; Step S4: The vector Vector generated in step S3 object Stored in vector database; The vector database indexes vectors to quickly find similar vectors in a large amount of vector data; Step S5: Encode the query information input by the user into a query vector Vector query , and retrieve the matching similar vectors from the vector database object ; Step S6: similar vector Vector retrieved from step S5 object , find the corresponding original video picture from the storage location of step S2 and return it; when returning the result, the system provides relevant additional information to help the user understand and use the returned data.

2. The method for searching video objects based on a multimodal model according to claim 1, characterized in that: The step S1 specifically includes: Step S11, pulling the video stream data produced by the camera; Step S12, detecting objects in the video stream, and extracting key people, non-motor vehicles, and motor vehicle objects; Step S13: Record the timestamp and position coordinates of the key object in step S12 appearing in the video stream.

3. The method for searching video objects based on a multimodal model according to claim 1, characterized in that: The step S3 comprises: The multimodal model Qwen2VL is used to perform multimodal encoding on the extracted key objects. A video image containing the key objects is multimodally encoded to generate a vector Vector. object ; The vector Vector object It contains the detailed features and semantic information of the image.

4. The method for searching video objects based on a multimodal model according to claim 1, characterized in that: The step S4 comprises: The encoded vector of key objects extracted from the video stream data. object The data will be continuously stored in the vector database, which can be updated regularly to reduce redundant data and improve query efficiency and data reliability.

5. The method for searching video objects based on a multimodal model according to claim 2, characterized in that: The step S5 specifically includes: Step S51: The user inputs query information, which is text or a picture; Step S52: Use the multimodal model Qwen2VL to perform multimodal encoding on the query information and convert the query information into a query vector Vector query , the multimodal encoding method is the same as step S3; Step S53: Based on the query vector Vector query Search in the vector database to find similar vectors that match the query information. object .

6. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to claim 1.

7. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method according to claim 1 when executed.

Images