Movable target retrieval device and retrieval method for large and small model fusion

Abstract

The present invention discloses a movable target retrieval device and method with large and small model fusion, wherein the device comprises: a hardware system, a lightweight small model engine, a large language model engine, a database engine, and a user interaction interface for accessing a video source, configuring retrieval tasks, and displaying target retrieval results; wherein the lightweight small model engine works in conjunction with the large language model engine to form a large and small model fusion architecture to optimize computing resource utilization and enhance the flexibility of target retrieval. The present invention significantly enhances the flexibility, real-time nature, and adaptability of target retrieval through the innovative design of the large and small model fusion architecture combined with edge computing technology.

Description

Technical Field

[0001] The present invention relates to the technical field of video surveillance and target retrieval, and in particular to a movable target retrieval device and method that combines natural language processing and computer vision technology and is suitable for the fusion of large and small models in an edge computing environment. Background Art

[0002] With the popularization and expansion of video surveillance technology, users have an increasing demand for quickly and accurately retrieving specific targets from a large amount of video content. Traditional video retrieval methods usually rely on predefined target categories and often require professionals to manually set rules or write code to perform specific tasks. However, this method is not only time-consuming and labor-intensive, but also lacks flexibility and is difficult to deal with unknown or changing targets.

[0003] In recent years, advances in natural language processing (NLP) and computer vision (CV) have made open object retrieval based on natural language instructions possible. By converting users' natural language queries into machine-understandable forms, the system can understand and respond to users' needs more intuitively and conveniently. In addition, the development of edge computing technology provides support for real-time processing, which allows complex computing tasks to be performed directly near the data source, reducing latency and improving privacy protection.

[0004] Despite this, existing solutions are mostly focused on cloud deployment or detection of known object types, and there are still challenges in open semantic object retrieval in offline environments. Especially in resource-constrained mobile scenarios, how to efficiently use edge devices for complex data processing and analysis has become an urgent problem to be solved.

[0005] In the prior art, there is a technology that uses a server at an edge node close to the device to complete the structured processing of video data. However, this method uses a server to perform video structured processing, and the server is not easy to move, which is inefficient for mobile scenes. In addition, this method does not provide a technology for target retrieval using natural language. Summary of the invention

[0006] The present invention proposes a movable target retrieval device and a retrieval method that integrates large and small models, aiming to provide an open target retrieval solution that is applicable to a variety of scenarios, especially those that may have network connection restrictions.

[0007] The technical solution of the movable target retrieval device with large and small model fusion of the present invention is as follows: A movable target retrieval device with large and small model fusion, the device comprising: The hardware system includes an edge smart box, which is equipped with an inference chip for performing edge computing. The inference chip supports NVIDIA GPU, Huawei Ascend NPU, Suanneng TPU or Cambrian MLU; A lightweight small model engine, deployed on the edge smart box, for performing preliminary target detection and attribute recognition on the video stream and generating structured attribute data; The large language model engine is used to receive natural language queries from users and parse them into structured query statements; A database engine, used to store the structured attribute data and image feature vectors, and to retrieve target data according to the structured query statement; User interaction interface, used to access video sources, configure retrieval tasks, and display target retrieval results; The lightweight small model engine works together with the large language model engine to form a large and small model fusion architecture to optimize computing resource utilization and enhance the flexibility of target retrieval.

[0008] Preferably, the lightweight small model engine performs real-time analysis on multiple video sources in a multi-channel concurrent manner and utilizes hardware acceleration technology to improve target detection efficiency.

[0009] “The hardware acceleration technology includes using GPU, NPU or TPU in edge devices to improve the efficiency of video decoding, image preprocessing and target detection through parallel computing. For example: 1. In the video decoding stage, hardware codecs (such as H.264, H.265) are used to achieve real-time video frame extraction.

[0010] 2. In the image preprocessing stage, multiple video streams are processed in parallel through the CUDA or OpenCL framework.

[0011] 3. In the target detection stage, the deep learning acceleration capability of Cambrian MLU is used to optimize the reasoning of lightweight small models, so that high-performance detection can still be achieved in a resource-constrained environment. "

[0012] Preferably, the large language model engine performs natural language parsing through a fine-tuned 3B model to adapt to the computing power limitations of edge computing scenarios.

[0013] The large language model engine implements natural language parsing through a 3B model based on the Transformer architecture. The parameter scale of the 3B model is 300 million. It is fine-tuned with a domain-specific corpus dataset to improve parsing accuracy. At the same time, quantization technology is used to compress the model weights from 32-bit floating point numbers to 8-bit integers, reducing memory usage and optimizing its operating efficiency in edge computing environments. During the inference process, the 3B model runs in conjunction with lightweight hardware (such as GPU or NPU), and the inference delay is controlled within 50 milliseconds, which can respond to natural language query instructions entered by users in real time.

[0014] Preferably, the database engine matches target data based on the image feature vector and attribute information in the query statement, and supports traceability retrieval.

[0015] Preferably, the device supports operation in an offline environment, and all data processing is completed locally, avoiding uploading data to the cloud and improving data privacy protection capabilities.

[0016] Preferably, the user interaction interface supports access to real-time video streams and offline video files, and returns target search results in the form of a picture list, and the user can click on the picture to view the target details.

[0017] Preferably, the lightweight small model engine adopts a lightweight and optimized target detection algorithm, which can adapt to the edge computing environment and implement target detection tasks on low-power devices. The lightweight and optimized target detection algorithm includes but is not limited to YOLOtiny, MobileNetSSD, NanoDet and EfficientDet, etc. The specific algorithm can be flexibly selected according to application requirements and device performance.

[0018] Preferably, the inference chip supports hardware acceleration functions of image decoding, preprocessing and model inference to improve data processing performance.

[0019] The inference chip supports hardware acceleration functions for image decoding, preprocessing, and model inference to improve data processing performance. The hardware acceleration functions include but are not limited to using dedicated decoding modules (such as NVDEC, VPU) for image decoding, and using CUDA, TensorRT or OpenCL frameworks to achieve parallel acceleration of preprocessing and inference tasks. The specific hardware acceleration method can be flexibly selected according to the chip type and task requirements.

[0020] The retrieval method of the movable target retrieval device with large and small model fusion of the present invention comprises the following steps: 1. Start the device: Start the movable target retrieval device with large and small model fusion through the edge intelligent box and load the video source; 2. Video source configuration: access real-time video streams or offline video files through the user interface and preview the video; 3. Video analysis: Perform preliminary target detection and attribute recognition on the video stream through a lightweight small model engine, and generate structured attribute data; 4. Data storage: storing the structured attribute data and image feature vector in a database engine; 5. Natural language query: natural language search instructions are input through the user interface, and the large language model engine parses and generates structured query statements; 6. Target matching and retrieval: The database engine matches the target's image feature vector according to the structured query statement and generates retrieval results; 7. Result display: The target search results are displayed in the form of a picture list through the user interaction interface, and users are supported to click on the picture to view the target details.

[0021] Preferably, this technical solution ensures that the lightweight small model focuses on real-time tasks, while the large model focuses on complex parsing, optimizing the resource utilization of both. In the real-time target detection task, the small model is responsible for detecting and generating structured data, and the large model processes the query semantic parsing. The dynamic allocation algorithm is then used to allocate tasks to the small model or the large model according to the task type and substantive needs. The algorithm formula is as follows: ; in, It is the task type, which indicates the specific task that needs to be processed at present; is the task priority, which indicates the weight score of the task on the real-time requirement, and the range is usually , high priority, close to 1, the task requires immediate response, such as target detection or emergency event recognition; low priority, close to 0, the task allows a certain delay, such as user query parsing or complex semantic analysis.

[0022] Priority can be calculated based on: Corresponding time constraints: for example, the real-time requirements of the detection task; Computational complexity: Complex tasks are usually given lower priority; Task Importance: Certain critical tasks may require priority allocation of resources.

[0023] Real-time threshold, the demarcation point of task real-time requirements, usually in the range of .

[0024] Decide whether to assign the task to the large model or the small model: If the task priority Higher than , then it is allocated to the small model (real-time priority); If the task priority Lower than , it is assigned to the large model (complex processing first).

[0025] Small and lightweight models are used to handle tasks with high real-time requirements.

[0026] Large models, models with large number of parameters and high computational complexity, are used to handle complex tasks.

[0027] This invention significantly improves the flexibility, real-time and adaptability of target retrieval through the innovative design of large and small model fusion architecture and the combination of edge computing technology. The main technical effects are as follows: 1. Flexibility of open target search The present invention introduces a large language model engine to enable the system to parse the user's natural language query, dynamically generate structured search conditions, and realize flexible search for any target. The user does not need to predefine the target category, and the system can quickly respond to complex and diverse query requirements.

[0028] 2. Real-time and mobility By deploying a lightweight small model engine on the edge device, the present invention realizes real-time target detection and attribute extraction of video streams, can run independently in network-restricted or offline environments, and is suitable for scenarios such as outdoor operations and emergency response.

[0029] 3. Optimization of resource utilization The present invention adopts a large-small model fusion architecture that combines a lightweight small model with a large language model to achieve high-performance computing in a resource-constrained edge computing environment. The small model is responsible for efficiently processing object detection in the video stream, and the large model is responsible for complex semantic parsing. The two work together to optimize resource utilization.

[0030] 4. Multi-channel concurrency and efficient processing The system supports concurrent analysis in multiple pipeline modes, can process multiple video sources in real time at the same time, and further improves computing efficiency through hardware acceleration technology, performing particularly well in multi-tasking scenarios.

[0031] 5. Data Privacy Protection All data processing in the present invention is completed locally without uploading to the cloud, thus avoiding the risk of sensitive data leakage. In scenarios with high privacy protection requirements, such as personal privacy protection and event investigation, it provides higher security and reliability.

[0032] 6. Convenience of user operation The system simplifies the operation process of complex search tasks through an intuitive user interface. Users can enter query conditions through natural language, and the system intuitively displays the search results in a combination of pictures and texts, allowing users to click to view target details, which improves the ease of use of the system.

[0033] 7. Adaptability and scalability The present invention designs a flexible architecture, and both the lightweight small model and the large language model can be replaced or adjusted according to different application scenarios to meet the needs of different fields. At the same time, the system can introduce domain knowledge by fine-tuning the large language model, thereby enhancing the retrieval ability in professional scenarios.

[0034] 8. Wide range of application scenarios The invention is applicable to security monitoring, incident investigation, emergency response and other fields, and especially shows significant advantages in environments with poor network conditions, limited computing resources or high privacy protection requirements. Through the efficient collaboration of large and small models, the reliability and practicality of the target retrieval system are significantly improved.

[0035] The present invention realizes efficient, intelligent and flexible target retrieval function through large and small model fusion technology and edge computing architecture, provides significant advantages in flexibility, real-time performance, resource optimization and privacy protection, and meets practical applications in multiple scenarios and multiple needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a system structure diagram of the present invention; Figure 2 It is a data flow diagram of the present invention; Figure 3 It is a flow chart of the steps of the present invention. DETAILED DESCRIPTION

[0037] Embodiment 1: Target retrieval device for single-channel real-time video stream In the security monitoring scenario, the system needs to process a single-channel real-time video stream from a surveillance camera and support users to query specific targets (such as "a person wearing a blue coat") through natural language.

[0038] The implementation steps of this embodiment are as follows: 1. Device startup: Start the edge smart box and load the lightweight small model engine, large language model engine, and database engine.

[0039] 2. Video source access: Users access real-time video streams (such as RTSP protocol) through the user interaction interface and preview the video.

[0040] 3. Video Analysis: The lightweight small model engine analyzes the video stream in real time, detects objects in the picture and recognizes their attributes (such as color, size, direction of movement, etc.).

[0041] Hardware acceleration features: The inference chip uses a dedicated decoding module (such as NVDEC and VPU) to perform image decoding of real-time video streams.

[0042] The CUDA framework is used to perform preprocessing operations in parallel on the GPU to optimize video frame data.

[0043] Use TensorRT to accelerate model reasoning of lightweight target detection algorithms and improve detection efficiency.

[0044] The generated structured attribute data is stored in the database.

[0045] 4. Target retrieval: Users enter natural language queries (such as "find red cars") through the interactive interface.

[0046] The large language model engine parses user queries and converts natural language into structured query statements (such as `{"color": "red", "type": "car"}`).

[0047] 5. Matching and result display: The database engine matches the corresponding target according to the query statement and returns the search results.

[0048] The user interface displays the matching results in the form of a list of pictures, and users can click to view specific target details.

[0049] Features and Benefits Real-time: Utilize hardware acceleration technology and lightweight optimized target detection algorithms to ensure millisecond-level detection response.

[0050] Flexibility: Supports users to describe search conditions through natural language.

[0051] Privacy protection: All data processing is done locally to avoid privacy leakage.

[0052] Embodiment 2: Target retrieval device for multiple offline video files In an event investigation scenario, users need to analyze multiple offline video files (such as dashcams or surveillance videos) and query specific targets (such as "black SUV vehicle") through natural language.

[0053] Implementation steps of this embodiment 1. Device startup: Start the edge smart box and load the lightweight small model engine, large language model engine, and database engine.

[0054] 2. Multiple video source access: Users upload multiple offline video files through the interactive interface.

[0055] The system allocates an independent Pipeline to each video file to support parallel processing of multiple videos.

[0056] 3. Multi-channel video analysis and task allocation: Object Detection: Each Pipeline independently runs a lightweight small model engine to perform object detection and attribute extraction on the corresponding video file.

[0057] Hardware acceleration features: In the video decoding stage, hardware codecs (such as H.264, H.265) are used to extract video frames in parallel.

[0058] Use CUDA and TensorRT frameworks to perform preprocessing and model inference operations on the GPU to accelerate the object detection process.

[0059] Task allocation algorithm: The system dynamically allocates work to small and large models based on the real-time nature and complexity of the tasks: ; Task, which indicates the specific task to be processed, which can be a real-time detection task or a complex parsing task; :Task priority is calculated based on the task response time requirement Deadline(T) and computational complexity(T): ; : Real-time threshold, used to divide high-priority and low-priority tasks.

[0060] Example Task Assignment Enter the task: Task T1: Real-time video target detection, priority P(T1)=0.8.

[0061] Task T2: User complex semantic analysis, priority P(T2)=0.3.

[0062] Real-time threshold: set up =0.5.

[0063] Allocation results: For T1: P(T1)=0.8≥θr, assigned to the small model.

[0064] For T2: P(T2) = 0.3 < θr, it is assigned to the large model.

[0065] Result storage: The detection results of the small model (such as bounding boxes, target types, color attributes, etc.) are stored in the database engine in a structured form and associated with the video file tags.

[0066] 4. Target retrieval: The user inputs a natural language query (such as "Find black SUV vehicles") through the interactive interface.

[0067] The large language model engine parses the query and converts it into a structured query statement (such as `{"color": "black", "type": "SUV"}`).

[0068] The system matches the query conditions parsed by the large model with the target detection data of the small model.

[0069] 5. Matching and result display: Matching algorithm:

[0070] The database engine completes the retrieval using a combination of attribute matching and feature similarity calculation: ; : Comprehensive matching score, used to measure the total similarity between the target in the database and the query conditions; : Attribute matching score; : Feature similarity score; , : Attribute weight parameter and feature similarity weight, dynamically adjusted according to task requirements.

[0071] Among them, the attribute matching formula is: ; Among them, are the target attribute and the query attribute; : The attribute value of the i-th target in the database (such as color, type); : The corresponding attribute value in the query conditions; : Attribute matching function, with a value of 0 or 1: 1 means the attribute values match; 0 means the attribute values do not match.

[0072] is the attribute weight, which is used to adjust the proportion of the attribute's influence on the matching score (for example, the color weight is higher than the size weight).

[0073] The feature similarity formula is: ; : feature similarity score; : The feature vector of the i-th target in the database; : Feature vector generated by query conditions; : feature vector The Euclidean norm of : ; : Dimension of the feature vector.

[0074] : The dot product of two vectors, the core part of technical similarity.

[0075] Results: The user interface displays the matching results in the form of a picture list, and indicates the time period and video file where the target is located.

[0076] After the user clicks on the picture, the video clip corresponding to the target can be played directly.

[0077] Features and Benefits 1. Task allocation and efficient collaboration: Through dynamic division of labor through task allocation algorithm, small models handle high-priority real-time tasks, large models handle complex tasks, and optimize resource utilization.

[0078] 2. High concurrency: Utilizing multi-Pipeline architecture and hardware acceleration technology, it supports simultaneous analysis of multiple video files, significantly improving processing efficiency.

[0079] 3. Flexible search: Combining large language models and lightweight target detection algorithms, it supports complex and diverse query requirements and achieves precise target retrieval.

[0080] 4. Adaptability: The number of pipelines and hardware resource allocation can be dynamically adjusted according to actual needs to flexibly adapt to different task loads.

[0081] 5. Smart matching: Attribute matching is combined with feature similarity calculation to ensure efficient matching of user query conditions and target data.

Claims

1. A movable target retrieval device with large and small model fusion, characterized in that: The device comprises: A hardware system, including an edge smart box, in which an inference chip for performing edge computing is configured; A lightweight small model engine, deployed on the edge smart box, for performing preliminary target detection and attribute recognition on the video stream and generating structured attribute data; The large language model engine is used to receive natural language queries from users and parse them into structured query statements; A database engine, used to store the structured attribute data and image feature vectors, and to retrieve target data according to the structured query statement; User interaction interface, used to access video sources, configure retrieval tasks, and display target retrieval results; The lightweight small model engine works together with the large language model engine to form a large and small model fusion architecture.

2. The movable target retrieval device with large and small model fusion according to claim 1 is characterized in that: The lightweight small model engine performs real-time analysis on multiple video sources in a multi-channel concurrent manner and uses hardware acceleration technology to improve target detection efficiency.

3. The movable target retrieval device with large and small model fusion according to claim 1 is characterized in that: The large language model engine performs natural language parsing through a fine-tuned 3B model to adapt to the computing power limitations of edge computing scenarios.

4. The movable target retrieval device with large and small model fusion according to claim 1 is characterized in that: The database engine matches target data based on the image feature vector and the attribute information in the query statement, and supports traceability retrieval.

5. The movable target retrieval device with large and small model fusion according to claim 1 is characterized in that: The device supports operation in an offline environment, and all data processing is completed locally.

6. The movable target retrieval device with large and small model fusion according to claim 1 is characterized in that: The user interaction interface supports access to real-time video streams and offline video files, and returns target search results in the form of a picture list. Users can click on the picture to view the target details.

7. The movable target retrieval device with large and small model fusion according to claim 1 is characterized in that: The lightweight small model engine adopts a lightweight optimized target detection algorithm.

8. The movable target retrieval device with large and small model fusion according to claim 1 is characterized in that: The inference chip supports hardware acceleration functions of image decoding, preprocessing and model inference.

9. A retrieval method for a movable target retrieval device with large and small model fusion as described in any one of claims 1 to 8, characterized in that: The following steps are involved: A. Start device: Start the movable target retrieval device of large and small model fusion through the edge intelligent box and load the video source; B. Video source configuration: access real-time video streams or offline video files through the user interface and preview the video; C. Video analysis: Perform preliminary target detection and attribute recognition on the video stream through a lightweight small model engine, and generate structured attribute data; D. Data storage: storing the structured attribute data and image feature vector in a database engine; E. Natural language query: natural language search instructions are input through the user interface, and the large language model engine parses and generates structured query statements; F. Target matching and retrieval: The database engine matches the target's image feature vector according to the structured query statement and generates retrieval results; G. Result display: The target search results are displayed in the form of a picture list through the user interaction interface, and users are supported to click on the picture to view the target details.

Images