Target object identification method fusing visual identification and multiple modes

Abstract

The invention discloses a visual identification and multi-modal fused target object identification method, and the method comprises the steps: constructing a model which comprises an image processing module, a rapid segmentation module and a multi-modal identification module, and enabling a candidate region outputted by the rapid segmentation module to enter a large model reasoning link after the candidate region is screened through a confidence threshold value; a confidence coefficient threshold value is dynamically adjusted through association adaptation of a threshold value parameter and target scene complexity in combination with real-time scene analysis, directional training is carried out by adopting a small-batch gradient descent method, a fine-tuned large model is used as a teacher model, and a lightweight student model is constructed through combined training of soft tags and hard tags. According to the method, the real-time advantage of a rapid segmentation module is reserved, the recognition precision is improved through the semantic understanding capability of the large model, the model volume is compressed to be less than 1 / 5 of the original large model on the premise of ensuring that the precision loss is less than 2%, the reasoning speed is improved by more than three times, and ineffective large model reasoning is reduced.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology and relates to a method for target object recognition that integrates visual recognition and multimodal recognition. Background Technology

[0002] In the field of computer vision, object recognition technology, as a core application area, has been widely used in various scenarios such as intelligent monitoring, autonomous driving, and industrial quality inspection. Currently, the mainstream object recognition solutions in existing technologies are mainly divided into two categories: single-stage recognition algorithms based on traditional deep learning, and multimodal recognition technologies based on large models. Single-stage target detection algorithms based on the YOLO series are among the most widely used solutions in current technology. Their core principle is to transform the target detection task into a regression problem by constructing a deep convolutional neural network, directly predicting the bounding box and class probability of the target in the image. These algorithms are known for their high real-time efficiency, achieving processing speeds of tens of frames per second on ordinary hardware devices. Therefore, they are widely used in scenarios with high real-time requirements, such as dynamic target tracking in video surveillance. However, their limitations are also quite apparent: due to the design constraints of the network structure, the recognition accuracy is often low for small targets, occluded targets, or targets with blurred features in complex scenes, especially in category segmentation tasks, where the error rate increases significantly. Multimodal large-scale model-based recognition technology is an advanced approach that has emerged in recent years. This technology integrates information from multiple modalities, such as images and text, and leverages the powerful semantic understanding capabilities of pre-trained large-scale models to achieve refined object recognition. Its advantage lies in its ability to utilize rich prior textual knowledge to improve the recognition accuracy of rare and complex targets, particularly excelling in cross-domain transfer tasks. However, this approach has significant drawbacks: large-scale models typically have billions or even tens of billions of parameters, resulting in extremely slow inference speeds and difficulty meeting real-time requirements; furthermore, its operation requires extremely high computational resources, significantly increasing server deployment costs and limiting its adoption in small and medium-sized application scenarios. In addition, there are a few attempts in the existing technology to combine traditional recognition algorithms with large models, but these solutions mostly adopt a simple concatenation method, which does not solve the problem of reasoning efficiency of large models, or loses the speed advantage of traditional algorithms in the process of fusion, and fails to achieve an effective balance between accuracy and efficiency. Summary of the Invention

[0003] The technical problem solved by this invention is to provide a target object recognition method that integrates visual recognition and multimodal recognition, which significantly improves the accuracy of target object recognition while ensuring real-time performance and avoiding increased server resource costs.

[0004] This invention is achieved through the following technical solution: A method for target object recognition that integrates visual recognition and multimodal methods includes the following operations: 1) Construct a model that includes an image processing module, a fast segmentation module, and a multimodal recognition module: The image processing module receives image data and preprocesses the image; the preprocessed image data is then transmitted to the fast segmentation module. After receiving the preprocessed image data, the fast segmentation module extracts image features through a convolutional neural network, locates and segments candidate regions in the image that may contain target objects. The multimodal recognition module receives candidate region image sub-blocks output by the fast segmentation module, extracts features from the image sub-blocks, generates image feature vectors, determines the category of the target object by calculating the similarity between the image feature vectors and the text feature vectors provided by the text description library, and outputs the recognition result. The text description library contains text feature vectors of text descriptions of various target objects; 2) The multimodal recognition module is trained using a dataset containing labeled images and the mini-batch gradient descent method. 3) Compress the image processing module, fast segmentation module, and trained multimodal recognition module based on soft and hard labels to generate a lightweight model; Among them, soft labels convey information about inter-class similarity, while hard labels ensure the learning of real annotations; 4) Deploy the lightweight model to a server or edge device. The lightweight model will identify the input image or video frame and output the identification result, which includes the target object category identifier, confidence score and coordinate information.

[0005] Furthermore, the preprocessing performed by the image preprocessing module includes size normalization and pixel value normalization, compressing pixel values ​​to 0-1.

[0006] Furthermore, the fast segmentation module employs either the YOLO11 algorithm or the Faster R-CNN algorithm; The operation of locating and segmenting candidate regions in an image that may contain the target object is as follows: By fusing image features from different levels using a feature pyramid network, the ability to perceive targets of different sizes can be enhanced. The bounding box coordinates and confidence scores of candidate regions are generated using the anchor box mechanism, and candidate regions with confidence scores greater than the threshold are selected. The image sub-blocks corresponding to the selected candidate regions are cropped out and used as input to the multimodal recognition module.

[0007] The bounding box coordinates and confidence level of the generated candidate region are: Multiple anchor frame sizes are preset based on the characteristics of the target object scene, covering the range of common target object sizes; By calculating the intersection-over-union (IoU) ratio between the anchor frame and the target object in the image, anchor frames with an IoU > 0.5 are selected as initial candidate regions. Then, the coordinates and size of the anchor frames are adjusted using a bounding box regression algorithm to make the anchor frames fit the outline of the target object better. Finally, the coordinates and confidence scores of the candidate region bounding boxes are output.

[0008] Furthermore, the complexity and priority of differentiating scenarios are also based on one or more of the following factors: the number of candidate regions output by the fast segmentation module, the region overlap rate, and the target area variance. And the threshold is dynamically adjusted based on the preset scene complexity-threshold mapping table; Resources are also adjusted based on the candidate region identification results: Allocate computing resources for the multimodal recognition module based on confidence scores; Hardware resources are dynamically scheduled based on the number of candidate regions. Memory resources are allocated based on scenario priority.

[0009] Furthermore, the multimodal recognition module adopts a multimodal recognition module based on the CLIP model or BLIP model. After generating the image feature vector, it determines the category of the target object by calculating the cosine similarity between the image feature vector and the text feature vector, and outputs the recognition result containing the target object category and confidence level. Furthermore, the training of the multimodal recognition module is as follows: A dataset containing 100,000 labeled images was selected for training. Mini-batch gradient descent with a batch size of 30-40 was used, and the number of training rounds was set to 50-100. A linear preheating combined with a cosine annealing strategy is used to gradually decay the learning rate in order to ensure training stability and convergence.

[0010] Furthermore, the compression includes the following operations: Using the trained multimodal recognition module as the teacher model, a student model with 1 / 4 to 1 / 5 the number of parameters of the teacher model was designed and constructed. By combining the probability distribution output by the teacher model as a soft label with the true class label of the sample as a hard label, the student model is jointly trained. The compressed model is lightweighted by compressing the 32-bit floating-point weights into a 4-bit representation and using 16-bit floating-point precision for calculations during inference.

[0011] Compared with the prior art, the present invention has the following beneficial technical effects: The target object recognition method integrating visual recognition and multimodal recognition provided by this invention adopts a hierarchical collaborative recognition mechanism of fast image segmentation and multimodal recognition modules: The image is segmented rapidly using a fast segmentation module (such as the YOLO algorithm). Only the segmented candidate regions are input into the multimodal recognition module for fine-grained identification, rather than performing large-scale model inference on the entire image. This retains the real-time advantage of YOLO while improving recognition accuracy through the semantic understanding capabilities of the large model. The first layer is the fast target region segmentation layer, led by the YOLO algorithm. Its core task is to quickly scan the entire preprocessed image, locate and segment candidate regions that may contain the target object, without involving fine-grained judgment of specific categories. It only outputs region bounding boxes and confidence scores to ensure real-time performance. The second layer is the fine-grained recognition layer, led by the multimodal recognition module. It only receives the candidate region sub-blocks output from the first layer and does not need to process irrelevant regions of the entire image. It focuses on accurately determining the category of the target object through feature matching between the image and text, ensuring recognition accuracy.

[0012] This invention also employs a fine-tuning strategy for the multimodal recognition module in specific recognition scenarios: Based on the contextual features of the target objects (such as the style of police uniforms and name tags in public security scenarios), a dedicated labeled dataset is constructed. Mini-batch gradient descent is used for targeted fine-tuning, which improves the model's sensitivity to the recognition of targets in specific domains. This solves the problem of insufficient accuracy of general large models in vertical scenarios. The fine-tuning parameter settings and dataset construction methods are unique. This invention also combines knowledge distillation and quantization compression to lightweight the model: Using a trained or fine-tuned large model as the teacher model, a lightweight student model is constructed through joint training with soft and hard labels. Simultaneously, 16-bit quantization is employed to further reduce model size. While maintaining an accuracy loss of less than 2%, the model size is compressed to less than 1 / 5 of the original large model, and inference speed is increased by more than 3 times. This is mainly due to the reduced input data volume after compression: the large model only processes candidate region sub-blocks (the area of ​​a single block is typically 10%-30% of the entire image), avoiding feature extraction and calculation of irrelevant regions such as the background, reducing data processing volume by more than 60%.

[0013] This invention has the advantage of lightweight model: by constructing a student model with only 1 / 5 of the parameters of the original model through knowledge distillation, and with 4-bit quantization compression, the amount of computation is greatly reduced and the memory read and write latency is reduced; further, it is empowered by parallel inference: the large model inference is split into two sub-tasks, feature extraction and similarity calculation, which are processed in parallel by multiple threads, reducing the serial waiting time of a single task and improving the processing efficiency per unit time.

[0014] This invention employs a dynamic adaptation mechanism between candidate region confidence screening and large model inference: Candidate regions output by the fast segmentation module need to be filtered by a confidence threshold (which can be dynamically adjusted) before entering the large model inference stage. The confidence threshold is dynamically adjusted by associating the threshold parameter with the complexity of the target scene and combining it with real-time scene analysis. The core is to make the threshold positively correlated with the density and occlusion probability of the target in the scene. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the model and recognition process of the present invention; Figure 2 This is a schematic diagram for segmenting regions in an image that may contain objects. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to embodiments. These descriptions are for illustrative purposes only and are not intended to limit the scope of the invention.

[0017] A method for target object recognition that integrates visual recognition and multimodal methods includes the following operations: 1) Construct a model that includes an image processing module, a fast segmentation module, and a multimodal recognition module: The image processing module receives image data and preprocesses the image; the preprocessed image data is then transmitted to the fast segmentation module. After receiving the preprocessed image data, the fast segmentation module extracts image features through a convolutional neural network, locates and segments candidate regions in the image that may contain target objects. The multimodal recognition module receives candidate region image sub-blocks output by the fast segmentation module, extracts features from the image sub-blocks, generates image feature vectors, determines the category of the target object by calculating the similarity between the image feature vectors and the text feature vectors provided by the text description library, and outputs the recognition result. The text description library contains text feature vectors of text descriptions of various target objects; 2) The multimodal recognition module is trained using a dataset containing labeled images and the mini-batch gradient descent method. 3) Compress the image processing module, fast segmentation module, and trained multimodal recognition module based on soft and hard labels to generate a lightweight model; Among them, soft labels convey information about inter-class similarity, while hard labels ensure the learning of real annotations; 4) Deploy the lightweight model to a server or edge device. The lightweight model will identify the input image or video frame and output the identification result, which includes the target object category identifier, confidence score and coordinate information.

[0018] The recognition model and workflow are explained in detail below.

[0019] 1. Recognition Model See Figure 1 The recognition model of this invention includes an image input module, a fast segmentation module, a multimodal recognition module, and a compression module; data transmission between the modules is achieved through the following standardized interfaces to ensure compatibility and transmission efficiency: Image data interface: A RESTful API using the HTTP / 2 protocol, supporting binary streaming of preprocessed image data with a transmission rate of ≥100MB / s and a latency of ≤5ms.

[0020] Feature Vector Interface: Adopts gRPC protocol, transmits image feature vectors and text feature vectors based on Protocol Buffers (Protobuf) serialization format, with a data compression ratio of 1:4, reducing transmission bandwidth usage.

[0021] Control signal interface: Adopts MQTT protocol for inter-module state synchronization (such as preprocessing completion signal, candidate area output signal), supports subscription-publish mode, and ensures reliable signal transmission.

[0022] 2. Workflow of each module 2.1 Image Preprocessing Module This module is responsible for receiving externally input image data, supporting common image formats (such as JPG, PNG, etc.), and preprocessing the images, including size standardization (uniformly adjusting to 640×640 pixels) and pixel value normalization (compressing pixel values ​​to the range of 0-1). The preprocessed image data is then transmitted to the YOLO fast segmentation module.

[0023] The following explanation uses a surveillance camera as the external input source: Surveillance cameras (such as network HD cameras and industrial cameras) push video streams in real time via the RTSP protocol. The image preprocessing module parses the video stream and extracts single-frame images (frame rate supports 15-30fps), which are then directly fed into the preprocessing process.

[0024] 2.2 Fast Segmentation Module Using the YOLO11 algorithm as the basic framework, this module receives preprocessed image data and then uses a convolutional neural network to quickly extract image features, locate and segment regions in the image that may contain objects (i.e., candidate regions).

[0025] The specific process is as follows: By using Feature Pyramid Network (FPN) to fuse image features from different levels, the ability to perceive targets of different sizes is enhanced. By using an anchor box mechanism (optimized for specific scenarios), the accuracy of candidate region localization is improved, and the bounding box coordinates and confidence scores of the candidate regions are generated. Preset anchor frame sizes: Based on the characteristics of the target object scene, three types of anchor frames are preset (small size 32×32 pixels, medium size 64×64 pixels, and large size 128×128 pixels), covering the common target object size range.

[0026] Anchor frame matching and adjustment: By calculating the intersection-over-union (IoU) ratio between the anchor frame and the target object in the image, anchor frames with an IoU > 0.5 are selected as initial candidate regions; then, the coordinates and size of the anchor frames are adjusted by the bounding box regression algorithm to make the anchor frames fit the outline of the target object better, and finally the accurate bounding box coordinates and confidence scores of the candidate regions are output.

[0027] Specifically, for example, candidate regions with a confidence level greater than 0.65 are selected, and the image sub-blocks corresponding to the selected candidate regions are cropped out and used as input to the multimodal recognition module.

[0028] The processing time of this module is controlled within 15 milliseconds to ensure the real-time performance of the system. 2.3 Multimodal Recognition Module A multimodal recognition module based on an improved CLIP model is employed. This module receives candidate region image sub-blocks output by the fast segmentation module and combines them with a pre-set text description library (containing standard text descriptions of various target objects) to perform target object recognition. The specific steps are as follows: Feature extraction is performed on image sub-blocks to generate image feature vectors; Encode the text in the text description library to generate text feature vectors (such as...) Figure 2 (as shown) For example, the following text description: Police uniform dress code: Blue standard police uniform, with police number and shoulder insignia, and the jacket has special police uniform insignia. The style conforms to the unified standards of public security.

[0029] Police uniforms not worn in accordance with regulations: Blue standard police uniforms, but lacking police numbers / shoulder insignia, or paired with non-standard clothing (such as civilian jackets), and the style is not standard.

[0030] Wearing a name tag: Wear a rectangular name tag on your chest, which includes your photo, name, and company logo. The name tag should be hung in accordance with the company's regulations.

[0031] Not wearing a name tag: No name tag on the chest, or the name tag is not hung in accordance with regulations (e.g., put in a pocket, or cover key information).

[0032] Taking "standard police uniform" as an example, the generated text feature vector, after being encoded by the CLIP model text encoder, generates a 512-dimensional feature vector: [0.123, 0.089, -0.056, 0.211, -0.134, 0.098,..., 0.176]. After quantization, this vector is compared with the image feature vector using cosine similarity calculation to determine the category of the target object and output the recognition result.

[0033] Figure 2 The image of a person captured by the surveillance camera shown is segmented by the fast segmentation module, which identifies the candidate region as the "upper body of the person". After feature extraction by the multimodal recognition module, a 512-dimensional image feature vector A is generated (partial dimensions: [0.112, 0.095, -0.062, 0.203, -0.128, 0.105, ..., 0.169]).

[0034] The text feature vector B corresponding to "standard police uniform" in the text description library (partial dimensions: [0.123, 0.089, -0.056, 0.211, -0.134, 0.098, ..., 0.176]).

[0035] The cosine similarity is calculated as: cosθ = (A・B) / (||A||×||B||). Substituting the vector, we get cosθ = 0.97, which is greater than the set threshold of 0.85. Therefore, the target category corresponding to the candidate region is determined to be "standard police uniform", and the output confidence level is 97%.

[0036] To improve recognition accuracy, this module underwent a second fine-tuning of the large model: using a dataset containing 100,000 labeled images (covering common targets and targets in complex scenes), it was trained using mini-batch gradient descent (batch size=32) for 50 training rounds, with the initial learning rate set to 0.0001, which was decayed to 1 / 10 of the original rate every 10 rounds. The training process is as follows: Data loading: 100,000 labeled images were divided into training, validation and test sets in an 8:1:1 ratio. The training set was expanded to 150,000 images through data augmentation (random cropping, flipping, brightness adjustment).

[0037] Model training: Mini-batch gradient descent is used, with 32 images input per iteration (batch size=32). The matching loss between the model prediction and the labeled text is calculated (using the cross-entropy loss function), and the model parameters are updated through backpropagation.

[0038] Training objective: To enable the model to learn the correspondence between image features and text descriptions of target objects in specific scenarios, thereby improving the sensitivity and accuracy of identifying targets in vertical domains (such as police uniforms and name tags).

[0039] 2.4 Model Compression Knowledge distillation is used to compress the finely tuned multimodal recognition module: Reduced deployment costs: Compression enables the model to run on ordinary servers (such as 16-core CPUs and 32GB of memory), eliminating the need for high-end GPU hardware and reducing hardware procurement and maintenance costs.

[0040] Improve inference speed: Compression reduces the number of model parameters and computational load, meeting real-time recognition requirements (such as a processing speed of 15-30fps in monitoring scenarios).

[0041] Reduced resource consumption: Compression reduces model memory usage and CPU / GPU utilization, avoiding resource contention with other systems and ensuring overall system stability.

[0042] Using a trained large-scale model as the teacher model, a student model with approximately one-fifth the number of parameters was designed and constructed. By combining the probability distribution (soft label) output by the teacher model with the true class label (hard label) of the sample, where the soft label conveys inter-class similarity information and the hard label ensures learning from the true annotation, the student model can maintain a recognition accuracy close to that of the teacher model while reducing the model parameter size and computational complexity. Specifically, the teacher model output (soft label) is as follows: the predicted probability distribution for a candidate region is "police uniform worn in accordance with regulations: 0.92, police uniform not worn in accordance with regulations: 0.05, wearing a name tag: 0.91, not wearing a name tag: 0.09" (including inter-class similarity information).

[0043] Sample True Label (Hard Label): The true category of this candidate region is "Standardized police uniform + wearing name tag", corresponding to the one-hot encoded label "[1, 0, 1, 0]" (clearly defining the true category).

[0044] The combination of soft and hard labels: During training, a weighted summation loss function is used, with the soft label loss weight set to 0.3 and the hard label loss weight set to 0.7. The total loss = 0.3 × soft label loss + 0.7 × hard label loss, so that the student model can learn the fine-grained inter-class relationships of the teacher model without deviating from the true labels.

[0045] Joint training of student models: Initialization: Based on the network structure of the teacher model, redundant layers are pruned (such as deleting some fully connected layers), the number of parameters is compressed to 1 / 5 of that of the teacher model, and the newly added parameters after pruning are randomly initialized.

[0046] Joint training: Input training set samples, simultaneously obtain the soft labels output by the teacher model and the true hard labels of the samples, calculate the total loss and backpropagate, update the student model parameters, set the training rounds to 30 rounds, the initial learning rate is 0.0001, and it decays to 1 / 10 of the original value every 10 rounds.

[0047] Fine-tuning and optimization: After training, the student model is fine-tuned using the validation set, adjusting the parameters of the last two fully connected layers to ensure that the accuracy loss is controlled within 2%.

[0048] The compressed model is quantized to 4-bit, compressing the original 32-bit floating-point weights into a 4-bit representation, and 16-bit floating-point precision is used for calculation during inference, further reducing model capacity and computational resource consumption.

[0049] The compressed model size is kept under 7GB, and the inference speed is 3 times faster than before compression.

[0050] After model compression, the student model's input consists of standardized image data or video frames. The model performs feature extraction and object detection on the input data, outputting detection results that include the object category, confidence score, and its location information (bounding box coordinates) in the image.

[0051] 2.5 Scene Complexity and Dynamic Adjustment: The complexity and priority of a scene are distinguished by one or more of the following factors: the number of candidate regions output by the fast segmentation module, the region overlap rate, and the target area variance. And the threshold is dynamically adjusted based on the preset scene complexity-threshold mapping table; Resources are also adjusted based on the candidate region identification results: Allocate computing resources for the multimodal recognition module based on confidence scores; Hardware resources are dynamically scheduled based on the number of candidate regions. Memory resources are allocated based on scenario priority.

[0052] Specifically, the complexity of the scene is quantified by three indicators: the number of candidate regions, the region overlap rate, and the target area variance output by the YOLO module (e.g., if there are more than 15 candidate regions and the overlap rate is more than 30%, it is judged as a high-complexity scene). Dynamically adjust the threshold based on a preset scene complexity-threshold mapping table: For low-complexity scenarios (such as open office areas with ≤5 candidate areas), the threshold is set to 0.75 to filter out low-confidence interference. For medium-complexity scenarios (such as shopping mall lobbies with 6-15 candidate areas), the threshold is set to 0.65. For highly complex scenarios (such as crowds at a train station with more than 15 candidate regions), the threshold is set to 0.55 to avoid missing occlusions or small targets.

[0053] Allocate computing resources based on threshold filtering results to reduce unnecessary large model computations and achieve dynamic allocation of system resources: For high-confidence candidate regions (confidence > 0.8), allocate basic computing resources (such as single-threaded processing and 3 iterations of feature extraction). For regions with low confidence but not filtered (confidence 0.55-0.8), allocate enhanced computing resources (such as multi-threaded processing and 5 iterations of feature extraction).

[0054] Dynamically scheduling hardware resources based on the target quantity is as follows: When there are ≤8 candidate regions, only CPU resources are used for processing; When there are more than 8 candidate regions, GPU-assisted computing is automatically activated, and one additional GPU processing thread is added for every 4 additional candidate regions.

[0055] Memory resources are allocated based on scenario priority as follows: Reserve 50% of memory for core scenarios (such as identification of classified areas in public security) to avoid resource contention; For common scenarios (such as retail product recognition), dynamic memory allocation is adopted, which can be flexibly adjusted according to real-time computing needs.

[0056] 2.6 Deployment of Lightweight Model and Display of Detection Results The lightweight model is an offline preprocessing module that performs compression only once before model deployment, and does not need to be run again during the deployment phase.

[0057] Offline compression: After the entire model training is completed, the model is compressed using knowledge distillation, quantization, or pruning techniques to generate a lightweight model file (such as .pth format). The compression process takes about 24 hours (based on a dataset of 100,000 labeled images).

[0058] Deployment phase: The compressed lightweight model is directly deployed to the server or edge device without carrying the original large model and the computational logic of the compressed module. Only the lightweight model needs to be loaded for inference computation.

[0059] Subsequent maintenance: If the scenario requirements change (such as adding a new target category), the large model needs to be retrained and compressed a second time through the compression module to generate a new lightweight model, which can then replace the original deployed model without changing the overall system architecture.

[0060] After the detection results are parsed by the lightweight model processing module, they can be displayed and alerted in the following ways: (1) Visualization: The target detection box, category label and confidence information are superimposed on the detection image, and the recognition results are displayed in real time through the front-end interface so that users can view them intuitively; (2) Structured output: Output the detection results in JSON or other standardized data formats, including category identifiers, confidence scores and coordinate information, for downstream business modules to call or perform statistical analysis; (3) Voice and sound and light prompts: When the detection result shows that there is a violation category (such as not wearing police uniform properly, not wearing work badge, etc.), the terminal automatically triggers the sound and light alarm device, emits a prompt sound and flashes a red indicator light to realize the real-time early warning prompt function.

[0061] Specific implementation examples are given below.

[0062] Example 1: Standardized Identification of Police Uniforms and Badges Wearing in Specific Areas by Public Security Departments In specific areas of the public security department (such as case-handling areas and confidential meeting rooms), it is necessary to identify personnel based on the standardization of their police uniform attire and name tag wearing. The implementation process is as follows: The image input module acquires real-time images (1280×720 resolution) from surveillance cameras deployed in a specific area, and preprocesses them into 640×640 pixel images; The YOLO fast segmentation module segmented the human body regions of all people in the area as candidate regions within 15 milliseconds, and a total of 8 candidate regions were segmented. The multimodal recognition module combines text descriptions from the text description library, such as "police uniform worn in accordance with regulations", "police uniform not worn in accordance with regulations", "wearing name tag", and "not wearing name tag", to identify each candidate region.

[0063] Application scenario: At the entrance to the police law enforcement and case handling area, two 1080P surveillance cameras are deployed with unobstructed and uniform lighting.

[0064] Number of personnel: There are 6 people present in the case-handling area at any given time.

[0065] Total processing time: The total time from acquisition to output of recognition results for a single image is 110 milliseconds, including 10 milliseconds for image preprocessing, 15 milliseconds for YOLO segmentation, 75 milliseconds for large model recognition, and 10 milliseconds for result encapsulation.

[0066] The recognition results show: Large screen display: The monitoring center in the case handling area displays the labeled monitoring footage in real time. The candidate areas for 6 people are marked with colored boxes. People who are dressed correctly and wearing name tags are marked with green boxes, those who do not wear name tags correctly are marked with yellow boxes, and those who do not wear police uniforms correctly are marked with red boxes. The confidence level is displayed next to the box.

[0067] Background Records: The server automatically records each recognition result, including timestamp, personnel location, violation type, and other information, and supports exporting Excel reports for easy subsequent verification.

[0068] The results show that the present invention can accurately identify 6 personnel who are wearing police uniforms and name tags in accordance with regulations (confidence level above 97%), 1 personnel who are not wearing name tags in accordance with regulations (confidence level 96.8%), and 1 personnel who are not wearing police uniforms in accordance with regulations (confidence level 97.2%). The compressed model runs on a monitoring server commonly used by public security departments (16-core CPU, 32GB memory). The total processing time for a single image is 110 milliseconds, which can meet the needs of real-time monitoring and recognition. The server resource utilization rate is reduced by 55% compared with the uncompressed large model, which can effectively ensure the stable operation of the monitoring system.

[0069] This solution achieves an effective balance between target object recognition accuracy and operational efficiency, while controlling server resource costs.

[0070] Other embodiments are given below.

[0071] Example 2 An alternative to the YOLO algorithm is the Faster R-CNN algorithm, which can be used to quickly segment target regions in images. Faster R-CNN generates candidate regions through a Region Proposal Network (RPN). Although its processing time per frame is slightly longer (approximately 60-80 milliseconds), it performs better in segmenting small targets.

[0072] Its collaborative mode with the multimodal recognition module can remain unchanged: first, Faster R-CNN outputs candidate regions, and then inputs them into a large model for recognition. This can also achieve the effect of fast localization and fine recognition, and is suitable for scenarios with higher requirements for small target recognition. Example 3

[0073] An alternative to the multimodal recognition module: The BLIP model can be used instead of the CLIP model as the basic multimodal model. BLIP, through a training method that combines contrastive and generative learning, has unique advantages in understanding the association between images and text. Its collaborative logic and secondary fine-tuning process (such as dataset construction and training parameter settings) with the front-end segmentation algorithm can be kept consistent. Only the interface adaptation of the model's feature extraction layer needs to be adjusted to achieve the same accuracy in target recognition. Example 4

[0074] Alternatives to model compression methods Replacement for compression techniques: Model pruning can be used to replace knowledge distillation for model lightweighting. By calculating the importance scores of neurons in each layer, redundant neurons are pruned (e.g., retaining 70% of the core neurons), and then fine-tuning is used to restore model accuracy. This approach eliminates the need to build a teacher-student model, directly simplifying the original large model, compressing the model size to 1 / 4-1 / 5 of its original size, and improving inference speed by 2-3 times, achieving results comparable to the original compression scheme. Example 5

[0075] Adjustment of quantization method: Hybrid precision quantization can be used instead of 16-bit full quantization. 16-bit quantization is used for precision-sensitive convolutional layers in the model, and 8-bit quantization is used for activation layers. While ensuring that the recognition accuracy loss does not exceed 3%, the model memory usage is further reduced (by 30% compared to 16-bit full quantization), and the inference speed is basically the same. This is suitable for scenarios with extremely limited memory resources.

[0076] Example 6 Dual threshold screening mechanism: In addition to the original single confidence threshold screening, a target region area threshold is added. When the area of ​​the candidate region output by YOLO is less than the set threshold (such as 0.5% of the total image area), it is directly identified as a small target and the inference weight of the large model is automatically increased (such as increasing the number of feature extraction iterations). It can dynamically adapt to targets of different sizes without manual parameter adjustment, and still maintain a balance between accuracy and efficiency. Parallel inference optimization: The inference task of the multimodal recognition module is split into two subtasks: feature extraction and similarity calculation, which are processed in parallel using multi-threading on the server side. When YOLO outputs multiple candidate regions, the feature extraction thread and the similarity calculation thread work alternately, reducing the waiting time for single-region processing and improving the overall processing efficiency by 15%-20%, which is suitable for scenarios where multiple targets are recognized simultaneously. The above alternative solutions can all achieve the core objectives of improving target recognition accuracy, ensuring real-time performance, and controlling server resource costs, and do not deviate from the technical concept framework of this invention.

[0077] The embodiments given above are preferred examples for implementing the present invention, and the present invention is not limited to the above embodiments. Any non-essential additions or substitutions made by those skilled in the art based on the technical features of the present invention are within the protection scope of the present invention.

Claims

1. A method of fusing visual recognition and multi-modal target recognition, characterized in that, The method comprises the following steps: 1) constructing a model comprising an image processing module, a fast segmentation module, and a multi-modal recognition module: The image processing module receives image data and pre-processes the image. The pre-processed image data is transmitted to the fast segmentation module. The fast segmentation module receives the pre-processed image data, extracts image features through a convolutional neural network, locates and segments the candidate region in the image that may contain the target object. The multi-modal recognition module receives the candidate region image sub-block output by the fast segmentation module, extracts features from the image sub-block, and generates an image feature vector. By calculating the similarity between the image feature vector and the text feature vector provided by the text description library, the category of the target object is determined, and the recognition result is output. The text description library contains text feature vectors of text descriptions of various target objects. 2) training the multi-modal recognition module using a dataset containing labeled images and using the mini-batch gradient descent method. 3) compressing the image processing module, the fast segmentation module, and the trained multi-modal recognition module based on soft labels and hard labels to generate a lightweight model. Wherein, the soft label transmits inter-class similarity information, and the hard label ensures learning of the true label. 4) deploying the lightweight model to a server or edge device, and using the lightweight model to recognize input images or video frames, and outputting a recognition result containing target object category identification, confidence value, and coordinate information.

2. The method of claim 1, wherein the fusion of visual recognition and multi-modal object recognition is performed by a computer system. The pre-processing performed by the image pre-processing module includes size standardization and pixel value normalization, which compresses the pixel value to 0-1.

3. The method of claim 1, wherein the fusion of visual recognition and multi-modal object recognition is performed by a computer system. The fast segmentation module uses YOLO11 algorithm or Faster R-CNN algorithm. The operation of locating and segmenting the candidate region in the image that may contain the target object is as follows: Using a feature pyramid network to fuse image features at different levels to enhance the perception of different sizes of targets. Generating the bounding box coordinates and confidence of the candidate region through the anchor box mechanism, and selecting the candidate region with a confidence greater than the threshold value. Cutting out the image sub-block corresponding to the selected candidate region as the input of the multi-modal recognition module.

4. The method of claim 3, wherein the fusion of visual recognition and multi-modal object recognition is performed by a neural network. The generation of the bounding box coordinates and confidence of the candidate region is as follows: Pre-setting multiple anchor box sizes based on the scene characteristics of the target object to cover the size range of common target objects. By calculating the intersection over union (IoU) of the anchor box and the target object in the image, selecting the anchor box with IoU>0.5 as the initial candidate region, and then adjusting the coordinates and size of the anchor box through the bounding box regression algorithm to make the anchor box more fit the target object contour, and then outputting the candidate region bounding box coordinates and confidence.

5. The method of claim 3, wherein the fusion of visual recognition and multi-modal object recognition is performed by a computer system. Also based on one or more of the number of candidate regions output by the fast segmentation module, the region overlap rate, and the target area variance to distinguish the complexity and priority of the scene; And based on the pre-set scene complexity-threshold mapping table to dynamically adjust the threshold value; Also according to the recognition result of the candidate region to adjust the resources: Based on the confidence result to allocate computing resources of the multi-modal recognition module; Based on the number of candidate regions to dynamically schedule hardware resources; Based on the scene priority to allocate memory resources.

6. The method of claim 5, wherein the fusion of visual recognition and multi-modal object recognition is performed by a computer system. According to the number of candidate regions, the complexity of the scene is divided into: The scene with less than 5 candidate regions is a low complexity scene, and the threshold is set to 0.75-0.80; The scene with 6-15 candidate regions is a medium complexity scene, and the threshold is set to 0.65-0.70; The scene with more than 15 candidate regions is a high complexity scene, and the threshold is set to 0.55-0.60; The allocation of computing resources of the multi-modal recognition module based on the confidence result is: For high-confidence candidate regions with a confidence greater than 0.8, basic computing resources are allocated; For low-confidence candidate regions with a confidence of 0.55-0.8 and not filtered, enhanced computing resources are allocated; The dynamic scheduling of hardware resources based on the target number is: When the number of candidate regions is less than 8, only CPU resources are called for processing; When the number of candidate regions is greater than 8, GPU auxiliary computing is automatically activated, and for every increase of 4 candidate regions, one additional GPU processing thread is added; The allocation of memory resources based on scene priority is: Reserve part of the memory for core scenes; For ordinary scenes, dynamic memory allocation is used, and the memory is flexibly adjusted according to the demand.

7. The method of claim 1, wherein the method further comprises: The multi-modal recognition module uses a multi-modal recognition module based on the CLIP model or the BLIP model, which, after generating an image feature vector, determines the category of the target object by calculating the cosine similarity between the image feature vector and the text feature vector, and outputs the recognition result containing the category and confidence of the target object; The task of the multi-modal recognition module is also divided into two sub-tasks: generating an image feature vector and similarity calculation, which are processed in multiple threads in parallel.

8. The method of claim 1, wherein the fusion of visual recognition and multi-modal object recognition is performed. The training of the multi-modal recognition module is: A dataset containing 100,000 labeled images is selected for training, and the small batch gradient descent method is used, with the training round set to 50-100 rounds; And linear preheating combined with cosine annealing strategy is used to gradually decay the learning rate to ensure the stability and convergence effect of the training.

9. The method of claim 1, wherein the fusion of visual recognition and multi-modal object recognition is performed by a computer system. The compression includes the following operations: Use the trained multi-modal recognition module as the teacher model, design and build a student model with a parameter quantity of 1 / 4-1 / 5 of the teacher model; Combine the probability distribution output by the teacher model as a soft label with the real class label of the sample as a hard label to jointly train the student model; Lightweight the compressed model: compress the 32-bit floating-point number weight to 4-bit representation, and use 16-bit floating-point precision calculation during inference.

10. The method of claim 6, wherein the fusion of visual recognition and multi-modal object recognition is performed by a computer system. Use model pruning technology instead of knowledge distillation for compression. By calculating the importance score of each layer of neurons, redundant neurons are pruned and core neurons are retained. Then, the model accuracy is restored through fine-tuning; Or, use mixed precision quantization instead of 16-bit full quantization. For multi-modal recognition modules that are sensitive to precision, use 16-bit quantization for convolution layers, and use 8-bit quantization for activation layers; And on the basis of threshold screening, increase the target area threshold; when the area of the candidate region is less than the set area threshold, it is directly determined as a small target and the inference weight of the multi-modal recognition module is automatically increased.

Images