Cockpit image pre-labeling method, system and device based on visual language model and medium

Abstract

The application discloses a cockpit image pre-labeling method, system, device and medium based on a visual language model. The method comprises the following steps: receiving cockpit image data to be labeled, inputting the cockpit image data into a pre-trained first target detection model, a second target detection model and a visual language detection model respectively to obtain first detection result data, second detection result data and third detection result data; performing three-stage fusion processing based on the first detection result data, the second detection result data and the third detection result data to obtain fused detection result data; wherein the three-stage fusion processing comprises main model result screening, auxiliary model missed detection target supplementing and same type detection result deduplication processing; generating standard format pre-labeling file data according to the fused detection result data and outputting the pre-labeling file data. The application achieves the technical effect of improving target detection precision and labeling accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, intelligent cockpit perception and data preprocessing technology, and in particular to a cockpit image pre-annotation method, system, device and medium based on a visual language model. Background Technology

[0002] In intelligent cockpit systems, accurately identifying objects and occupants within the cockpit is crucial for achieving in-vehicle safety monitoring, behavioral warnings, and human-machine interaction. Currently, the industry commonly employs single-object detection models for cockpit image perception and automatic annotation. These models rely on large-scale labeled datasets for end-to-end training and directly output object detection boxes and corresponding category labels during inference, representing the mainstream approach for cockpit visual perception and data annotation.

[0003] However, when single-target detection models are applied to cockpit scenarios, they face numerous unavoidable technical challenges. Firstly, the cockpit contains a large number of small targets such as cigarettes, small bottles, and the edges of mobile phones. These targets are small in size and have weak texture features. Combined with complex conditions such as backlighting, nighttime infrared imaging, and target occlusion, they are highly susceptible to missed detections and false detections. Secondly, due to their inherent architectural limitations, conventional detection models struggle to maintain consistent accuracy and stability in detecting and recognizing small targets under complex imaging and interference scenarios.

[0004] Secondly, the images captured by the cockpit include various types such as visible light color images and infrared night vision black and white images, and the visual features of the same target differ significantly under different imaging modalities. If a unified model is used to adapt to multimodal images, the detection performance will be significantly reduced in some modalities; if independent models are trained for different image modalities, the number of models will increase, increasing the complexity of system deployment, operation and maintenance.

[0005] Furthermore, although there are attempts in the industry to combine multiple detection models, the fusion methods used are relatively simple, mostly involving weighted operations or conventional non-maximum suppression. These methods fail to fully leverage the technical advantages of different architectures, making it difficult for the models to effectively complement each other. This results in limited improvement in detection performance and a higher risk of generating additional false detection results.

[0006] Meanwhile, the initial labeled data output by existing automatic labeling schemes contains a lot of noise, and the problem of missing labels for small targets is particularly prominent. A lot of manpower must be invested in manual review and correction, which not only increases the overall cost of data labeling, but also lengthens the dataset production cycle, which is not conducive to the rapid construction and iterative optimization of large-scale cockpit image datasets.

[0007] Some existing technologies have also attempted to introduce visual language models to assist in object detection. However, these solutions often use local feature descriptions or general prompts in the text input stage, which cannot take into account the characteristics of the cockpit scene and thus cannot fully utilize the model's capabilities, resulting in less than ideal improvement in detection performance. Summary of the Invention

[0008] This invention provides a cockpit image pre-annotation method, system, device, and medium based on a visual language model, to achieve high-precision detection of small targets in cockpit images, adaptive processing of multimodal images, and to improve the quality of automatic annotation and reduce the cost of manual annotation.

[0009] According to one aspect of the present invention, a cockpit image pre-annotation method based on a visual language model is provided, the method comprising: Receive cockpit image data to be labeled, and input the cockpit image data into a pre-trained first target detection model, a second target detection model, and a visual language detection model respectively to obtain first detection result data, second detection result data, and third detection result data; Based on the first detection result data, the second detection result data, and the third detection result data, a three-stage fusion process is performed to obtain the fused detection result data; wherein, the three-stage fusion process includes main model result screening, auxiliary model missed target supplementation, and deduplication of similar detection results; Based on the fused detection results data, a pre-labeled file data in a standard format is generated and output.

[0010] According to another aspect of the present invention, a cockpit image pre-annotation system based on a visual language model is provided, the system comprising: The model detection module is used to receive cockpit image data to be labeled, and input the cockpit image data into a pre-trained first target detection model, a second target detection model and a visual language detection model respectively to obtain first detection result data, second detection result data and third detection result data; The fusion processing module is used to perform a three-stage fusion processing based on the first detection result data, the second detection result data, and the third detection result data to obtain fused detection result data; wherein, the three-stage fusion processing includes main model result filtering, auxiliary model missed target supplementation, and deduplication of similar detection results; The annotation file generation module is used to generate pre-annotated file data in a standard format based on the fused detection result data, and output the pre-annotated file data.

[0011] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and memory that is communicatively connected to at least one processor; The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to execute the cockpit image pre-annotation method based on a visual language model according to any embodiment of the present invention.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the cockpit image pre-annotation method based on a visual language model according to any embodiment of the present invention.

[0013] The technical solution of this invention solves the problems of low accuracy of single model in detecting small targets, poor fusion effect of multi-model, weak multimodal image adaptation, and high manual cost of automatic annotation by inputting cockpit image data into three types of models in parallel to obtain multiple sets of detection results, performing three-stage fusion processing on multiple sets of detection results, and regenerating and outputting standard format pre-annotated files. It achieves the beneficial effects of giving full play to the complementary advantages of different models, reducing missed detections, false detections and duplicate annotations, improving scene generalization ability, reducing manual workload, and being easy to deploy and highly compatible.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a cockpit image pre-annotation method based on a visual language model, provided in an embodiment of the present invention; Figure 2a A flowchart illustrating another cockpit image pre-annotation method based on a visual language model provided in this embodiment of the invention; Figure 2b A fusion processing flowchart of a cockpit image pre-annotation method based on a visual language model provided in an embodiment of the present invention; Figure 3 A schematic diagram of a cockpit image pre-annotation system based on a visual language model provided in an embodiment of the present invention; Figure 4A schematic diagram of the structure of an electronic device for implementing a cockpit image pre-annotation method based on a visual language model, according to an embodiment of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Figure 1 This flowchart illustrates a cockpit image pre-annotation method based on a visual language model, provided in this embodiment of the invention. This method is applicable to scenarios involving intelligent cockpit visual perception and automated annotation of in-vehicle image data, particularly suitable for cockpit image recognition and annotation in situations with small targets, target occlusion, and complex lighting. It is also adaptable to data processing of various imaging modalities, including visible light color images and infrared night vision black and white images. Furthermore, it can be used for large-scale cockpit dataset construction, iterative training of in-vehicle target detection models, and in-vehicle driving behavior safety monitoring. This method can be executed by a cockpit image pre-annotation system based on a visual language model. This device can be implemented in hardware and / or software, and the system can be configured in an electronic device. Figure 1 As shown, the method specifically includes the following steps: S110. Receive the cockpit image data to be labeled, and input the cockpit image data into the pre-trained first target detection model, second target detection model and visual language detection model respectively to obtain the first detection result data, the second detection result data and the third detection result data.

[0020] Cockpit image data can be understood as image data collected for smart cockpit scenarios, including visual content such as passengers and in-vehicle items.

[0021] The first object detection model can be understood as a pre-trained detection model used to identify target objects within an image. For example, in this embodiment, the first object detection model is the YOLOv10 model. The second object detection model can be understood as another detection model used for image object recognition, with a different architecture and characteristics than the first object detection model. For example, in this embodiment, the second object detection model is the DETR model. The visual language detection model can be understood as a multimodal model that integrates visual perception and language understanding capabilities, using text assistance to complete image object detection. For example, in this embodiment, the visual language detection model is the multimodal DINO model. The first detection result data can be understood as the target recognition result output by the first object detection model for the cockpit image data. The second detection result data can be understood as the target recognition result output by the second object detection model for the cockpit image data. The third detection result data can be understood as the target recognition result output by the visual language detection model for the cockpit image data.

[0022] Specifically, the system receives cockpit image data to be labeled from an external source, and then feeds the same cockpit image data in parallel into three pre-trained target detection models: a first target detection model, a second target detection model, and a visual-language detection model. The three models perform target recognition operations on the image content respectively, and finally output the corresponding first, second, and third detection result data in sequence, completing multi-model parallel inference and result output. The first, second, and third detection result data all contain target category information, confidence scores representing target credibility, and bounding box coordinates used to select the target's position and size. After the three models complete their operations, they output the corresponding first, second, and third detection result data in sequence, completing multi-model parallel inference and result output.

[0023] Optionally, before inputting the cockpit image data into the visual language detection model, the method further includes: Global text description data corresponding to the cockpit image data is generated based on a large language model, and the cockpit image data and the global text description data are input together into the visual language detection model.

[0024] Among these, the large language model can be understood as an artificial intelligence model with text generation and semantic understanding capabilities. Global text description data can be understood as text description data generated based on the overall content of the entire cockpit image.

[0025] Specifically, before inputting cockpit image data into the visual language detection model, the system first calls the large language model and generates corresponding global text description data based on the overall content of the current cockpit image. Then, the cockpit image data and the generated global text description data are input into the visual language detection model together. The text information is used to assist the visual language detection model in completing target recognition and enhance the model's detection capability in cockpit scenarios.

[0026] S120. Perform a three-stage fusion process based on the first detection result data, the second detection result data, and the third detection result data to obtain the fused detection result data.

[0027] The three-stage fusion process includes main model result screening, auxiliary model target omission supplementation, and deduplication of similar detection results. The three-stage fusion process can be understood as a data processing flow that integrates, screens, and optimizes multiple sets of detection results in three sequential steps. Main model result screening can be understood as the operation of selecting valid detection data based on the output of one detection model. Auxiliary model target omission supplementation can be understood as the operation of supplementing target data not identified by the main model using the results of other detection models. Deduplication of similar detection results can be understood as the operation of removing duplicate target annotation data generated by multiple model detections. The fused detection result data can be understood as the final target detection data obtained after the three-stage fusion process.

[0028] Specifically, the system retrieves the first, second, and third detection result data obtained in the previous steps and performs a three-stage fusion process. The processing flow is divided into three stages: main model result screening, auxiliary model missing target supplementation, and deduplication of similar detection results. According to the established rules, the three sets of detection results are screened, supplemented, and deduplicated layer by layer to eliminate defects in single models and redundant data caused by parallel detection of multiple models, and finally generate the integrated detection result data.

[0029] Optionally, the main model result filtering is performed based on the first detection result data, the second detection result data, and the third detection result data, including: using the first detection result data as the benchmark detection data, determining the first target detection box data among the detection box data included in the first detection result data, whose intersection-union ratio (IU) with the detection box data in the second detection result data or the third detection result data reaches a first preset IU threshold; determining the second detection box data whose IU with the detection box data included in the first detection result data does not reach the first preset IU threshold, but whose self-confidence reaches a first preset confidence threshold; retaining the first target detection box data and the second target detection box data, and discarding the remaining detection box data.

[0030] In this context, the baseline detection data can be understood as the main detection result data used as the basis for selection and judgment. The detection box data can be understood as the coordinate data of the target location within the image output by the model. The intersection-over-union ratio (IoU) can be understood as a calculated metric measuring the degree of overlap between two detection boxes. The first preset IoU threshold can be understood as a pre-set value used to determine whether the overlap of the detection boxes meets the standard. The first target detection box data can be understood as valid detection box data whose overlap degree meets the requirements. The first preset confidence threshold can be understood as a pre-set value used to determine the credibility of the target. The second detection box data can be understood as valid detection box data whose overlap degree does not meet the standard but whose own credibility meets the standard. Confidence score can be understood as the confidence score of the model in determining the actual existence of the target.

[0031] Specifically, using the first detection result data as the baseline detection data, all detection box data contained in this data are traversed. First, the first target detection box data that has an intersection-union ratio (IU) of 1 / 2 with the detection box data in the second or third detection result data reaches a first preset IU threshold is selected. Then, the second detection box data that has an IU ratio that does not reach the first preset IU threshold but whose own confidence level reaches a first preset confidence threshold is selected. The first preset IU threshold and the first preset confidence threshold can be preset based on experience, and this embodiment does not impose specific restrictions on them. Finally, the first target detection box data and the second detection box data are retained, and all other unqualified detection box data are removed, completing the main model result selection.

[0032] For example, a first preset intersection-union (IU) threshold is set to 0.6, and a first preset confidence threshold is set to 0.7. All detection frames in the first detection result data are iterated. For detection frame A, which marks a mobile phone target in the cockpit, the IU with the corresponding target detection frames in the second detection result data is calculated to be 0.72, which is greater than 0.6. Therefore, detection frame A is identified as the first target detection frame and retained. Next, detection frame B, which marks a beverage bottle, is selected from the first detection result data. Its IU with the corresponding detection frames in the other two sets of detection results is only 0.45, which does not reach the threshold of 0.6. However, the confidence level of this detection frame is 0.81, which is higher than the first preset confidence threshold of 0.7. Therefore, detection frame B is identified as the second detection frame and retained. For detection frame C, which marks debris, its IU with the other detection frames is 0.38, and its confidence level is only 0.52. Both indicators do not meet the requirements, so this detection frame is directly discarded. The first preset intersection-union ratio threshold and the first preset confidence threshold can be preset based on experience, and this embodiment does not impose specific restrictions on them.

[0033] Optionally, supplementing the missed detection targets of the auxiliary model based on the first detection result data, the second detection result data, and the third detection result data includes: extracting the remaining detection box data that did not participate in the main model result screening from the second detection result data and the third detection result data; if the intersection-union ratio of the remaining detection box data of the second detection result data and the remaining detection box data of the third detection result data reaches a second preset intersection-union ratio threshold, and if the average confidence of the two sets of detection box data reaches a second preset confidence threshold, or the confidence of any set of detection box data reaches a third preset confidence threshold, then the corresponding detection box data is supplemented into the already retained detection box data set.

[0034] The remaining detection box data can be understood as the auxiliary model detection box data that was not included in the main model result filtering and was retained. The second preset intersection-union threshold can be understood as the pre-set detection box overlap judgment value in the omission filling stage. The average confidence score can be understood as the average of the confidence scores of the two sets of detection boxes. The second preset confidence threshold can be understood as the pre-set average confidence judgment value in the omission filling stage. The third preset confidence threshold can be understood as the pre-set confidence judgment value of a single detection box in the omission filling stage. The detection box dataset can be understood as the collection of all valid detection box data retained after filtering.

[0035] Specifically, the remaining detection box data that did not participate in the main model result screening are first extracted from the second and third detection result data. The two sets of remaining detection box data are compared one by one. When the intersection-union ratio (IU) of the two sets reaches a second preset IU threshold, it is determined whether the average confidence level of the two sets of data reaches a second preset confidence threshold, or whether the confidence level of any one set of detection box data reaches a third preset confidence threshold. The second preset IU threshold, the second preset confidence threshold, and the third preset confidence threshold can be preset based on experience; this embodiment does not impose specific restrictions on them. When any condition is met, the set of detection box data is added to the already retained detection box data set, thereby completing the supplementation of missed targets by the auxiliary model.

[0036] For example, a second preset intersection-over-union (IoU) threshold is set to 0.5, a second preset confidence threshold to 0.65, and a third preset confidence threshold to 0.8. The remaining detection boxes that did not participate in the previous round of main model screening are extracted from the second and third detection result data, including two sets of detection boxes targeting cigarettes inside the cabin. The IoU of the two sets of detection boxes is calculated to be 0.58, meeting the second preset IoU threshold requirement. In the first case: the confidence levels of the two sets of detection boxes are 0.66 and 0.68 respectively, and the calculated average confidence level is 0.67, exceeding the second preset confidence threshold. Therefore, this set of detection boxes is added to the already retained detection box dataset. In the second case: the confidence levels of the two sets of detection boxes are 0.71 and 0.82 respectively, with the latter reaching the third preset confidence threshold. Similarly, this set of detection boxes is also added. If the cross-union ratio (CUP) of the two sets of detection frames is 0.42, which does not meet the second preset CUP threshold, or if the CUP meets the standard but the average confidence score and the confidence score of a single set do not meet the threshold requirements, then no supplementary operation will be performed. The second preset CUP threshold, the second preset confidence score threshold, and the third preset confidence score threshold can be preset based on experience, and this embodiment does not impose specific restrictions on them.

[0037] Optionally, a three-stage fusion process is performed based on the first detection result data, the second detection result data, and the third detection result data to obtain fused detection result data, including: grouping all the filtered and supplemented detection box data according to the target category; performing non-maximum suppression processing on the detection box data under the same category to remove duplicate detection box data, thereby obtaining fused detection result data.

[0038] Here, the target category can be understood as the classification of objects to be identified within the cabin, such as bottles, cigarettes, and mobile phones. Non-maximum suppression processing can be understood as a conventional data processing method that removes duplicate detection boxes for the same type of target.

[0039] Specifically, all detection bounding box data, after being filtered by the main model and supplemented by the auxiliary model, are grouped and categorized according to their corresponding target categories to ensure centralized processing of detection bounding boxes within the same category. Non-maximum suppression is then performed on the detection bounding box data within each group to automatically remove duplicate detection bounding boxes generated for the same target, ultimately resulting in fused detection results data that is free of redundancy and has well-organized content.

[0040] For example, after filtering and filling in the gaps, all existing detection box data are first grouped according to target category, specifically into eight categories: mobile phones, cigarettes, beverage bottles, water cups, keys, glasses, car accessories, and driver / passenger hands. Taking the mobile phone category as an example, there are three detection boxes in this group, all corresponding to the same mobile phone target in the cabin, which are duplicate labels. Non-maximum suppression is performed on this group of detection boxes, and based on the intersection-union ratio and confidence level of the detection boxes, the detection box with the highest confidence is retained, and the other two duplicate detection boxes are removed. Taking the beverage bottle category as another example, there are two highly overlapping detection boxes in this category pointing to the same beverage bottle. After performing non-maximum suppression, only the optimal detection box is retained. The same processing is performed on the detection box data of the other categories such as cigarettes, water cups, keys, glasses, car accessories, and driver / passenger hands, removing duplicate detection boxes of the same type of target one by one, and finally obtaining the fused detection result data after removing redundant labels and data normalization.

[0041] S130. Generate standard format pre-labeled file data based on the fused detection result data, and output the pre-labeled file data.

[0042] Pre-annotated file data can be understood as file data generated in a standard format and used for image annotation work.

[0043] Specifically, based on the fused detection results data obtained in the previous step, the system generates corresponding standard format pre-annotated file data according to the general annotation specifications, and then outputs the pre-annotated file data for direct use in subsequent downstream steps such as data verification and model training.

[0044] Optionally, the output pre-annotated file data is YOLO format file data, which includes target category number data, target normalized coordinate data, and confidence score data; the cockpit image data includes color image data and black-and-white infrared image data.

[0045] Among them, the YOLO format (YouOnlyLookOnce, the standard format for object detection) can be understood as a commonly used image annotation file format in the field of computer vision. Target category ID data can be understood as data using numerical codes to represent different target object categories. Target normalized coordinate data can be understood as coordinate data that has been normalized and used to identify the target's position and size. Color image data can be understood as color cockpit images acquired under normal visible light conditions. Black and white infrared image data can be understood as black and white cockpit images acquired using infrared devices at night or in low-light environments.

[0046] Specifically, the pre-annotated file data output by this method is uniformly in YOLO format. This file contains three core types of content: target category IDs for distinguishing object types, normalized coordinates for determining target location and size, and confidence scores representing target credibility. Furthermore, this method is compatible with two mainstream cockpit image data sources: it can process color image data from everyday scenarios and adapt to black-and-white infrared image data acquired by nighttime infrared devices, covering the full-scene imaging needs of the cockpit.

[0047] For example, this embodiment pre-defines and numbers common targets within the cockpit, such as assigning category number 0 to mobile phones, category number 1 to cigarettes, category number 2 to beverage bottles, category number 3 to water cups, and category number 4 to glasses. After completing the detection of a cockpit color image data, an example of a single valid annotation in the output YOLO format file data is: 00.2150.3620.1280.1940.92. The first digit 0 represents the target category number, and the sequentially arranged 0.215, 0.362, 0.128, and 0.194 are the target normalized coordinate data, representing the normalized values ​​of the target's center point's horizontal and vertical coordinates, as well as the target's width and height, respectively. The last digit 0.92 is the confidence score for that target. This solution can process color image data collected under normal lighting conditions and is also compatible with black and white infrared image data collected by the device at night or in low-light environments. It can generate YOLO format pre-annotated file data with a unified format and standardized content for both types of images.

[0048] The technical solution of this invention solves the problems of low accuracy of single model in detecting small targets, poor fusion effect of multi-model, weak multimodal image adaptation, and high manual cost of automatic annotation by inputting cockpit image data into three types of models in parallel to obtain multiple sets of detection results, performing three-stage fusion processing on multiple sets of detection results, and regenerating and outputting standard format pre-annotated files. It achieves the beneficial effects of giving full play to the complementary advantages of different models, reducing missed detections, false detections and duplicate annotations, improving scene generalization ability, reducing manual workload, and being easy to deploy and highly compatible.

[0049] Figure 2a This is a flowchart of another cockpit image pre-annotation method based on a visual language model provided in this embodiment of the invention. Based on the above embodiments, this embodiment is a further refinement of the above embodiments, and its specific implementation can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2a As shown, the method specifically includes the following steps: S101. Collect multi-type cockpit image sample data and divide the multi-type cockpit image sample data into a training subset and a validation subset.

[0050] The multi-type cockpit image sample data can be understood as the original cockpit image samples containing different imaging styles used for model training. The training subset can be understood as a dataset specifically used for iterative model learning, partitioned from the sample data. The validation subset can be understood as a dataset specifically used to verify the model training effect, partitioned from the sample data. The labeled sample data can be understood as image samples that have been manually labeled with information such as target category and location.

[0051] Specifically, firstly, a sufficient amount of multi-type cockpit image sample data is collected. Combining the sample characteristics and usage requirements, the overall sample data is divided into a training subset and a validation subset. The two subsets are independent of each other and respectively undertake the training and validation functions.

[0052] S102. Input the labeled sample data into the first object detection model, the second object detection model, and the visual language detection model to be trained respectively for iterative training. Use the validation subset to verify the model performance until the model converges, and obtain each pre-trained model.

[0053] Iterative training can be understood as a cyclical training process in which the model continuously learns sample features and optimizes parameters. Model convergence can be understood as the state where the model training effect tends to stabilize and performance no longer improves significantly.

[0054] Specifically, all labeled sample data are input into the first object detection model, the second object detection model, and the visual language detection model for iterative training. During training, a validation subset is continuously used to verify the model's recognition performance, and the model's effect is monitored in real time until the model reaches convergence. At this point, the training process is stopped, and the three types of models are finally obtained and ready for direct use.

[0055] The technical solution of this invention employs a multi-type sample-based approach to divide the training and validation subsets for model training. This allows the model to fully learn target features under different scenarios and imaging modalities, effectively improving the model's generalization ability and recognition stability in cockpit scenarios. The iterative training combined with performance verification ensures model training quality, avoiding overfitting and performance issues, and guaranteeing accurate and reliable detection results from subsequent model inference. By completing the training of all models in advance, the models can be directly called during the subsequent actual annotation stage without temporary training, simplifying the online deployment process and improving the overall efficiency of the annotation work.

[0056] Figure 2b This is a flowchart illustrating the fusion processing of a cockpit image pre-annotation method based on a visual language model, as provided in an embodiment of the present invention. Figure 2bAs shown, in this embodiment, the output of the first object detection model (Model A) is the primary factor, while the outputs of the second object detection model (Model B) and the visual language detection model (Model C) are secondary factors. A three-stage fusion process is performed on the three sets of detection result data. The specific process is as follows: Phase 1: Filtering the results of the main model (taking the intersection) Using the detection results of model A as a benchmark, iterate through each detection box: If the intersection-union ratio between the detection box of model A and the detection box of model B or model C reaches a preset threshold, it is determined to be a reliable target, and the detection box is directly added to the fusion result set. If the detection box of model A has no valid intersection with the detection boxes of models B and C, then it is further determined whether the confidence level of the detection box itself reaches the preset confidence threshold: If the confidence level meets the standard, the detection box is retained; If the confidence level does not meet the standard, it is considered a low-confidence target and is eliminated.

[0057] This stage preserves both the goal of consistent results between the main model and the auxiliary model, as well as the independent targets with high confidence in the main model, thus forming a set of basic and effective detection results.

[0058] Phase Two: Supplementing Missed Targets in the Auxiliary Model (Union Set) From the detection results of Model B and Model C, the remaining detection boxes that did not participate in the first stage screening are extracted, and cross-matching is performed on the remaining detection boxes of Model B and Model C: For each remaining detection box in model B, find matching boxes in the remaining detection boxes of model C whose intersection-union ratio reaches a preset threshold. If a matching box is found, calculate the average confidence score of the two sets of detection boxes: If the average confidence level reaches the preset average confidence level threshold, or if the confidence level of any detection box reaches the preset high confidence level threshold, then the detection box (coordinates can be the average value) corresponding to the target is added to the fusion result set. If no matching box is found, it is considered an invalid supplementary target and is not included in the fusion result.

[0059] This stage utilizes the collaborative validation of two auxiliary models to supplement the targets missed by the main model, effectively reducing the overall false negative rate.

[0060] Phase 3: Deduplication of similar detection results (NMS (Non-Maximum Suppression) deduplication, optional) All detection bounding box data obtained from Phase 1 and Phase 2 are grouped according to target category; non-maximum suppression processing is performed on the detection bounding box data under the same category, for example, setting the intersection-union ratio threshold to 0.5 to remove duplicate or redundant detection bounding box data for the same target, and finally obtaining regular and non-redundant fused detection result data.

[0061] After completing the above three-stage processing, the final fused detection result can be output, which can be used to generate pre-annotated file data in a standard format.

[0062] The technical solution of this invention effectively overcomes the shortcomings of single-model detection of small targets and the limited gain of multi-model fusion by using the main model results as a benchmark and supplementing them with dual-model collaborative verification and hierarchical deduplication. Specifically, Stage 1 uses intersection and high-confidence dual screening to retain consensus targets from both the main and auxiliary models while eliminating low-confidence noise, ensuring the reliability of the basic results. Stage 2 utilizes cross-matching and confidence verification of the two auxiliary models to efficiently supplement targets missed by the main model without introducing additional false positives, significantly reducing the overall false negative rate. Stage 3 eliminates duplicate annotations generated by parallel detection of multiple models by performing non-maximum suppression processing by category, making the final results more regular and usable. The overall process fully leverages the complementary advantages of the three types of models without significantly increasing computational overhead, achieving a dual improvement in detection accuracy and recall, and providing a reliable data foundation for the subsequent generation of high-quality pre-labeled files.

[0063] Figure 3 This is a schematic diagram of a cockpit image pre-annotation system based on a visual language model, provided as an embodiment of the present invention. Figure 3 As shown, the device includes: a model detection module 310, a fusion processing module 320, and a annotation file generation module 330.

[0064] The model detection module 310 receives cockpit image data to be labeled, and inputs the cockpit image data into a pre-trained first target detection model, a second target detection model, and a visual language detection model to obtain first detection result data, second detection result data, and third detection result data. The fusion processing module 320 performs a three-stage fusion processing based on the first detection result data, the second detection result data, and the third detection result data to obtain fused detection result data. The three-stage fusion processing includes main model result filtering, auxiliary model missed target supplementation, and deduplication of similar detection results. The annotation file generation module 320 generates standard format pre-annotated file data based on the fused detection result data and outputs the pre-annotated file data.

[0065] The technical solution of this invention solves the problems of low accuracy of single model in detecting small targets, poor fusion effect of multi-model, weak multimodal image adaptation, and high manual cost of automatic annotation by inputting cockpit image data into three types of models in parallel to obtain multiple sets of detection results, performing three-stage fusion processing on multiple sets of detection results, and regenerating and outputting standard format pre-annotated files. It achieves the beneficial effects of giving full play to the complementary advantages of different models, reducing missed detections, false detections and duplicate annotations, improving scene generalization ability, reducing manual workload, and being easy to deploy and highly compatible.

[0066] Optionally, the system further includes: The global text generation module is used to generate global text description data corresponding to the cockpit image data based on a large language model before inputting the cockpit image data into the visual language detection model, and then input the cockpit image data and the global text description data into the visual language detection model together.

[0067] Optionally, the fusion processing module 320 includes: The first screening unit is used to use the first detection result data as the benchmark detection data to determine the first target detection box data among the detection box data included in the first detection result data, whose intersection-union ratio with the detection box data in the second detection result data or the third detection result data reaches a first preset intersection-union ratio threshold. The second filtering unit is used to determine the second detection box data whose intersection-union ratio (IU) with the detection box data in the first detection result data and the detection box data in the second detection result data or the third detection result data does not reach the first preset IU threshold, but whose confidence level reaches the first preset confidence threshold. The third filtering unit is used to retain the first target detection box data and the second target detection box data, and to remove the remaining detection box data.

[0068] Optionally, the fusion processing module 320 includes: The detection box extraction unit is used to extract the remaining detection box data that did not participate in the main model result filtering from the second detection result data and the third detection result data; The target supplementation unit is used to supplement the corresponding detection box data into the already retained detection box data set when the intersection-union ratio of the remaining detection box data of the second detection result data and the remaining detection box data of the third detection result data reach the second preset intersection-union ratio threshold, and the average confidence of the two sets of detection box data reaches the second preset confidence threshold, or the confidence of any set of detection box data reaches the third preset confidence threshold.

[0069] Optionally, the fusion processing module 320 includes: The category grouping unit is used to group all the filtered and supplemented detection box data according to the target category; The deduplication unit is used to perform nonmaximum suppression on the detection bounding box data of the same category, remove duplicate detection bounding box data, and obtain the fused detection result data.

[0070] Optionally, the system further includes: The sample acquisition module is used to acquire multiple types of cockpit image sample data before inputting the cockpit image data into the pre-trained first target detection model, second target detection model and visual language detection model, and to divide the multiple types of cockpit image sample data into training subset and validation subset; The model training module is used to input the labeled sample data into the first object detection model, the second object detection model, and the visual language detection model to be trained for iterative training. The model performance is verified using a validation subset until the model converges, and each pre-trained model is obtained.

[0071] Optionally, the output pre-annotated file data is YOLO format file data, which includes target category number data, target normalized coordinate data, and confidence score data; the cockpit image data includes color image data and black-and-white infrared image data.

[0072] The cockpit image pre-annotation system based on visual language model provided in this embodiment of the invention can execute the cockpit image pre-annotation method based on visual language model provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0073] Figure 4 This is a schematic diagram of the structure of an electronic device for implementing the cockpit image pre-annotation method based on a visual language model, as described in this embodiment of the invention. The electronic device is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0074] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0075] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as cockpit image pre-annotation based on a visual language model.

[0077] In some embodiments, the method for cockpit image pre-annotation based on a visual language model can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for cockpit image pre-annotation based on a visual language model described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for cockpit image pre-annotation based on a visual language model by any other suitable means (e.g., by means of firmware).

[0078] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0082] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0083] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0084] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A cockpit image pre-annotation method based on a visual language model, characterized in that, include: Receive cockpit image data to be labeled, and input the cockpit image data into a pre-trained first target detection model, a second target detection model, and a visual language detection model respectively to obtain first detection result data, second detection result data, and third detection result data; Based on the first detection result data, the second detection result data, and the third detection result data, a three-stage fusion process is performed to obtain the fused detection result data; wherein, the three-stage fusion process includes main model result screening, auxiliary model missed target supplementation, and deduplication of similar detection results; Based on the fused detection results data, a pre-labeled file data in a standard format is generated and output.

2. The method according to claim 1, characterized in that, Before inputting the cockpit image data into the visual language detection model, the method further includes: Global text description data corresponding to the cockpit image data is generated based on a large language model, and the cockpit image data and the global text description data are input together into the visual language detection model.

3. The method according to claim 1, characterized in that, Based on the first detection result data, the second detection result data, and the third detection result data, the main model result filtering is performed, including: Using the first detection result data as the benchmark detection data, determine the first target detection box data among the detection box data included in the first detection result data that has an intersection-union ratio of up to a first preset intersection-union ratio threshold with the detection box data in the second detection result data or the third detection result data; The second detection frame data is determined to have a confidence level that reaches a first preset confidence level, but the intersection-union ratio (IUU) of each detection frame data included in the first detection result data with the detection frame data in the second detection result data or the third detection result data does not reach a first preset IUU threshold. The first target detection box data and the second target detection box data are retained, and the remaining detection box data are discarded.

4. The method according to claim 1, characterized in that, Based on the first detection result data, the second detection result data, and the third detection result data, supplementary detection targets are added using the auxiliary model, including: Extract the remaining detection box data that did not participate in the main model result filtering from the second detection result data and the third detection result data; If the intersection-union ratio (IUU) of the remaining detection box data of the second detection result data and the remaining detection box data of the third detection result data reaches the second preset IUU threshold, and if the average confidence of the two sets of detection box data reaches the second preset confidence threshold, or if the confidence of any set of detection box data reaches the third preset confidence threshold, then the corresponding detection box data will be added to the set of retained detection box data.

5. The method according to claim 3, characterized in that, A three-stage fusion process is performed based on the first detection result data, the second detection result data, and the third detection result data to obtain fused detection result data, including: All detected bounding box data, after filtering and supplementation, are grouped according to the target category; Non-maximum suppression is performed on the detection bounding box data of the same category to remove duplicate detection bounding box data, and the fused detection result data is obtained.

6. The method according to claim 1, characterized in that, Before inputting the cockpit image data into the pre-trained first object detection model, second object detection model, and visual language detection model, the process also includes: Collect cockpit image sample data of multiple types, and divide the multi-type cockpit image sample data into training subset and validation subset; The labeled sample data is input into the first object detection model, the second object detection model, and the visual language detection model to be trained for iterative training. The performance of the models is verified by using a validation subset until the models converge, and each pre-trained model is obtained.

7. The method according to claim 1, characterized in that, The output pre-annotated file data is YOLO format file data, which includes target category number data, target normalized coordinate data, and confidence score data; the cockpit image data includes color image data and black and white infrared image data.

8. A cockpit image pre-annotation system based on a visual language model, characterized in that, include: The model detection module is used to receive cockpit image data to be labeled, and input the cockpit image data into a pre-trained first target detection model, a second target detection model and a visual language detection model respectively to obtain first detection result data, second detection result data and third detection result data; The fusion processing module is used to perform a three-stage fusion processing based on the first detection result data, the second detection result data, and the third detection result data to obtain fused detection result data; wherein, the three-stage fusion processing includes main model result filtering, auxiliary model missed target supplementation, and deduplication of similar detection results; The fusion processing module is used to generate pre-labeled file data in a standard format based on the fused detection result data, and output the pre-labeled file data.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the cockpit image pre-annotation method based on a visual language model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the cockpit image pre-annotation method based on a visual language model as described in any one of claims 1-7.

Images