Motor vehicle photo compliance detection method based on image-text multi-mode

Abstract

The invention provides an image-text multi-mode-based motor vehicle photo compliance detection method. The accuracy of motor vehicle photo compliance detection in a complex scene can be effectively improved. According to the method, a lightweight MobileCLIP model is adopted to fuse image visual features and text semantics of detection rules, and end-to-end detection of multiple compliance contents in motor vehicle photos is realized at the same time. Through multi-modal feature fusion and joint training, the problem that a traditional single-modal method is insufficient in semantic understanding is solved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a motor vehicle photo compliance detection method based on image-text multimodality. Background Art

[0002] With the continuous growth in the number of motor vehicles, the volume of motor vehicle inspections is growing. During the motor vehicle registration process, vehicle photos used for registration are subject to compliance checks. These checks include, but are not limited to, checking for the presence of a motor vehicle, background clutter, shooting angle, image quality, vehicle body ratio, contour integrity, screenshot or secondary photo detection, and vehicle body occlusion and portrait detection. In the early days, staff were required to manually review each photo to determine whether it met the relevant standards for motor vehicle photography. This manual inspection method is time-consuming and labor-intensive, and it is difficult to meet the needs of large-scale, real-time business operations. With the advancement of image recognition technology, computer vision detection methods have been applied to motor vehicle photo compliance inspection scenarios.

[0003] However, existing computer vision detection methods have limitations when dealing with complex scenes. For example, models based solely on image features struggle to effectively integrate semantic information (such as the specific definition of the shooting angle), resulting in insufficient accuracy for compliance detection tasks with strong regularity. Furthermore, motor vehicle photos are diverse, and different lighting conditions, shooting equipment, and environmental interference (such as background debris and pedestrians) can affect the generalization ability of the detection model. For example, in cases of overexposure or underexposure, traditional image models may not be able to accurately identify the outline of the vehicle body; when there are complex obstructions in the background, it can easily be misjudged as background clutter. These limitations result in insufficient accuracy for compliance detection of motor vehicle photos in complex scenes. Summary of the Invention

[0004] In order to solve the problem that the accuracy of existing computer vision detection methods is still insufficient when dealing with motor vehicle photo compliance detection in complex scenes, the present invention provides a motor vehicle photo compliance detection method based on image and text multimodality, which can effectively improve the accuracy of motor vehicle photo compliance detection in complex scenes.

[0005] The technical solution of the present invention is as follows: a method for detecting compliance of motor vehicle photos based on multimodality of images and texts, comprising the following steps: S1: Build a compliance detection model; S2: Organize sample data and build training and validation sets based on the sample data; S3: Training the compliance detection model based on the training set to obtain the trained compliance detection model; S4: performing compliance inspection on the registered motor vehicle photo based on the trained compliance inspection model; Its characteristics are: The compliance verification model includes: an input module, a multimodal feature extraction module, a feature fusion module and a fully connected classification head connected in sequence; The multimodal feature extraction module is constructed based on the MobileCLIP model. The image to be detected input by the input module is input into the multimodal feature extraction module, and the multimodal feature extraction module extracts the text feature vector t corresponding to the input image. i , dimension D t , where i is the compliance detection item number, extract the image feature vector v corresponding to the input image, the dimension is D v ; The feature fusion module fuses the image feature vector output by the multimodal feature extraction module and each of the text vectors based on a serial fusion method, and outputs a fused feature vector f i =[v;t i ], dimension D v +D t ; The fully connected classification head includes: a fully connected layer and an activation function, each of the fused feature vectors f i Input the corresponding fully connected classification head respectively, and output the test results corresponding to each compliance test item.

[0006] It is further characterized by: The activation function in the fully connected classification head is implemented based on the softmax function, and the compliance probability is output for each compliance detection item; It also includes the following steps: S5: Reading a preset compliance judgment threshold, and comparing the compliance probability output by the compliance verification model with the compliance judgment threshold; If the test result is greater than the compliance judgment threshold, the corresponding compliance test item is judged to be compliant; Otherwise, it is judged as non-compliant; In step S2, the sample data includes: pre-processed image data and text data corresponding to each image data; The text data is a text prompt containing detection rules generated for each compliance detection; The compliance inspection includes 9 inspection items in total; In step S3, before training the compliance detection model, a task loss function is constructed and joint training is performed for all detection tasks; The loss function is the weighted sum of the cross entropy losses of each detection item, specifically expressed as: ; Among them, Li is the cross entropy loss of the i-th detection task, λ i is the weight coefficient of each task.

[0007] The present application provides a method for compliance detection of motor vehicle photos based on multimodality of images and texts, which uses a lightweight MobileCLIP model to fuse image visual features with the textual semantics of detection rules, thereby realizing end-to-end detection of multiple compliance contents in motor vehicle photos at the same time. Through multimodal feature fusion and joint training, the problem of insufficient semantic understanding of traditional single-modal methods is solved. It has the advantages of high precision, strong real-time performance, and good generalization ability, and is suitable for automated compliance detection in motor vehicle inspection scenarios. Through the joint image and text encoding capability of MobileCLIP, this method deeply fuses the textual semantics of detection rules with image visual features, enhances the ability to understand complex compliance rules, and effectively improves detection accuracy. By processing 9 compliance detection tasks simultaneously with a single model, the complex deployment problem of traditional multi-model combinations is avoided, and the system complexity and computational cost are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is the architecture diagram of the multimodal motor vehicle photo compliance detection model in this application. DETAILED DESCRIPTION

[0009] The present application includes a motor vehicle photo compliance detection method based on image and text multimodality, which includes the following steps.

[0010] S1: Build a compliance detection model.

[0011] Multimodal technology has made significant progress in recent years. Image and text pre-training models represented by CLIP (Contrastive Language-Image Pretraining) can map image and text features to a unified semantic space through contrastive learning, achieving cross-modal semantic alignment. However, the traditional CLIP model has a large number of parameters and is difficult to deploy directly on edge devices or real-time detection scenarios. The MobileCLIP model uses a lightweight design to reduce computational costs while maintaining multimodal semantic understanding capabilities, making end-to-end real-time compliance detection possible. The lightweight multimodal architecture based on MobileCLIP in this application effectively solves the shortcomings of traditional methods in multi-rule semantic understanding and complex scenario generalization capabilities through image and text feature fusion and joint training, and can provide an efficient and accurate automated solution for motor vehicle inspection services. At the same time, the MobileCLIP model has significantly fewer parameters than the traditional CLIP model and can be deployed on edge devices or mobile terminals to achieve end-to-end real-time detection, meeting the processing speed requirements in motor vehicle inspection scenarios.

[0012] like Figure 1 As shown, the compliance verification model in this application includes: an input module, a multimodal feature extraction module, a feature fusion module and a fully connected classification head connected in sequence.

[0013] The multimodal feature extraction module is built based on the MobileCLIP model. The MobileCLIP model consists of a lightweight image encoder (MobileNet architecture) and a text encoder (Transformer architecture). It learns cross-modal semantic alignment through pre-training on a dataset of image and text pairs. After acquiring the MobileCLIP model, the model parameters are initialized and pre-trained weights from a general image and text dataset are loaded. The specific method can be implemented based on existing technologies. The MobileCLIP model loaded with pre-trained weights is then used to train business logic using the training set constructed using this method.

[0014] The image to be detected sent by the input module is sent to the multimodal feature extraction module, and the text encoder extracts the text feature vector t corresponding to the input image i , dimension D t , where i is the compliance detection item number, and the value of i is i∈[1,2,...,9]; the image encoder extracts the image feature vector v corresponding to the input image, with a dimension of D v .

[0015] The feature fusion module combines the image feature vector v output by the multimodal feature extraction module with each text vector t i And based on the serial fusion method, the fusion feature vector f is output i =[v;t i ], dimension D v +D t .

[0016] The fully connected classification head includes: fully connected layer and activation function, which transforms each fusion feature vector f i Each of these inputs is fed into the corresponding fully connected classification head, which outputs the test results for each compliance test item. The activation function in the fully connected classification head is implemented based on the softmax function, which outputs the compliance probability.

[0017] S2: Organize sample data and build training and validation sets based on the sample data; In step S2 , the sample data includes: pre-processed image data and text data corresponding to each image data, where the text data is the test result of all compliance tests for each image data.

[0018] The complex scenarios targeted in this application include a total of 9 compliance checks. Each sample must be annotated with the following 9 check items.

[0019] 1) Motor Vehicle Presence Detection: Accurately distinguish whether a photo contains a motor vehicle requiring registration, to avoid mistakenly identifying a photo without a vehicle or a non-motorized vehicle (such as a bicycle) as a valid photo. Assuming the text data corresponding to the sample data is: "This image contains a bicycle and an electric bicycle," the corresponding detection result is "failed this test." 2) Background clutter detection: In real-world scenes, densely populated billboards, buildings, and other vehicles often interfere with the vehicle's main subject, making it difficult to distinguish it from the surrounding image. Therefore, the complexity of the background must be determined based on the scene's semantics. Assuming the sample data corresponds to text data containing densely populated billboards, the corresponding detection result is a failure. 3) Shooting angle detection: Different car models require different shooting angles. For example, the shooting angle of a passenger car is required to be 45 degrees from the left front. Traditional methods based on image geometric features have difficulty robustly handling the changes in different car models and shooting postures. Assuming that the text data corresponding to the sample data is: the shooting angle is 45 degrees from the left rear, the corresponding detection result is failure of this test; 4) Image quality detection: Overexposure, underexposure, blur, or color cast can affect photo readability, requiring multimodal feature fusion (such as textual descriptions of image quality and image visual features) to improve detection accuracy. Assuming the textual data corresponding to the sample data is: "The image is blurry and cannot be recognized," the corresponding detection result is failure in this test. 5) Vehicle body proportion and outline integrity detection: A vehicle body proportion that is too small may result in missing key vehicle feature information. Therefore, a joint judgment based on semantic rules (e.g., the compliance standard is: "the vehicle body proportion should exceed 2 / 3 of the image") and image features is required. If the text data corresponding to the sample data indicates that the vehicle body proportion is less than 1 / 3, the corresponding test result is failure. If the text data corresponding to the sample data indicates that the vehicle body proportion is 3 / 4 of the image, the corresponding test result is passing. 6) Contour integrity test: An incomplete contour may result in missing key vehicle feature information. Assuming the sample data corresponds to the text data: 1 / 3 of the contour is missing, the corresponding test result is failure; 7) Screenshot or re-photograph detection: These photos often exhibit features such as reduced resolution and moiré patterns, making them difficult for traditional image algorithms to effectively identify. However, multimodal models can improve detection accuracy by learning the relationship between the textual semantics of the screenshot and the corresponding image features. Assume that the textual data corresponding to the sample data is: "This photo is a re-photograph," and the corresponding detection result is "fail this test." 8) Vehicle occlusion: An obstructed vehicle body violates the inspection specifications and requires a joint analysis using the semantic features of the target detection information in the image. Assuming the text data corresponding to the sample data is: vehicle body is unobstructed, the corresponding detection result is a pass. 9) Portrait Detection: The presence of clear portraits in an image violates privacy protection and verification regulations. A joint analysis must be performed using both the target detection information in the image and the semantic features of "clear facial portraits." Assuming the text data corresponding to the sample data is: the image includes two clear portraits, the corresponding test result is failure.

[0020] During implementation, photos of both motor vehicles and non-compliant vehicles are collected, covering a variety of conditions, including vehicle models (cars, motorcycles, trailers), shooting angles, lighting conditions, and background scenes. The sample data consists of no fewer than 60,000 motor vehicle photos (including both compliant and non-compliant samples). For each sample data point, a text prompt containing the detection rules is generated according to similar logic for each detection item. Different images will present different situations and will also correspond to different compliant or non-compliant text labels. The combination of this textual information and image information effectively improves the MobileCLIP model's recognition capabilities for complex scenes. Images are scaled to a uniform input size of 224×224 pixels. Each sample data point is annotated with the ground truth labels for nine detection items.

[0021] The pre-trained MobileCLIP model can capture the semantics of each sentence of text information. The MobileCLIP model is fine-tuned based on the semantic information and image information of the sample data in the training set, making the MobileCLIP model suitable for the business environment used in this application. It can more accurately extract the features of motor vehicle photos in complex scenes, thereby improving the accuracy of subsequent detection results.

[0022] S3: Train the compliance detection model based on the training set to obtain a trained compliance detection model; In step S3, before training the compliance detection model, a task loss function is constructed and joint training is performed for all detection tasks; The loss function is the weighted sum of the cross entropy losses of each detection item, specifically expressed as: ; Among them, L i is the cross entropy loss of the i-th detection task, λ i is the weight coefficient of each task; the specific weight coefficient λ i Set according to the degree of data imbalance. i Adjust the weights of the loss functions for different classes during model training, forcing the model to focus more on the minority class during training and improving the model's generalization ability for imbalanced data. The specific method is based on existing technologies.

[0023] The Adam optimizer is used in the training process, and the initial learning rate is set to 1×10 -4, with a batch size of 32 and end-to-end training. The Adam optimizer performs better for sparse gradients (such as the embedding layer in natural language processing) because the adaptive learning rate amplifies the learning rate of frequently updated parameters and reduces the learning rate of sparsely updated parameters.

[0024] By training on diverse training data (different vehicle models, scenarios, and lighting conditions), the model can effectively cope with complex interference factors in practical applications (such as background clutter and image blur), improving the robustness of detection results.

[0025] S4: During model deployment, the trained compliance verification model is converted to ONNX format and integrated into the business system. Compliance verification is automatically triggered upon uploading a photo to be tested. Based on the trained compliance verification model, compliance verification is performed on the registered vehicle photo. Specifically, for each registered vehicle photo, the compliance verification model outputs a compliance probability for each of the nine test items.

[0026] S5: Reading a preset compliance judgment threshold, and comparing the compliance test output by the compliance verification model with the compliance judgment threshold; If the test result is greater than the compliance judgment threshold, the result is judged as compliance; Otherwise, it is judged as non-compliant.

[0027] A photo of a registered motor vehicle to be monitored is fed into a trained compliance verification model along with a textual template for nine inspection criteria. Based on the compliance judgment results for each inspection criteria, the compliance verification model's detection head outputs a compliance probability for each inspection criteria based on a softmax layer. All these results are combined to form an overall compliance conclusion for the photo. For example, the compliance inspection results for a particular registered motor vehicle photo are: presence of a motor vehicle: 70%; background clutter: 20%; shooting angle: 90%; image quality: 80%; vehicle proportion and outline integrity: 65%; outline integrity: 20%; screenshot or secondary photo: 45%; vehicle occlusion: 90%; and portrait detection: 30%.

[0028] The compliance result for the "Motor Vehicle Presence Detection" item is "The presence of a motor vehicle requiring registration." The model outputs a probability of 70%, indicating a 70% probability of a motor vehicle being present in the image. The default compliance threshold is 50%. If 70% > 50%, the input image data passes the "Motor Vehicle Presence Detection" test.

[0029] For compliant photos, they will automatically pass the inspection; for non-compliant photos, specific non-compliant items (such as "cluttered background", "wrong shooting angle", etc.) will be returned to assist in manual review.

[0030] By using the technical solution of the present invention, end-to-end motor vehicle photo compliance detection is achieved through a lightweight model, thereby improving detection efficiency and accuracy, reducing labor costs, and solving the problems of low efficiency and insufficient accuracy of traditional manual detection methods. This application is based on the lightweight multimodal architecture of MobileCLIP. Through the fusion of graphic and text features and joint training, it effectively solves the shortcomings of traditional methods in multi-rule semantic understanding and complex scenario generalization capabilities, and can provide an efficient and accurate automated solution for motor vehicle inspection services.

Claims

1. A method for detecting compliance of motor vehicle photos based on multimodality of images and text, comprising the following steps: S1: Build a compliance detection model; S2: Organize sample data and build training and validation sets based on the sample data; S3: Training the compliance detection model based on the training set to obtain the trained compliance detection model; S4: performing compliance inspection on the registered motor vehicle photo based on the trained compliance inspection model; Its characteristics are: The compliance verification model includes: an input module, a multimodal feature extraction module, a feature fusion module and a fully connected classification head connected in sequence; The multimodal feature extraction module is constructed based on the MobileCLIP model. The image to be detected input by the input module is input into the multimodal feature extraction module, and the multimodal feature extraction module extracts the text feature vector t corresponding to the input image. i , dimension D t , where i is the compliance detection item number, extract the image feature vector v corresponding to the input image, the dimension is D v ; The feature fusion module fuses the image feature vector output by the multimodal feature extraction module and each of the text vectors based on a serial fusion method, and outputs a fused feature vector f i =[v;t i ], dimension D v +D t ; The fully connected classification head includes: a fully connected layer and an activation function, each of the fused feature vectors f i Input the corresponding fully connected classification head respectively, and output the test results corresponding to each compliance test item.

2. The method for detecting compliance of motor vehicle photos based on multimodal image and text according to claim 1, characterized in that: The activation function in the fully connected classification head is implemented based on the softmax function, and the compliance probability is output for each compliance detection item.

3. The method for detecting compliance of motor vehicle photos based on multimodal image and text according to claim 2, characterized in that: It also The following steps are involved: S5: Reading a preset compliance judgment threshold, and comparing the compliance probability output by the compliance verification model with the compliance judgment threshold; If the test result is greater than the compliance judgment threshold, the corresponding compliance test item is judged to be compliant; Otherwise, it is judged as non-compliant.

4. The method for detecting compliance of motor vehicle photos based on multimodal image and text according to claim 1, characterized in that: In step S2, the sample data includes: pre-processed image data and text data corresponding to each image data; The text data is a text prompt containing detection rules generated for each compliance detection.

5. The method for detecting compliance of motor vehicle photos based on multimodality of images and texts according to claim 1, characterized in that: The compliance inspection includes 9 inspection items in total.

6. The method for detecting compliance of motor vehicle photos based on multimodal image and text according to claim 5, characterized in that: In step S3, before training the compliance detection model, a task loss function is constructed and joint training is performed for all detection tasks; The loss function is the weighted sum of the cross entropy losses of each detection item, specifically expressed as: ； Among them, L i is the cross entropy loss of the i-th detection task, λ i is the weight coefficient of each task.

Images