A configurable intelligent cockpit image visual recognition application method
Through OpenAI's CLIP pre-trained model and Zero-Shot/Few-Shot learning method, the problem of insufficient business data in the smart cockpit image visual recognition application is solved, efficient and accurate image recognition is achieved, and the generalization ability of the model is improved.
Patent Information
- Application Number
- CN202411856983.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-08
AI Technical Summary
The prior art faces the problem of insufficient business data in smart cockpit image visual recognition applications, resulting in poor generalization capabilities of models and difficulty in achieving low-cost reuse and efficient identification.
OpenAI's CLIP pre-trained model is adopted, combined with Zero-Shot and Few-Shot learning methods, to realize specific tasks such as image classification, similar calculation and object detection. By providing a simple requirement description by users, demand expansion and information expansion are automatically generated to generate appropriate data models for inference recognition.
The efficiency and accuracy of smart cockpit image visual recognition is improved, effective recognition can be achieved without a large amount of pre-training data, and the generalization ability of the model is enhanced through Few-Shot fine-tuning.
Smart Images
Figure CN119785328A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of vehicle intelligent configuration, and in particular relates to a configurable intelligent cockpit image visual recognition application method. Background Art
[0002] With the development of smart cockpit technology and the dependence of drivers and passengers on the convenience of the cockpit as the third mobile space, fatigue driving detection, dangerous driving detection, face recognition, personalized vehicle control, etc. have shown increasingly important roles. These technologies are all pointing to the field of image vision. From the advent of AlexNet to ResNet winning the ImageNet competition, new image vision algorithms are constantly being created every day, and visual technologies such as image classification, target detection, image segmentation, and target tracking have penetrated into all aspects of industrial applications. However, when faced with specific business scenarios, we often lack high-quality business data sets of a certain scale to help us train models. Moreover, due to the differences between business demands, artificial data sets that consume resources to annotate for one scenario are difficult to reuse in other scenarios.
[0003] To address the problem of insufficient business data, image enhancement technology can be used to expand the training data set to train the classification model; in addition, there are many image algorithm solutions that are based on some open source data sets (such as ImageNet) to train the base model, and then combine it with actual business data (usually small in magnitude) for transfer learning (fine-tuning). Image enhancement technology (rotation, cropping) relies on the amount and quality of labeled data for data expansion, and the amount of samples that can be expanded is limited; the solution based on pre-training + transfer learning, because of the use of supervisory signals, limits the generalization ability of the model. Taking image classification as an example, when identifying new object categories, new data can only be collected each time and retrained (data in many scenarios is difficult to collect), which is very limited and cannot be reused at a very low cost. The model is pre-trained based on ImageNet, and then transfer learning is performed through manually labeled samples. For samples not covered by manual annotation, the generalization ability is poor (mainly limited by the magnitude of samples). The business scenario data set is expanded through the open source interface reverse annotation solution. Although the scale of the data set can be expanded to a larger level compared to the manual annotation solution, this solution is prone to introduce mislabeled samples, which brings difficulties to model training.
[0004] Invention patent content
[0005] In response to the above problems, the present invention provides a configurable intelligent cockpit image visual recognition application method, which is based on the pre-trained model CLIP of OpenAI using text-image pairs, and uses a combination of Zero-Shot and Few-Shot to implement: image classification, similarity calculation, target detection and other specific tasks. For different business needs, users only need to provide the simplest demand description (configurable), and the application will automatically expand the demand: for example, for cockpit clothing color recognition, users configure the colors to be recognized (eg: red, orange, yellow, green, cyan, blue, and purple), and the application will automatically generate category descriptions based on information such as scenes (cockpits) and clothing categories (T-shirts, shirts, suits, jackets, hoodies, etc.) for building Few-Shot or Zero-Shot reasoning services. The purpose of the present invention is achieved through the following technical solutions:
[0006] The present invention provides a configurable intelligent cockpit image visual recognition application method, which comprises:
[0007] Step S100: Identify user demand information; preferably, step S100 includes:
[0008] Step S110: receiving a user's basic demand description;
[0009] Step S120: Generate user demand information based on the user's basic demand description by performing information expansion. For example, when expanding the demand information, relevant scene information can be added to expand the demand description, such as the scene feature is in the car, the clothing category is shirt, suit or T-shirt, etc. With the help of information expansion, the business demand can be configurable, which reduces the use threshold on the one hand, and the user only needs to know his or her own needs without prior model knowledge to achieve out-of-the-box image vision needs; on the other hand, the demand description can be automatically expanded in combination with the business scenario, which enhances the accuracy of the reasoning service;
[0010] Step S200: judging whether there is corresponding business data according to the user's demand, if the judgment result is yes, jumping to step S4, if the judgment result is no, jumping to step S3;
[0011] Step S300: If there is no corresponding business data, a cockpit data model is constructed through zero-shot learning based on CLIP;
[0012] Step S400: If there is corresponding business data, the cockpit data model is adjusted based on the business data through few-shot learning; preferably, few-shot fine-tuning learning is adopted. In order to avoid the problem of overfitting caused by the limited data used for fine-tuning learning and the large number of parameters in the original CLIP model, a small number of neural network layers and residual connections are added to the CLIP model to perform fine-tuning so as to better generalize the actual task. Specifically, step S400 includes:
[0013] Step S410: reading existing cockpit data model parameters;
[0014] Step S420: Freeze the original parameters, keeping the original Text Encoder and Image Encoder parameters of CLIP unchanged (frozen parameters);
[0015] Step S430: adding additional neural network layers (Additional Layers) for fine-tuning learning;
[0016] Step S440: Add residual connections between the original TextEncoder, Image Encoder and additional neural network layers (AdditionalLayers) to adaptively mix new and old features.
[0017] Step S500: Infer and identify cockpit objects based on the latest cockpit data model.
[0018] Furthermore, the cockpit data model includes a cockpit personnel recognition model and a cockpit behavior recognition model.
[0019] Furthermore, the image encoding of the cockpit data model is implemented by ResNet and / or Vision Transformer, and the text encoding is implemented by Transformer. The model training method is mainly based on contrastive learning, and effective zero-shot migration is performed by linking visual features and natural language features together. The multimodal image-text data used for pre-training comes from the WIT dataset.
[0020] Furthermore, the model performance is evaluated by the cross entropy loss function when building or adjusting the cockpit data model.
[0021] The beneficial effects of the present invention are:
[0022] (1) The image visual recognition application method of the present invention omits the target detection link, greatly improving the recognition efficiency. It does not need to collect a large number of driving pictures in advance for pre-training, but uses multimodal natural language signals to guide the visual model for recognition. It can automatically expand the demand description in combination with the business scenario, thereby enhancing the accuracy of the reasoning service.
[0023] (2) In the solution of the present invention, if there is no collected business image data, zero-shot reasoning can be performed directly. If there is a certain amount of business image data, Few-Shot fine-tuning (a small number of new model parameters) is performed to generate a new model, and then reasoning is performed. Both solutions can achieve an acceptable recognition effect, greatly improving the efficiency of the intelligent cockpit image vision application. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 Shown is a flow chart of a configurable intelligent cockpit image visual recognition application method of the present invention;
[0025] Figure 2 It is a schematic diagram of residual connection when the present invention adopts few-shot fine-tuning learning. DETAILED DESCRIPTION
[0026] The technical scheme of the preferred embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] This embodiment provides a configurable intelligent cockpit image visual recognition application method, such as Figure 1 As shown, the method includes:
[0028] Step S100: Identify user demand information; in a preferred embodiment, step S100 includes:
[0029] Step S110: receiving a user's basic demand description;
[0030] Step S120: Generate user demand information by expanding information according to the user's basic demand description.
[0031] Step S200: judging whether there is corresponding business data according to the user's demand, if the judgment result is yes, jumping to step S4, if the judgment result is no, jumping to step S3;
[0032] Step S300: If there is no corresponding business data, a cockpit data model is constructed through zero-shot learning based on CLIP;
[0033] Step S400: If there is corresponding business data, the cockpit data model is adjusted based on the business data through few-shot learning; in a preferred embodiment, few-shot is used for fine-tuning and the cockpit model data is adjusted through residual connection, such as Figure 2 As shown, specifically, step S400 includes:
[0034] Step S410: reading existing cockpit data model parameters;
[0035] Step S420: Freeze the original parameters, keeping the original TextEncoder and Image Encoder parameters of CLIP unchanged (frozen parameters);
[0036] Step S430: adding additional neural network layers (Additional Layers) for fine-tuning learning;
[0037] Step S440: Add residual connections between the original TextEncoder, Image Encoder and additional neural network layers (AdditionalLayers) to adaptively mix new and old features.
[0038] Step S500: Infer and identify cockpit objects based on the latest cockpit data model.
[0039] In a preferred embodiment, the cockpit data model includes a cockpit personnel identification model and a cockpit behavior identification model, wherein personnel identification includes face recognition authentication, clothing recognition, etc., and the cockpit behavior identification includes dangerous driving behavior identification, fatigue driving behavior identification, etc.
[0040] In a preferred embodiment, the image encoding of the cockpit data model is implemented by ResNet and / or VisionTransformer, and the text encoding is implemented by Transformer. The model training method is mainly based on contrastive learning, and effective zero-shot migration is performed by linking visual features and natural language features together. The multimodal image-text data used for pre-training comes from the WIT dataset. When building or adjusting the cockpit data model, the model performance is evaluated by the cross entropy loss function.
[0041] The method of the present invention is used for cockpit image visual recognition, which omits the target detection link and greatly improves the recognition efficiency. In addition, the method does not need to collect a large number of pictures of driving in the car for pre-training in advance. For example, in the conventional method, if you want to recognize 30 colors of clothing, you must collect pictures of these 30 colors of clothing in advance, which is not easy to do. In the scheme of this embodiment, multimodal natural language signals are used to guide the visual model for recognition. If there is no collected business image data, zero-shot reasoning can be performed directly. If there is a certain amount of business image data, Few-Shot fine-tuning (a small number of new model parameters) is performed to generate a new model, and then reasoning is performed. Both schemes can achieve an acceptable recognition effect, which greatly improves the efficiency of the intelligent cockpit image vision application.
[0042] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention. The dimensions described in the drawings and embodiments are not related to the specific objects and are not used to limit the protection scope of the present invention. The dimensions of the objects can be selected and changed according to actual needs.
Claims
1. A configurable intelligent cockpit image visual recognition application method, comprising: Step S100: identifying user demand information; Step S200: judging whether there is corresponding business data according to the user's demand, if the judgment result is yes, jumping to step S4, if the judgment result is no, jumping to step S3; Step S300: If there is no corresponding business data, a cockpit data model is constructed through zero-sample learning based on CLIP; Step S400: If there is corresponding business data, adjusting the cockpit data model through few-sample learning based on the business data; Step S500: Infer and identify cockpit objects based on the latest cockpit data model.
2. The configurable intelligent cockpit image vision application method according to claim 1, characterized in that: The step S100 includes: Step S110: receiving a user's basic demand description; Step S120: Generate user demand information by expanding information according to the user's basic demand description.
3. The configurable intelligent cockpit image vision application method according to claim 1, characterized in that: In step S400, fewshot is used for fine-tuning and the cockpit model data is adjusted by residual connection.
4. The configurable intelligent cockpit image vision application method according to claim 3 is characterized in that: The step S400 includes: Step S410: reading existing cockpit data model parameters; Step S420: Freeze original parameters; Step S430: adding an additional neural network layer for fine-tuning learning; Step S440: Add a residual connection between the original parameters and the additional neural network layer to adaptively mix new and old features.
5. A configurable intelligent cockpit image vision application method according to any one of claims 1 to 4, characterized in that: The cockpit data model includes a cockpit personnel recognition model and a cockpit behavior recognition model.
6. A configurable intelligent cockpit image visual application method according to any one of claims 1 to 5, characterized in that: The image encoding of the cockpit data model is implemented by ResNet and / or Vision Transformer, and the text encoding is implemented by Transformer.
7. A configurable intelligent cockpit image vision application method according to any one of claims 1 to 6, characterized in that: The cross entropy loss function is used to evaluate model performance when building or adjusting the cockpit data model.