Vehicle attribute recognition method, device, electronic equipment and medium

Abstract

The present application provides a vehicle attribute recognition method, device, electronic device and medium, the method comprising: in response to an attribute recognition instruction issued by a user for a vehicle to be recognized, obtaining a vehicle image and vehicle attribute query information of the vehicle to be recognized, inputting the vehicle image and vehicle attribute query information into a multimodal model for processing to obtain a first vehicle attribute, inputting the vehicle image and vehicle attribute query information into the multimodal model again for processing to obtain a second vehicle attribute, if the first vehicle attribute is different from the second vehicle attribute, inputting the vehicle image into a picture recognition model for processing to obtain at least one third vehicle attribute, and then determining the target vehicle attribute, and displaying the vehicle image and the target vehicle attribute as the attribute recognition result of the vehicle to be recognized to the user. The present application can improve the accuracy and diversity of attribute recognition, determine vehicle attributes according to user query information, realize interactive output, and improve user experience.

Description

Technical Field

[0001] The present application relates to the field of vehicle identification, and in particular to a method, device, electronic equipment and medium for identifying vehicle attributes. Background Art

[0002] With the rapid development of today's automobile industry, vehicle attributes such as vehicle style, vehicle type, and vehicle color are becoming more and more diverse. Identifying vehicle attributes has become an important part of improving user experience. However, existing vehicle attribute recognition methods have significant limitations in attribute extraction.

[0003] First, the vehicle attributes extracted by the existing vehicle attribute recognition methods are relatively single, which limits the diversity and flexibility of the attributes. Second, the existing vehicle attribute recognition methods have poor results in extracting attributes of new models and are not effective when processing unseen vehicle data. In addition, the existing recognition methods cannot achieve interactive output and cannot provide dynamic, user-involved feedback. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application propose a vehicle attribute recognition method, device, electronic device and medium.

[0005] In a first aspect of the implementation of the present application, a method for identifying attributes of a vehicle is first provided, the method comprising:

[0006] In response to an attribute recognition instruction issued by a user for a vehicle to be recognized, acquiring a vehicle image of the vehicle to be recognized and vehicle attribute query information issued by the user for the vehicle to be recognized;

[0007] Inputting the vehicle image and the vehicle attribute query information into a pre-trained multimodal model for processing, and outputting a first vehicle attribute corresponding to the vehicle to be identified;

[0008] Inputting the vehicle image and the vehicle attribute query information into the pre-trained multimodal model again for processing, and outputting a second vehicle attribute corresponding to the vehicle to be identified;

[0009] In the case where the first vehicle attribute is different from the second vehicle attribute, inputting the vehicle image into a pre-trained image recognition model for processing, and outputting at least one third vehicle attribute corresponding to the vehicle to be identified;

[0010] Determining a target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information;

[0011] The vehicle image and the target vehicle attributes are presented to the user as attribute recognition results of the vehicle to be identified.

[0012] Optionally, the method further comprises:

[0013] In the case where the first vehicle attribute is the same as the second vehicle attribute, the first vehicle attribute or the second vehicle attribute is determined as the target vehicle attribute corresponding to the vehicle to be identified.

[0014] Optionally, determining a target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information includes:

[0015] Determine the target vehicle attribute category according to the vehicle attribute query information;

[0016] Determine the third vehicle attribute corresponding to the target vehicle attribute category from at least one of the third vehicle attributes, and use the third vehicle attribute corresponding to the target vehicle attribute category as a fourth vehicle attribute;

[0017] combining the fourth vehicle attribute with the vehicle attribute inquiry information to form attribute confirmation information;

[0018] The vehicle image and the attribute confirmation information are input into the pre-trained multimodal model for processing, and the target vehicle attributes corresponding to the vehicle to be identified are output.

[0019] Optionally, the image recognition model is trained in the following manner:

[0020] Obtaining a picture recognition model training sample and a picture recognition model verification sample; the picture recognition model training sample includes a sample vehicle image and a sample vehicle standard attribute; the picture recognition model verification sample includes a verification vehicle image;

[0021] The sample vehicle image is used as the input of the image recognition model, the sample vehicle standard attribute is used as the output of the image recognition model, and the image recognition model is trained using the image recognition model training sample to obtain the image recognition model to be verified; the sample vehicle standard attribute includes at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color;

[0022] Input the verification vehicle image into the image recognition model to be verified, and obtain the sample vehicle verification attribute output by the image recognition model to be verified; the sample vehicle verification attribute includes at least one of the verification vehicle image area, verification sample vehicle style, verification sample vehicle type, and verification sample vehicle color;

[0023] Calculating a loss function corresponding to the image recognition model according to the sample vehicle standard attributes and the sample vehicle verification attributes;

[0024] The loss function corresponding to the image recognition model is used to adjust the model parameters of the image recognition model to be verified, and the steps of taking the sample vehicle image as the input of the image recognition model, taking the sample vehicle standard attributes as the output of the image recognition model, and using the image recognition model training samples to train the image recognition model are re-executed until a preset first training stop condition is reached to obtain a trained image recognition model.

[0025] Optionally, obtaining a picture recognition model training sample and a picture recognition model verification sample includes:

[0026] Obtain a sample video shot for a sample vehicle;

[0027] Segmenting the sample video to obtain a plurality of sample video frames, and using the sample video frames containing the sample vehicles as the sample vehicle images and the verification vehicle images;

[0028] Using the verification vehicle image as a verification sample of the picture recognition model;

[0029] Taking the image region containing the sample vehicle in the sample vehicle image as the sample vehicle image region;

[0030] Determine at least one of the sample vehicle style, the sample vehicle type, and the sample vehicle color corresponding to the sample vehicle according to the sample vehicle image;

[0031] Checking whether the sample vehicle image matches at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, and the sample vehicle color;

[0032] If they match, at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, and the sample vehicle color is used as the sample vehicle standard attribute;

[0033] The sample vehicle image and the sample vehicle standard attributes are used as training samples for the image recognition model.

[0034] Optionally, the image recognition model includes at least one residual structure, a compressed excitation structure, and at least one fully connected layer, and the step of inputting the verification vehicle image into the image recognition model to be verified to obtain the sample vehicle verification attribute output by the image recognition model to be verified includes:

[0035] Inputting the verification vehicle image into the residual structure for processing to obtain initial image features output by the residual structure;

[0036] Inputting the initial image features into the compression excitation structure for processing to obtain target image features output by the compression excitation structure;

[0037] The target image features are input into the fully connected layer for processing to obtain the sample vehicle verification attributes.

[0038] Optionally, the multimodal model is trained in the following manner:

[0039] Acquire a multimodal model training sample and a multimodal model verification sample; the multimodal model training sample includes a sample vehicle image, sample attribute query information, and a sample vehicle response attribute; the multimodal model verification sample includes a verification vehicle image and verification attribute query information;

[0040] Taking the sample vehicle image and the sample attribute query information as inputs of the multimodal model, taking the sample vehicle reply attributes as outputs of the multimodal model, and training the multimodal model using the multimodal model training samples to obtain a multimodal model to be verified;

[0041] Inputting the verification vehicle image and the verification attribute query information into the multimodal model to be verified, and obtaining the verification vehicle reply attribute output by the multimodal model to be verified;

[0042] Calculating a loss function corresponding to the multimodal model according to the sample vehicle response attributes and the verification vehicle response attributes;

[0043] The loss function corresponding to the multimodal model is used to adjust the model parameters of the multimodal model to be verified, and the steps of taking the sample vehicle image and the sample attribute query information as the input of the multimodal model, taking the sample vehicle response attributes as the output of the multimodal model, and training the multimodal model using the multimodal model training samples are re-executed until a preset second training stop condition is reached to obtain a trained multimodal model.

[0044] Optionally, the obtaining of multimodal model training samples and multimodal model verification samples includes:

[0045] Determine the sample vehicle position according to the position of the center point of the sample vehicle image area in the sample video frame;

[0046] Determining a sample vehicle sales volume and a sample vehicle evaluation corresponding to the sample vehicle according to the sample vehicle image;

[0047] Determine the sample attribute query information and the verification attribute query information according to at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle orientation, the sample vehicle sales volume, and the sample vehicle evaluation;

[0048] Using the verification vehicle image and the verification attribute query information as verification samples of the multimodal model;

[0049] At least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle position, the sample vehicle sales volume, and the sample vehicle evaluation corresponding to the sample attribute query information is used as a sample vehicle response attribute;

[0050] The sample vehicle image, the sample attribute query information, and the sample vehicle response attributes are used as training samples for the multimodal model.

[0051] Optionally, the multimodal model includes an image encoder, a text encoder, a feature fusion structure, and a decoder, and the step of inputting the verification vehicle image and the verification attribute query information into the multimodal model to be verified to obtain the verification vehicle reply attribute output by the multimodal model to be verified includes:

[0052] Inputting the verification vehicle image into the image encoder for processing to obtain image coding features output by the image encoder;

[0053] Inputting the verification attribute query information into the text encoder for processing to obtain text encoding features output by the text encoder;

[0054] Inputting the image coding feature and the text coding feature into the feature fusion structure for processing, and obtaining the text image splicing feature output by the feature fusion structure;

[0055] The text image splicing features are input into the decoder for processing to obtain the verification vehicle reply attributes.

[0056] In a second aspect of the implementation of the present application, there is also provided a vehicle attribute recognition device, the device comprising:

[0057] A data acquisition module, for responding to an attribute recognition instruction issued by a user for a vehicle to be recognized, and acquiring a vehicle image of the vehicle to be recognized and vehicle attribute inquiry information issued by the user for the vehicle to be recognized;

[0058] A first vehicle attribute acquisition module, configured to input the vehicle image and the vehicle attribute query information into a pre-trained multimodal model for processing, and output a first vehicle attribute corresponding to the vehicle to be identified;

[0059] A second vehicle attribute acquisition module, used for inputting the vehicle image and the vehicle attribute query information into the pre-trained multimodal model for processing again, and outputting a second vehicle attribute corresponding to the vehicle to be identified;

[0060] A third vehicle attribute acquisition module, configured to input the vehicle image into a pre-trained image recognition model for processing and output at least one third vehicle attribute corresponding to the vehicle to be identified when the first vehicle attribute is different from the second vehicle attribute;

[0061] A target vehicle attribute acquisition module, used to determine a target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information;

[0062] The recognition result acquisition module is used to display the vehicle image and the target vehicle attributes as the attribute recognition result of the vehicle to be identified to the user.

[0063] In the third aspect of the implementation of the present application, there is also provided an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the vehicle attribute identification method as described above is implemented.

[0064] In a fourth aspect of the implementation of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the vehicle attribute identification method as described above is implemented.

[0065] The embodiments of the present application have the following advantages:

[0066] In an embodiment of the present application, in response to an attribute recognition instruction issued by a user for a vehicle to be recognized, a vehicle image of the vehicle to be recognized and vehicle attribute query information issued by the user for the vehicle to be recognized are obtained, the vehicle image and the vehicle attribute query information are input into a pre-trained multimodal model for processing, and a first vehicle attribute corresponding to the vehicle to be recognized is output. The vehicle image and the vehicle attribute query information are processed by the multimodal model, and the accuracy and diversity of attribute recognition can be improved. Then the vehicle image and the vehicle attribute query information are input into the pre-trained multimodal model again for processing, and a second vehicle attribute corresponding to the vehicle to be recognized is output, and further verification is performed by the multimodal model to ensure the reliability of the attribute recognition result. In the case where the first vehicle attribute is different from the second vehicle attribute, the vehicle image is input into a pre-trained picture recognition model for processing, and at least one third vehicle attribute corresponding to the vehicle to be recognized is output. When the attribute recognition results output by the multimodal model twice are inconsistent, the picture recognition model is introduced for supplementary recognition, and the robustness and accuracy of recognition can be enhanced even in the face of new models. According to the vehicle attribute query information, the target vehicle attribute corresponding to the vehicle to be recognized is determined in at least one third vehicle attribute, and the vehicle image and the target vehicle attribute are displayed to the user as the attribute recognition result of the vehicle to be recognized. The target vehicle attributes can be determined based on the user's query information, achieving interactive output with user participation and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0068] Figure 1 This is a schematic diagram of a picture classification process provided by an embodiment of the present application;

[0069] Figure 2 It is a structural schematic diagram of a vehicle attribute recognition process provided by an embodiment of the present application;

[0070] Figure 3 It is a flowchart of a method for obtaining vehicle attribute recognition results provided by an embodiment of the present application;

[0071] Figure 4 It is a structural schematic diagram of an image recognition model provided by an embodiment of the present application;

[0072] Figure 5 This is a schematic diagram of a process for obtaining training samples provided by an embodiment of the present application;

[0073] Figure 6 It is a structural schematic diagram of a multimodal model provided in one embodiment of the present application;

[0074] Figure 7 It is a structural diagram of a fusion model provided by an embodiment of the present application;

[0075] Figure 8 It is a structural schematic diagram of a vehicle attribute recognition device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0076] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not constitute any limitation, and the various embodiments can be combined with each other and referenced to each other without contradiction.

[0077] When performing vehicle attribute recognition, the relevant technology can be based on deep learning and use a single image for classification training. First, a large amount of image data needs to be collected and labeled, and then standardized and data enhanced. Then, a convolutional neural network (CNN) or other feature extraction network is used to extract features from the image, and a deep learning model containing deep learning network layers such as convolution layer, attention layer, residual network layer, and fully connected layer is designed. When training the model, select the appropriate regression loss function and optimizer, and improve performance through hyperparameter tuning. In this way, the vehicle attributes can be predicted as accurately as possible when the information of a single image is limited.

[0078] Although the relevant technology is flexible, it is very ineffective when faced with image data that has not been seen or trained. Since the supervised training model has a data gap problem for new data, the untrained image data will appear random during the prediction stage. Especially when faced with the attribute recognition of new models, it is impossible to get the desired correct results through simple post-processing. As a result, the attribute recognition model in the relevant technology has poor robustness and generalization ability for vehicle attribute recognition, and is prone to overfitting problems, and cannot provide reliable prediction effects in practical applications. In order to obtain good results, it is necessary to annotate the attribute recognition model with new data and further fine-tune the training.

[0079] Some improvements can also be made based on the above-mentioned related technologies. In the task of training vehicle attribute classification, a feature head layer can be added. The attribute recognition model can directly obtain the vehicle-related attribute categories in the inference stage. It can also extract features of corresponding dimensions for unseen vehicles through the feature head layer, and then calculate the similarity with the features already saved in the training sample database to obtain the vehicle attribute category. This model structure can simply deal with zero-shot situations.

[0080] Although this method adds a feature head extraction layer and can obtain more accurate vehicle attribute information when facing unknown vehicle data by calculating the similarity between the data in the training sample database and the unknown vehicle data, the feature data in the training sample database will not be very rich and diverse, while the unknown vehicle data will be diverse and will encounter various situations, which also causes the attribute recognition model to still encounter robustness and accuracy problems.

[0081] In a specific implementation, the related technology can first input a training image, then perform feature extraction on the training image to obtain training features, and use the training image and its corresponding training features to perform model training. After the model training is completed, the input single image can be processed by the trained model to output the corresponding features, which can be directly mapped to the task target type to obtain the vehicle attribute classification result, or the similarity can be calculated based on the features output by the trained model and the features already saved in the training sample database to obtain the vehicle attribute category.

[0082] However, the methods in the related art cannot aggregate the results obtained, and are obtained in the form of questions and answers, which has no interactive performance and cannot enable users to obtain vehicle attributes in the form of questions and answers. It is no longer in line with the trend and is relatively backward. In order to improve the accuracy and interactivity of prediction, it is necessary to explore more advanced recognition technologies.

[0083] Reference Figure 1 , showing a step flow chart of a vehicle attribute identification method provided by an embodiment of the present application.

[0084] In the embodiment of the present application, the vehicle attribute recognition method can be specifically implemented by a cloud server.

[0085] It should be noted that cloud servers can refer to virtual servers that provide computing resources and services through the Internet. Cloud servers are usually managed and maintained by cloud service providers (such as Amazon Web Services, Google Cloud, Microsoft Azure, etc.). Users can access these servers through the Internet and dynamically adjust computing resources according to demand.

[0086] Among them, the attributes of the vehicle may include but are not limited to vehicle style, vehicle type, vehicle color, vehicle location, vehicle sales, vehicle evaluation, etc.

[0087] Among them, the vehicle attribute recognition can be to identify the vehicle style, vehicle type, vehicle color, vehicle location, vehicle sales, vehicle evaluation and other information of the vehicle to be identified.

[0088] The method may specifically include the following steps:

[0089] S101: In response to an attribute recognition instruction issued by a user for a vehicle to be recognized, a vehicle image of the vehicle to be recognized and vehicle attribute query information issued by the user for the vehicle to be recognized are acquired.

[0090] It should be noted that the attribute identification instruction may be an instruction issued by a user to instruct vehicle attribute identification. The attribute identification instruction may be issued by a user in a vehicle through a vehicle computer, or may be issued by a user in a vehicle identification application on a variety of mobile terminals. In the embodiment of the present application, the need for attribute identification is determined by the attribute identification instruction.

[0091] The vehicle image may be an image containing the vehicle to be identified, may be a cropped image containing only the vehicle to be identified, may be an image captured from various videos, or may be an image captured from a video in front of the vehicle captured in real time by a driving recorder. In the embodiment of the present application, vehicle attributes may be identified based on the vehicle image.

[0092] The vehicle attribute query information may be a specific attribute query issued by the user for the vehicle to be identified, such as "What is the brand of this car?" or "What is the color of this car?" In the embodiment of the present application, the user's intention, that is, the vehicle attribute information desired by the user, may be determined based on the vehicle attribute query information.

[0093] In the embodiment of the present application, in response to the attribute recognition instruction issued by the user for the vehicle to be recognized, the vehicle image of the vehicle to be recognized and the vehicle attribute query information issued by the user for the vehicle to be recognized can be obtained. The vehicle attribute query information can be in the form of text directly input by the user, or in the form of text obtained by converting the voice directly input by the user.

[0094] S102: Input the vehicle image and the vehicle attribute query information into a pre-trained multimodal model for processing, and output a first vehicle attribute corresponding to the vehicle to be identified.

[0095] It should be noted that a multimodal model may refer to a machine learning model that can process and fuse multiple different types of data (such as images, text, audio, etc.). By combining data of different modalities, a multimodal model can extract richer features, thereby improving the performance and accuracy of the model. In an embodiment of the present application, a vehicle image and vehicle attribute query information can be processed by a multimodal model to obtain vehicle attributes.

[0096] In an embodiment of the present application, the vehicle image and vehicle attribute query information may be input into a pre-trained multimodal model for processing, and a first vehicle attribute corresponding to the vehicle to be identified may be output.

[0097] It should be noted that the vehicle image region may refer to the portion of the vehicle used for identification and analysis in the image containing the vehicle. The vehicle image region generally includes the front, rear, side, roof, etc. of the vehicle. By identifying and analyzing the vehicle image region, characteristic information of the vehicle may be extracted.

[0098] Vehicle style can refer to the make and / or model of a vehicle, such as a certain model of a certain brand.

[0099] The vehicle type may refer to a basic classification of vehicles, such as a sedan, SUV, MPV, off-road vehicle, pickup truck, truck, engineering vehicle, tanker, and others. The vehicle type information is used to distinguish different types of vehicles.

[0100] Vehicle color may refer to the color of the vehicle exterior, such as red, green, white, yellow, silver, gray, blue, black, purple, brown, other, etc. Vehicle color information is used to distinguish vehicles of different colors.

[0101] Vehicle orientation can refer to the direction and position of the vehicle in the image, such as the front, left front, right front, etc. of the vehicle in the image. Vehicle orientation information is used to determine the position and direction of the vehicle in the image, helping the model to identify and analyze different parts of the vehicle.

[0102] Vehicle sales can refer to the number of vehicles sold in the market. Vehicle sales information is used to analyze the popularity and market performance of vehicles.

[0103] Vehicle evaluation can refer to the evaluation of vehicle performance, quality, safety, etc. Vehicle evaluation information is used to evaluate the performance and quality of the vehicle.

[0104] In the embodiment of the present application, the first vehicle attribute may be at least one of the attribute information of the vehicle image area, vehicle style, vehicle type, vehicle color, vehicle position, vehicle sales, vehicle evaluation, etc. of the vehicle to be identified output by the multimodal model. The first vehicle attribute may also be a vehicle attribute expressed in the form of a reply sentence.

[0105] S103: inputting the vehicle image and the vehicle attribute query information into the pre-trained multimodal model again for processing, and outputting a second vehicle attribute corresponding to the vehicle to be identified.

[0106] In an embodiment of the present application, the vehicle image and vehicle attribute query information can be input into the pre-trained multimodal model again for processing, and the second vehicle attribute corresponding to the vehicle to be identified can be output. The second vehicle attribute can be at least one of the attribute information such as the vehicle image area, vehicle style, vehicle type, vehicle color, vehicle position, vehicle sales, and vehicle evaluation of the vehicle to be identified output by the multimodal model, and the second vehicle attribute is obtained based on the same basis as the first vehicle attribute, that is, the second vehicle attribute and the first vehicle attribute are two results obtained by inputting the same vehicle image and the same vehicle attribute query information into the multimodal model twice. The second vehicle attribute can also be a vehicle attribute expressed in the form of a reply sentence.

[0107] S104: When the first vehicle attribute is different from the second vehicle attribute, the vehicle image is input into a pre-trained image recognition model for processing, and at least one third vehicle attribute corresponding to the vehicle to be identified is output.

[0108] The image recognition model can be an edge model used to identify vehicle attributes based on vehicle images. The edge model can refer to a small machine learning model that can run on edge devices (such as smartphones, embedded systems, IoT devices, etc.). These models usually have a small size and low computational complexity, and can run efficiently on resource-constrained devices. In an embodiment of the present application, when the output result of the multimodal model is unreliable, the image recognition model can be used to identify the vehicle image.

[0109] In an embodiment of the present application, when the first vehicle attribute is different from the second vehicle attribute, the vehicle image can be input into a pre-trained image recognition model for processing, and at least one third vehicle attribute corresponding to the vehicle to be identified can be output. The first vehicle attribute is different from the second vehicle attribute, which can mean that the first vehicle attribute and the second vehicle attribute have different meanings. In this case, it can be considered that the result output by the multimodal model is not reliable enough. In addition, the third vehicle attribute can be at least one of the vehicle style, vehicle type, vehicle color, and vehicle image area of ​​the vehicle to be identified output by the image recognition model. The third vehicle attribute can also be a vehicle attribute expressed in the form of a label.

[0110] S105: Determine a target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information.

[0111] In an embodiment of the present application, a target vehicle attribute corresponding to the vehicle to be identified can be determined from at least one third vehicle attribute according to the vehicle attribute query information, wherein the target vehicle attribute can refer to the vehicle attribute of the vehicle to be identified that the user expects to obtain corresponding to the vehicle attribute query information.

[0112] S106: Displaying the vehicle image and the target vehicle attributes as attribute recognition results of the to-be-recognized vehicle to the user.

[0113] In an embodiment of the present application, the vehicle image and the target vehicle attributes can be displayed to the user as the attribute recognition result of the vehicle to be identified. Among them, the display to the user can be displayed in the user's vehicle through the vehicle computer on the display screen inside the vehicle, or it can be displayed in the vehicle identification application of the terminal device. It can also be displayed in the form of voice broadcast, or combined with head-up display technology to display the vehicle attribute information on the front windshield of the vehicle. It should be noted that Head-Up Display (HUD) can refer to a technology that projects key driving information (such as vehicle speed, navigation instructions, warning information, etc.) onto the front windshield of the vehicle. This technology is designed to reduce the driver's line of sight diversion and improve driving safety and comfort.

[0114] In the specific implementation, you can refer to Figure 2 The process of realizing vehicle attribute recognition, Figure 2 A structural schematic diagram of a vehicle attribute recognition process provided by an embodiment of the present application is shown.

[0115] The structure of the vehicle attribute recognition process is as follows: Figure 2 As shown, it mainly includes three parts

[0116] Data part: Image data and text data will be used in the training stage. In the embodiment of the present application, the image data and text data can be training samples corresponding to the image recognition model and the multimodal large model.

[0117] Model part: supervised training of image recognition model and fine-tuning of multimodal model based on labeled data;

[0118] Reasoning part: After the image recognition model and the multimodal large model are trained, the question is input, the results of the image recognition model and the multimodal large model are fused, and the interactive results are output.

[0119] The present application obtains a vehicle image of the vehicle to be identified and vehicle attribute query information issued by the user for the vehicle to be identified in response to an attribute recognition instruction issued by the user for the vehicle to be identified, inputs the vehicle image and the vehicle attribute query information into a pre-trained multimodal model for processing, and outputs a first vehicle attribute corresponding to the vehicle to be identified. The vehicle image and the vehicle attribute query information are processed by the multimodal model to improve the accuracy and diversity of attribute recognition. Then the vehicle image and the vehicle attribute query information are input into the pre-trained multimodal model for processing again, and the second vehicle attribute corresponding to the vehicle to be identified is output, and further verification is performed by the multimodal model to ensure the reliability of the attribute recognition result. In the case where the first vehicle attribute is different from the second vehicle attribute, the vehicle image is input into the pre-trained picture recognition model for processing, and at least one third vehicle attribute corresponding to the vehicle to be identified is output. When the attribute recognition results output by the multimodal model twice are inconsistent, the picture recognition model is introduced for supplementary recognition, which can enhance the robustness and accuracy of recognition even in the face of new models. According to the vehicle attribute query information, the target vehicle attribute corresponding to the vehicle to be identified is determined in at least one third vehicle attribute, and the vehicle image and the target vehicle attribute are displayed to the user as the attribute recognition result of the vehicle to be identified. The target vehicle attributes can be determined based on the user's query information, achieving interactive output with user participation and improving the user experience.

[0120] In an optional embodiment of the present application, the method further includes the following steps:

[0121] S11: When the first vehicle attribute is the same as the second vehicle attribute, determining the first vehicle attribute or the second vehicle attribute as a target vehicle attribute corresponding to the vehicle to be identified.

[0122] In the embodiment of the present application, when the first vehicle attribute is the same as the second vehicle attribute, the first vehicle attribute or the second vehicle attribute can be determined as the target vehicle attribute corresponding to the vehicle to be identified. The first vehicle attribute being the same as the second vehicle attribute can mean that the first vehicle attribute and the second vehicle attribute have the same meaning. In this case, the result output by the multimodal model can be considered to be sufficiently reliable.

[0123] In this application, when the first vehicle attribute is the same as the second vehicle attribute, the result output by the multimodal model can be considered to be sufficiently reliable and determined as the target vehicle attribute corresponding to the vehicle to be identified. This method ensures that the output vehicle attributes are consistent through two independent multimodal model inferences, thereby improving the reliability of the results, reducing the errors that may be introduced by a single inference, enhancing the robustness of the system, avoiding misjudgments caused by inaccurate single inference results, and improving user experience and system performance.

[0124] In an optional embodiment of the present application, step S105 further includes the following sub-steps:

[0125] S21: Determine the attribute category of the target vehicle according to the vehicle attribute query information;

[0126] S22: determining the third vehicle attribute corresponding to the target vehicle attribute category from at least one of the third vehicle attributes, and using the third vehicle attribute corresponding to the target vehicle attribute category as the fourth vehicle attribute;

[0127] S23: combining the fourth vehicle attribute with the vehicle attribute inquiry information to form attribute confirmation information;

[0128] S24: Input the vehicle image and the attribute confirmation information into the pre-trained multimodal model for processing, and output the target vehicle attributes corresponding to the vehicle to be identified.

[0129] It should be noted that the target vehicle attribute category may be the category of the vehicle attribute corresponding to the vehicle attribute query information. In the embodiment of the present application, the target vehicle attribute category may reflect the category of the vehicle attribute that the user expects to obtain.

[0130] The attribute confirmation information may be text information obtained by splicing the vehicle attributes and the vehicle attribute query information, and the spliced ​​text may be, for example, "[Vehicle model] Please answer based on the above content: What kind of car is this?" In the embodiment of the present application, the attribute confirmation information may be input into the multimodal model to obtain response information that is more in the form of question and answer.

[0131] In an embodiment of the present application, the target vehicle attribute category can be determined according to the vehicle attribute query information, the third vehicle attribute corresponding to the target vehicle attribute category can be determined in at least one third vehicle attribute, the third vehicle attribute corresponding to the target vehicle attribute category can be used as the fourth vehicle attribute, the fourth vehicle attribute and the vehicle attribute query information can be combined to form attribute confirmation information, the vehicle image and the attribute confirmation information can be input into a pre-trained multimodal model for processing, and the target vehicle attribute corresponding to the vehicle to be identified can be output. Among them, the target vehicle attribute and the fourth vehicle attribute have the same meaning, but the target vehicle attribute and the fourth vehicle attribute may be expressed differently. In an optional embodiment of the present application, the target vehicle attribute can be a reply sentence that reflects the fourth vehicle attribute.

[0132] In an embodiment of the present application, a user in a vehicle cabin can ask questions about the properties of a vehicle in a scene outside the vehicle, thereby interactively outputting various properties of the vehicle.

[0133] The present application determines the target vehicle attribute category according to the vehicle attribute query information, determines the third vehicle attribute corresponding to the target vehicle attribute category in at least one third vehicle attribute, uses it as the fourth vehicle attribute, combines the fourth vehicle attribute with the vehicle attribute query information to form attribute confirmation information, inputs the vehicle image and the attribute confirmation information into a pre-trained multimodal model for processing, and outputs the target vehicle attribute corresponding to the vehicle to be identified. By combining the fourth vehicle attribute with the vehicle attribute query information to form the attribute confirmation information, it is ensured that the target vehicle attribute output by the multimodal model is consistent with the user query information, and the expression method of the target vehicle attribute can be optimized.

[0134] In the specific implementation, you can refer to Figure 3 Realize the acquisition of vehicle attribute recognition results, Figure 3 A schematic diagram of a process for obtaining vehicle attribute recognition results provided in an embodiment of the present application is shown.

[0135] The vehicle attribute recognition result can be obtained in the inference stage of the entire data processing process. It is a process of integrating the results of the multimodal model and the image recognition model. The specific steps are as follows: when the user inputs a picture and asks questions such as: "What kind of car is this?" "What is the sales volume of this car?", etc. First, the multimodal model will obtain a first vehicle attribute. In order to ensure the reliability of the first vehicle attribute, it will make a reflection judgment on the first vehicle attribute. If the answer is: yes, it will go to the branch of target detection in the image recognition model, obtain the box of the vehicle, and output them together; if the answer is: no, it means that the first result is not very credible, and the image recognition model will be called for attribute recognition. The final result is the fourth vehicle attribute. Then the fourth vehicle attribute is spliced ​​using the following template: {Fourth vehicle attribute} Please answer according to the above content: What kind of car is this, input the multimodal model, obtain the vehicle target attribute, and then go to the branch of target detection in the image recognition model to obtain the vehicle image, and output the vehicle target attribute and the vehicle image in json format. And the same vehicle image can correspond to multiple vehicle target attributes for output.

[0136] In an optional embodiment of the present application, the image recognition model is trained in the following manner:

[0137] S31: Acquire a picture recognition model training sample and a picture recognition model verification sample; the picture recognition model training sample includes a sample vehicle image and a sample vehicle standard attribute; the picture recognition model verification sample includes a verification vehicle image;

[0138] S32: using the sample vehicle image as the input of the image recognition model, using the sample vehicle standard attribute as the output of the image recognition model, and using the image recognition model training sample to train the image recognition model to obtain the image recognition model to be verified; the sample vehicle standard attribute includes at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color;

[0139] S33: Input the verification vehicle image into the image recognition model to be verified, and obtain the sample vehicle verification attribute output by the image recognition model to be verified; the sample vehicle verification attribute includes at least one of the verification vehicle image area, verification sample vehicle style, verification sample vehicle type, and verification sample vehicle color;

[0140] S34: Calculating a loss function corresponding to the image recognition model according to the sample vehicle standard attribute and the sample vehicle verification attribute;

[0141] S35: Use the loss function corresponding to the image recognition model to adjust the model parameters of the image recognition model to be verified, and re-execute the steps of taking the sample vehicle image as the input of the image recognition model, taking the sample vehicle standard attributes as the output of the image recognition model, and using the image recognition model training sample to train the image recognition model, until the preset first training stop condition is reached, thereby obtaining a trained image recognition model.

[0142] It should be noted that the image recognition model training samples may refer to image data sets used to train the image recognition model. These samples usually contain a large amount of image data, and each image has a corresponding label to indicate the category to which the image belongs. The quality and quantity of the training samples directly affect the performance and accuracy of the model. In an embodiment of the present application, the image recognition model training samples may include sample vehicle images and sample vehicle standard attributes.

[0143] The image recognition model verification sample may refer to an image data set used to verify the image recognition model. The image recognition model verification sample may include a verification vehicle image. In an embodiment of the present application, the verification vehicle image is an image used to verify the image recognition model to be verified.

[0144] In an embodiment of the present application, image recognition model training samples and image recognition model verification samples can be obtained; the image recognition model training samples include sample vehicle images and sample vehicle standard attributes; the image recognition model verification samples include verification vehicle images.

[0145] Then, the sample vehicle image can be used as the input of the image recognition model, and the sample vehicle standard attributes can be used as the output of the image recognition model. The image recognition model can be trained using the image recognition model training samples to obtain the image recognition model to be verified; the sample vehicle standard attributes include at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color.

[0146] The verification vehicle image can be input into the image recognition model to be verified to obtain the sample vehicle verification attributes output by the image recognition model to be verified; the sample vehicle verification attributes include at least one of the verification vehicle image area, verification sample vehicle style, verification sample vehicle type, and verification sample vehicle color.

[0147] The loss function corresponding to the image recognition model can be calculated based on the sample vehicle standard attributes and the sample vehicle verification attributes.

[0148] The loss function corresponding to the image recognition model can be used to adjust the model parameters of the image recognition model to be verified, and the sample vehicle image is used as the input of the image recognition model, the sample vehicle standard attribute is used as the output of the image recognition model, and the image recognition model training sample is used to train the image recognition model until the preset first training stop condition is reached to obtain a trained image recognition model. Among them, the first training stop condition can be set according to the actual situation, and this application does not limit this.

[0149] This application combines image recognition model training samples and image recognition model verification samples to realize the training of the image recognition model, and adjusts the parameters of the image recognition model by calculating the loss function, thereby ensuring the training effect of the image recognition model and laying a solid foundation for obtaining more accurate vehicle attribute information.

[0150] In an optional embodiment of the present application, step S31 further includes the following sub-steps:

[0151] S41: Obtain a sample video shot for a sample vehicle;

[0152] S42: Segment the sample video to obtain a number of sample video frames, and use the sample video frames containing the sample vehicles as the sample vehicle images and the verification vehicle images;

[0153] S43: Using the verification vehicle image as a verification sample for the image recognition model;

[0154] S44: taking the image region containing the sample vehicle in the sample vehicle image as the sample vehicle image region;

[0155] S45: Determine at least one of the sample vehicle style, the sample vehicle type, and the sample vehicle color corresponding to the sample vehicle according to the sample vehicle image;

[0156] S46: Check whether the sample vehicle image matches at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, and the sample vehicle color;

[0157] S47: If there is a match, at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, and the sample vehicle color is used as the sample vehicle standard attribute;

[0158] S48: Using the sample vehicle image and the sample vehicle standard attributes as training samples for the image recognition model.

[0159] It should be noted that the sample vehicle may be a vehicle to be identified in a training sample; the sample video may be a video used as a training sample, and the sample video includes a sample vehicle.

[0160] The standard attributes of the sample vehicle may refer to the vehicle attributes corresponding to the sample vehicle that are confirmed to be correct and can be used as standards.

[0161] In the embodiment of the present application, a sample video shot for a sample vehicle can be obtained, the sample video can be segmented to obtain a number of sample video frames, the sample video frames containing the sample vehicle can be used as sample vehicle images and verification vehicle images, and the verification vehicle images can be used as image recognition model verification samples. The sample video can be shot by a driving recorder of the vehicle where the user is, or can be obtained by other means.

[0162] Then, the image area containing the sample vehicle in the sample vehicle image can be used as the sample vehicle image area, and at least one of the sample vehicle style, sample vehicle type, and sample vehicle color corresponding to the sample vehicle can be determined according to the sample vehicle image. Check whether the sample vehicle image matches at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color. If they match, at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color is used as the sample vehicle standard attribute. In a specific implementation, a multimodal model can be used to check whether the sample vehicle image matches at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color.

[0163] If the sample vehicle image matches at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color, it can be considered that at least one of the vehicle attributes of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color corresponding to the sample vehicle image is accurate, and it can be considered that the data annotation of the vehicle attribute corresponding to the sample vehicle image is completed. If the sample vehicle image does not match any of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color, the data annotation of the vehicle attribute corresponding to the sample vehicle image is performed again.

[0164] Finally, the sample vehicle images and the sample vehicle standard attributes can be used as training samples for the image recognition model.

[0165] This application obtains sample vehicle images and verification vehicle images by segmenting the video, and then obtains the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color, and then obtains the image recognition model training samples and image recognition model verification samples. This can ensure the accuracy of the training samples and verification samples, improve the training quality and accuracy of the image recognition model, and avoid the degradation of model performance due to inaccurate data labeling.

[0166] In an optional embodiment of the present application, the image recognition model includes at least one residual structure, a compressed excitation structure, and at least one fully connected layer, and step S33 further includes the following sub-steps:

[0167] S51: inputting the verification vehicle image into the residual structure for processing to obtain initial image features output by the residual structure;

[0168] S52: Inputting the initial image features into the compression excitation structure for processing to obtain target image features output by the compression excitation structure;

[0169] S53: Input the target image features into the fully connected layer for processing to obtain the sample vehicle verification attributes.

[0170] It should be noted that the residual structure refers to a structure in a deep neural network that solves the gradient vanishing and gradient exploding problems by introducing skip connections. The residual structure makes it easier for the network to learn identity mapping by adding the input directly to the output, thereby improving the depth and performance of the network. In an embodiment of the present application, a picture recognition model can be constructed by using several residual structures.

[0171] The Squeeze-and-Excitation Structure (SE structure for short) is a technology used to enhance the representation capability of convolutional neural networks. The SE structure improves the performance of the network by explicitly modeling the interdependence between channels and adaptively recalibrating the weights of channel features. In an embodiment of the present application, a picture recognition model can be constructed by combining a plurality of residual structures with the Squeeze-and-Excitation Structure.

[0172] A fully connected layer is usually used to map input features to output features. Each neuron is connected to all neurons in the previous layer, hence the name "fully connected". In the embodiment of the present application, different fully connected layers can be used to achieve outputs of different attributes.

[0173] In an embodiment of the present application, the verification vehicle image can be input into the residual structure for processing to obtain the initial image features output by the residual structure, and then the initial image features can be input into the compressed excitation structure for processing to obtain the target image features output by the compressed excitation structure, and finally the target image features can be input into the fully connected layer for processing to obtain the sample vehicle verification attributes.

[0174] In another embodiment of the present application, the image recognition model may also include a residual network, a compression excitation structure, a pooling layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, a fourth fully connected layer, a weight adjustment layer, a convolution structure, and a convolution splicing layer, and the residual network includes a first residual structure, a second residual structure, a third residual structure, and a fourth residual structure.

[0175] It should be noted that the residual network can be composed of multiple residual structures stacked together. In the embodiment of the present application, the residual network can be used as the basis of the image recognition model. Each residual structure can include the following main components:

[0176] Convolutional layer: used to extract features of input data.

[0177] Batch Normalization Layer: Used to speed up the training process and reduce internal covariate shift.

[0178] ReLU activation function: used to introduce nonlinear characteristics.

[0179] Skip connections: Add the input directly to the output, making it easier for the network to learn the identity mapping.

[0180] Pooling layer may refer to a type of layer commonly used in convolutional neural networks (CNNs) to reduce the spatial dimension of feature maps, thereby reducing computational complexity and memory consumption. Pooling layers are usually used after convolutional layers to extract the main features of feature maps through downsampling operations (such as maximum pooling or average pooling). In an embodiment of the present application, image features can be obtained through the pooling layer.

[0181] The weight adjustment layer may refer to a layer in a neural network that is used to adjust model parameters. For example, the ArcFace layer introduces an angle cosine term based on the Softmax loss, making the features more compact and separable in the feature space. In an embodiment of the present application, the loss function can be adjusted by the weight adjustment layer.

[0182] The convolution structure may refer to a structure used to extract features of input data in a convolutional neural network. The convolution structure is usually composed of multiple convolutional layers, activation functions, and pooling layers, and extracts features of input data through convolution operations and nonlinear transformations. In an embodiment of the present application, image features can be processed through a convolutional layer in an image recognition model to obtain a vehicle image area.

[0183] A convolutional splicing layer may refer to a layer in a convolutional neural network that is used to splice together the output feature maps of multiple convolutional layers. A convolutional splicing layer is typically used in multi-scale feature fusion or multi-branch network structures to extract richer features by splicing feature maps of different scales. In an embodiment of the present application, image features at different stages may be spliced ​​together through a convolutional splicing layer.

[0184] The step S33 may further include the following sub-steps:

[0185] Inputting the verification vehicle image into the first residual structure for processing to obtain a first initial image feature output by the first residual structure;

[0186] Inputting the first initial image feature into the second residual structure for processing to obtain a second initial image feature output by the second residual structure;

[0187] Inputting the second initial image feature into the third residual structure for processing to obtain a third initial image feature output by the third residual structure;

[0188] Inputting the third initial image feature into the fourth residual structure for processing to obtain a fourth initial image feature output by the fourth residual structure;

[0189] Inputting the first initial image feature, the second initial image feature, the third initial image feature, and the fourth initial image feature into the compression excitation structure for processing respectively, and obtaining the first target image feature, the second target image feature, the third target image feature, and the fourth target image feature output by the compression excitation structure;

[0190] Inputting the fourth target image feature into the pooling layer for processing to obtain a first compressed image feature;

[0191] Inputting the first compressed image feature into the first fully connected layer for processing, and obtaining the verification sample vehicle type output by the first fully connected layer;

[0192] Inputting the first compressed image feature into the second fully connected layer for processing, and obtaining the verification sample vehicle color output by the second fully connected layer;

[0193] Inputting the first compressed image features into the third fully connected layer and the weight adjustment layer for processing respectively, to obtain the first sample vehicle style output by the third fully connected layer and the vehicle style weight output by the weight adjustment layer;

[0194] Adjusting the first sample vehicle style according to the vehicle style weight to obtain the verification sample vehicle style;

[0195] Inputting the second target image feature, the third target image feature, and the fourth target image feature into the convolution structure respectively, to obtain a fifth target image feature, a sixth target image feature, and a seventh target image feature respectively output by the convolution structure;

[0196] splicing the fifth target image feature, the sixth target image feature, and the seventh target image feature through the convolutional splicing layer to obtain an eighth target image feature;

[0197] The eighth target image feature is input into the fourth fully connected layer for processing to obtain the verification vehicle image region output by the fourth fully connected layer.

[0198] In the specific implementation, you can refer to Figure 4 Realize the construction of image recognition model, Figure 4 A structural diagram of an image recognition model provided in one embodiment of the present application is shown.

[0199] The network structure of the image recognition model is as follows Figure 4, the image recognition model can integrate multiple tasks in one network structure, and they belong to different types of tasks. The first is the feature extraction network. Here, the basic model adopts a residual network, and adds a new SE module internally, a compression and excitation network to improve the representation ability of the original network. The network output of this module has four feature dimensions, namely [B, dim, CH, CW], [B, dim*2, CH / 2, CW / 2], [B, dim*4, CH / 4, CW / 4], [B, dim*8, CH / 8, CW / 8], corresponding to different feature stages.

[0200] In the vehicle type and vehicle color headers, the feature output of the last layer [B, dim*8, CH / 8, CW / 8] is used. First, it passes through the global average pooling layer, followed by the respective FC fully connected layers. The output category number vehicle type can be: sedan, SUV, MPV, off-road vehicle, pickup truck, truck, engineering vehicle, tanker, other, 9 categories. The vehicle color is: red, green, white, yellow, silver, gray, blue, black, purple, brown, other, 11 categories. The losses of these two classification headers are cross entropy loss_type and loss_color.

[0201] In the vehicle model head, the feature output of the last layer [B, dim*8, CH / 8, CW / 8] is also used. After the global average pooling layer, it will enter the FC fully connected layer and the ArcFace layer in parallel. The number of output categories of the FC fully connected layer depends on the data situation. In this method, it is 3100 categories, which basically includes all major mainstream car models. This part of the loss is calculated as cross entropy loss_chekuan1. There is a learnable weight in the ArcFace layer that enters. The ArcFace layer does not output during the training stage, but participates in the loss calculation, which is arcface loss loss_chekuan2.

[0202] Finally, the vehicle image area head uses the last three layers of features output by the feature extraction network, fea1=[B, dim*2, CH / 2, CW / 2], fea2= [B, dim*4, CH / 4, CW / 4], fea3= [B, dim*8, CH / 8, CW / 8]. fea1, fea2, and fea3 undergo a layer of convolution, BN, and Relu layers respectively. The output dimensions of the convolution layers are the same, and the three layers of convolution outputs are concat together. Then they enter the FC fully connected layer, and the number of output categories is 4, representing the upper left corner coordinates x0, y0 and the lower right corner coordinates x1, y1 of the target box. The loss of this part uses the mean square error loss loss_mse.

[0203] Finally, the total loss of the current network loss = w1*loss_type + w2*loss_color + w3*(w31*loss_chekaun1 + w32*loss_chekuan2) + w4*loss_mse;

[0204] w1,w2,w3,w31,w32,w4 can all be initially set to 1.0.

[0205] This application constructs a residual network through a residual structure and adds a compressed excitation structure, which can improve the representation ability of the original network, and can output different attributes of image features through different fully connected layers. It integrates multiple small models without reducing the accuracy, unifies the training and optimization management, and can output various attributes of the vehicle when a vehicle picture is input, thereby improving the efficiency of users in obtaining vehicle attributes.

[0206] In an optional embodiment of the present application, the multimodal model is trained in the following manner:

[0207] S61: Acquire a multimodal model training sample and a multimodal model verification sample; the multimodal model training sample includes a sample vehicle image, sample attribute query information, and a sample vehicle response attribute; the multimodal model verification sample includes a verification vehicle image and verification attribute query information;

[0208] S62: using the sample vehicle image and the sample attribute query information as inputs of the multimodal model, using the sample vehicle reply attributes as outputs of the multimodal model, and using the multimodal model training samples to train the multimodal model to obtain a multimodal model to be verified;

[0209] S63: inputting the verification vehicle image and the verification attribute query information into the multimodal model to be verified, and obtaining the verification vehicle reply attribute output by the multimodal model to be verified;

[0210] S64: Calculating a loss function corresponding to the multimodal model according to the sample vehicle response attribute and the verification vehicle response attribute;

[0211] S65: Use the loss function corresponding to the multimodal model to adjust the model parameters of the multimodal model to be verified, and re-execute the steps of taking the sample vehicle image and the sample attribute query information as the input of the multimodal model, taking the sample vehicle response attributes as the output of the multimodal model, and using the multimodal model training samples to train the multimodal model, until a preset second training stop condition is reached, thereby obtaining a trained multimodal model.

[0212] It should be noted that the image recognition model training samples may refer to image data sets used to train multimodal models. These samples usually contain a large amount of image data, and each image is accompanied by a corresponding label to indicate the attribute question and answer information corresponding to the image. In an embodiment of the present application, the image recognition model training samples may include sample vehicle images, sample attribute query information, and sample vehicle response attributes. The sample attribute query information may be query information issued for the sample vehicle image, and the sample vehicle response attributes may be statements characterizing vehicle attributes that are responded to the sample vehicle image and the sample attribute query information.

[0213] The image recognition model verification sample may refer to an image data set used to verify the multimodal model. The multimodal model verification sample may include a verification vehicle image and verification attribute query information. In an embodiment of the present application, the verification attribute query information is query information used to verify the image recognition model to be verified.

[0214] In an embodiment of the present application, multimodal model training samples and multimodal model verification samples can be obtained; the multimodal model training samples include sample vehicle images, sample attribute query information, and sample vehicle response attributes; the multimodal model verification samples include verification vehicle images and verification attribute query information.

[0215] Then, the sample vehicle image and sample attribute query information can be used as the input of the multimodal model, the sample vehicle response attributes can be used as the output of the multimodal model, and the multimodal model can be trained using the multimodal model training samples to obtain the multimodal model to be verified.

[0216] The verification vehicle image and verification attribute query information can be input into the multimodal model to be verified, and the verification vehicle response attributes output by the multimodal model to be verified can be obtained.

[0217] The loss function corresponding to the multimodal model can be calculated based on the sample vehicle response attributes and the verification vehicle response attributes.

[0218] The loss function corresponding to the multimodal model can be used to adjust the model parameters of the multimodal model to be verified, and the sample vehicle image and sample attribute query information are used as the input of the multimodal model, the sample vehicle reply attribute is used as the output of the multimodal model, and the steps of training the multimodal model using the multimodal model training sample are re-executed until the preset second training stop condition is reached to obtain a trained multimodal model. Among them, the second training stop condition can be set according to the actual situation, and this application does not limit this.

[0219] This application combines multimodal model training samples and multimodal model verification samples to realize the training of the multimodal model, and adjusts the parameters of the multimodal model by calculating the loss function, thereby ensuring the training effect of the multimodal model and laying a good foundation for obtaining more accurate vehicle attribute information.

[0220] In an optional embodiment of the present application, step S61 further includes the following sub-steps:

[0221] S71: Determine the sample vehicle position according to the position of the center point of the sample vehicle image area in the sample video frame;

[0222] S72: Determine the sample vehicle sales volume and sample vehicle evaluation corresponding to the sample vehicle according to the sample vehicle image;

[0223] S73: Determine the sample attribute query information and the verification attribute query information according to at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle orientation, the sample vehicle sales volume, and the sample vehicle evaluation;

[0224] S74: using the verification vehicle image and the verification attribute query information as the multimodal model verification sample;

[0225] S75: taking at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle position, the sample vehicle sales volume, and the sample vehicle evaluation corresponding to the sample attribute query information as a sample vehicle response attribute;

[0226] S76: Use the sample vehicle image, the sample attribute query information, and the sample vehicle response attributes as training samples for the multimodal model.

[0227] In the embodiment of the present application, the sample vehicle orientation may be determined according to the position of the center point of the sample vehicle image region in the sample video frame, wherein the vehicle orientation may represent the position of the sample vehicle in the sample video frame.

[0228] Then, the sample vehicle sales and sample vehicle evaluation corresponding to the sample vehicle can be determined according to the sample vehicle image, wherein the sample vehicle sales and sample vehicle evaluation can be obtained through search engine search.

[0229] The sample attribute query information and the verification attribute query information may be determined based on at least one of a sample vehicle image area, a sample vehicle style, a sample vehicle type, a sample vehicle color, a sample vehicle location, a sample vehicle sales volume, and a sample vehicle evaluation.

[0230] The verification vehicle image and verification attribute query information are used as multimodal model verification samples.

[0231] At least one of the sample vehicle image area, sample vehicle style, sample vehicle type, sample vehicle color, sample vehicle location, sample vehicle sales volume, and sample vehicle evaluation corresponding to the sample attribute query information is used as the sample vehicle response attribute.

[0232] The sample vehicle image, the sample attribute query information, and the sample vehicle answer attribute are used as the multimodal model training samples, wherein the sample attribute query information and the sample vehicle answer attribute can be a question-answer pair obtained for the same sample vehicle image.

[0233] This application obtains sample vehicle locations, sample vehicle sales, and sample vehicle evaluations to obtain multimodal model training samples and multimodal model verification samples. This can ensure the accuracy of training samples and verification samples, improve the training quality and accuracy of the multimodal model, and avoid the degradation of model performance due to inaccurate data labeling.

[0234] In an optional embodiment of the present application, the multimodal model includes an image encoder, a text encoder, a feature fusion structure, and a decoder, and step S63 further includes the following sub-steps:

[0235] S81: Inputting the verification vehicle image into the image encoder for processing to obtain image coding features output by the image encoder;

[0236] S82: Inputting the verification attribute query information into the text encoder for processing to obtain text encoding features output by the text encoder;

[0237] S83: Inputting the image coding feature and the text coding feature into the feature fusion structure for processing, and obtaining the text image splicing feature output by the feature fusion structure;

[0238] S84: Input the text image splicing features into the decoder for processing to obtain the verification vehicle reply attributes.

[0239] It should be noted that the image encoder can be a neural network structure for encoding image data into a high-dimensional feature vector. The image encoder is usually composed of multiple convolutional layers, pooling layers, and fully connected layers, and extracts image features through convolution operations and nonlinear transformations. In the embodiment of the present application, the vehicle image can be converted into a code by the image encoder;

[0240] A text encoder can be a neural network structure for encoding text data into a high-dimensional feature vector. A text encoder is usually composed of multiple embedding layers, recurrent neural networks (RNNs) or transformer layers, and extracts text features through word embedding and sequence modeling. In an embodiment of the present application, the verification attribute query information can be converted into a code by a text encoder.

[0241] The feature fusion structure can be a neural network structure for fusing feature vectors of different modalities. The feature fusion structure usually fuses image features and text features together through splicing, weighted sum or attention mechanism to extract richer feature representation. In an embodiment of the present application, the encoding obtained by splicing images and texts respectively can be obtained by splicing the feature fusion structure.

[0242] The decoder can be a neural network structure for decoding a high-dimensional feature vector into output data. The decoder is usually composed of multiple fully connected layers, deconvolution layers, or recurrent neural network (RNN) layers, and decodes the feature vector into output data through nonlinear transformation and inverse operation. In the embodiment of the present application, the spliced ​​image and text can be converted into the obtained codes respectively, and the vehicle attributes can be obtained by processing through the decoder.

[0243] In an embodiment of the present application, the verification vehicle image can be input into an image encoder for processing to obtain image coding features output by the image encoder, and the verification attribute query information can be input into a text encoder for processing to obtain text coding features output by the text encoder.

[0244] The image coding features and the text coding features can be input into the feature fusion structure for processing, and the text image splicing features output by the feature fusion structure can be obtained. The text image splicing features can be input into the decoder for processing to obtain the verified vehicle response attributes.

[0245] In another embodiment of the present application, the multimodal model includes an image segmentation structure, an image compression structure, an image encoder, a text encoder, a feature fusion structure, and a decoder.

[0246] It should be noted that the image segmentation structure may refer to a structure that divides an image into multiple regions or pixel levels in an image processing task. The image segmentation structure is usually used for tasks such as semantic segmentation, instance segmentation, and panoptic segmentation. By dividing the image into different regions or objects, the semantic information of the image is extracted. In an embodiment of the present application, the vehicle image can be segmented by the image segmentation structure.

[0247] An image compression structure may refer to a structure used to compress image data in an image processing task. An image compression structure is generally used to reduce the storage space and transmission bandwidth of image data, and to achieve image data compression by removing redundant information and quantizing image data. In an embodiment of the present application, the segmented vehicle image may be compressed by the image compression structure.

[0248] Step S63 also includes the following sub-steps:

[0249] Inputting the verification vehicle image into the image segmentation structure for processing to obtain a fifth initial image feature output by the image segmentation structure;

[0250] Inputting the fifth initial image feature into the image compression structure for processing to obtain a sixth initial image feature output by the image compression structure;

[0251] Inputting the sixth initial image feature into the image encoder for processing to obtain an image coding feature output by the image encoder;

[0252] Inputting the verification attribute query information into the text encoder for processing to obtain text encoding features output by the text encoder;

[0253] Inputting the image coding feature and the text coding feature into the feature fusion structure for processing, and obtaining the text image splicing feature output by the feature fusion structure;

[0254] The text image splicing features are input into the decoder for processing to obtain the verification vehicle reply attributes.

[0255] The present application forms a multimodal model through an image encoder, a text encoder, a feature fusion structure, and a decoder, and decodes the image coding features after splicing and processing the text coding features to obtain vehicle attributes. The multimodal model can process and understand information from different modalities (such as text, images), which makes them more effective in understanding and processing complex data, and can be better interactively used during use. The multimodal model can be used to answer vehicle pictures and query information issued for vehicle pictures, realize vehicle attribute identification through interactive question and answer, and enhance user experience.

[0256] In the specific implementation, you can refer to Figure 5 To obtain training samples, Figure 5 A schematic diagram of a process for obtaining training samples provided by an embodiment of the present application is shown.

[0257] First, we need to collect a large amount of material for the current cockpit perspective. The data source is the dashcam. Since the collected data is video data, we need to cut the data into single pictures. Then we need to create a data set, which includes pictures and question-answer text pairs corresponding to the pictures. Therefore, at this stage, we also need to create question-answer text pairs for single pictures, that is, QA question-answer data. Since the target is the car in the picture, we need to extract the relevant attribute information of the vehicle.

[0258] The first step is to extract the vehicle location information. Use the object detection model to extract the box information of the vehicle in the image data. The extraction result is [x0, y0, x1, y1], which corresponds to the coordinates of the upper left corner and lower right corner of the rectangular box. Then the vehicle orientation information can be extracted through the vehicle orientation recognition model. According to the previous vehicle coordinate frame information, the coordinates of the center point of the target frame cx=x0+(x1-x0) / 2 and cy=y0+(y1-y0) / 2 are calculated. In order to clarify the orientation information, the orientation information is unified into three directions: front, left front, and right front, which are determined according to the position of the center point of the target frame in a single picture. According to the division rule, the coordinates of the first vertical line on the left in a single picture are (3 / 8 * w, 0) and (3 / 8 * w, h / 2); the coordinates of the first vertical line on the right in a single picture are (5 / 8 * w, 0) and (5 / 8 * w, h / 2), where w and h are the length and width of the image, and then they are connected to the lower left corner and the lower right corner to obtain the current orientation division area. After connecting the line segments, the closed area on the left side of the single picture is the left front, the closed area in the middle is the left front, and the closed area on the right is the right front.

[0259] After the above steps, the coordinate frame and orientation information of the vehicle have been obtained, that is, the vehicle area and vehicle orientation in the embodiment of the present application. The second step is to extract the vehicle attribute information, that is, to obtain the vehicle style, vehicle color, and vehicle type of the vehicle. The vehicle is cropped out from the video frame according to the coordinate frame. First, the vehicle style classification model, vehicle color classification model, and vehicle type classification model are used to obtain the vehicle style, vehicle color, and vehicle type. Then, these three parts of the label are processed into a template for scoring to ensure the reliability of the result. The standard template style can be "This is a red SUV of xx car xx model". This paragraph and the cropped vehicle picture are input into the multimodal model together. "This is a red SUV of xx car xx model" is used as the context prompt. The input question is: "Please judge whether the prompt information is correct based on the picture. You only need to answer right or wrong. No other irrelevant information is required." Through the auxiliary judgment of the multimodal model, the manual inspection workload is reduced, and only the label with the answer of "no" is further checked.

[0260] After the above steps, the vehicle style, color, and type of the vehicle have been obtained. The third step is to generate text description information, that is, to obtain the description information of the entire picture and the sales and evaluation of the vehicle. First, the entire picture and the question: "Please describe the content of the picture in detail" are input into the multimodal model together, and several text answers of varying lengths will be obtained. These text answers of varying lengths can be used as the description information of the entire picture. Then, based on the previously obtained vehicle model information, sales and evaluations are searched in external search engines to obtain the vehicle sales and vehicle evaluations corresponding to the vehicle model information. Then, the large language model is used to summarize the two parts of vehicle sales and vehicle evaluations, and the redundant length information can be refined into short and useful text information, and the refined text information can be used as the final vehicle sales and vehicle evaluations. In this way, the necessary information based on the picture is sorted out.

[0261] After the above steps, the vehicle location information, vehicle attribute information, and text description information are obtained. Specifically, it may include the vehicle image area, vehicle orientation, vehicle style, vehicle color, vehicle type, vehicle sales, vehicle evaluation, and description information of the entire picture. Then, question and answer data as training data is generated based on this information. The format of the question and answer data in this application can be a question and answer text pair, such as a QA question and answer format. First, a list of questions is formulated for the information obtained, including basic questions such as "What car is this?", "What color is this car?", "How is the sales of this car?", and "What is the reputation of this car?". In order to enrich the diversity of questions, these questions are data enhanced through a large language model. At this step, a rich pair of questions and answers has been obtained. In order to further enrich the diversity of data and context, these questions and answer pairs can be randomly arranged and combined, and artificially constructed into question and answer text pairs. The specific form is: single QA question and answer, two-two QA question and answer... multiple QA question and answer, etc. The specific combination method is to combine different types and different numbers of question and answer pairs respectively. During the combination process, a complete question and answer pair will not be split, and the question and answer pair itself still maintains a one-to-one correspondence. Finally, the image and the corresponding question-answer text pair are used as training samples. This series of operations not only enhances the data of the question itself, but also enhances the contextual information that the question itself depends on, which is more conducive to the generalization of subsequent model training.

[0262] In the specific implementation, you can refer to Figure 6 Realize the construction of multimodal models, Figure 6 A schematic diagram of the structure of a multimodal model provided in one embodiment of the present application is shown.

[0263] Multimodal network structure such as Figure 6First, the image requires an input size of 448x448. The image is input into the image segmentation structure and divided according to patch=16. Then the 448x448 image is divided into 28x28=784 patches, and each patch dimension is 16x16x3=768. The hidden layer dimension here is also 768. The 784 patches divided here need to be compressed to reduce the number of patches to 96 to reduce the calculation pressure. Then the data is sent to the image encoder, and the feature im_hidden_state dimension is 96x768. In order to indicate that this part of the feature is an image, placeholders need to be added before and after the image feature.<im_start> ,<im_end> , the empty placeholder is<unk_token> , then combine the text with the image placeholder to get the text and image placeholder, im_text_id = bos_id + (<im_start> +<unk_token> * 96 +<im_end> ) * num_slice + tokenizer.encode(“<User>What kind of car is this? <ai>A certain brand and model") + eos_id, in order to reduce the calculation pressure, here num_slice = 1, bos_id and eos_id represent the text start and end characters of the LLM language model, then input im_text_id into the text encoder, such as the LLM language encoder, and get the feature llm_embedding dimension is [L, 768]. At this time, input im_hidden_state and llm_embedding into the feature fusion structure for fusion, and remove the placeholder of the image first.<unk_token> The position feature is replaced by im_hidden_state to obtain the new feature llm_im_embedding with unchanged dimension. Then the new feature llm_im_embedding is input into the decoder network for fine-tuning training.

[0264] In the specific implementation, you can refer to Figure 7 Realize the construction of fusion model, Figure 7 A structural diagram of a fusion model provided in an embodiment of the present application is shown.

[0265] The fusion model is divided into two parts. The first part is a small model that can be run on the terminal side, that is, the image recognition model of this application, including a vehicle style classification model, a vehicle color classification model, a vehicle type classification model, and a target detection model. The input of this part is a picture, and the image dimension of the picture is [B, C, H, W], where B is the batch size, which represents how much data is sent to the model at a time, C is the number of channels, and H and W are the height and width of the picture. The second part is a multimodal model, where the multimodal large model MiniCPM-V2.5 is used. Its input consists of two parts, pictures and question-answer text pairs. The image dimension of the picture is also [B, C, H, W], and the text dimension of the question-answer text pair is [B, L], where L is the length of the input text.

[0266] Reference Figure 8 , shows a schematic diagram of the structure of a vehicle attribute recognition device provided by an embodiment of the present application, the device comprising:

[0267] The data acquisition module 801 is used to respond to the attribute recognition instruction issued by the user for the vehicle to be recognized, and obtain the vehicle image of the vehicle to be recognized and the vehicle attribute query information issued by the user for the vehicle to be recognized;

[0268] A first vehicle attribute acquisition module 802, configured to input the vehicle image and the vehicle attribute query information into a pre-trained multimodal model for processing, and output a first vehicle attribute corresponding to the vehicle to be identified;

[0269] A second vehicle attribute acquisition module 803 is used to input the vehicle image and the vehicle attribute query information into the pre-trained multimodal model for processing again, and output a second vehicle attribute corresponding to the vehicle to be identified;

[0270] A third vehicle attribute acquisition module 804 is used to input the vehicle image into a pre-trained image recognition model for processing and output at least one third vehicle attribute corresponding to the vehicle to be identified when the first vehicle attribute is different from the second vehicle attribute;

[0271] A target vehicle attribute acquisition module 805, configured to determine a target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information;

[0272] The recognition result acquisition module 806 is used to present the vehicle image and the target vehicle attributes as the attribute recognition result of the vehicle to be recognized to the user.

[0273] In an optional embodiment of the present application, the device further includes:

[0274] The target vehicle attribute first acquisition submodule is used to determine the first vehicle attribute or the second vehicle attribute as the target vehicle attribute corresponding to the vehicle to be identified when the first vehicle attribute is the same as the second vehicle attribute.

[0275] In an optional embodiment of the present application, the target vehicle attribute acquisition module 805 includes:

[0276] A target vehicle attribute category determination module, used to determine the target vehicle attribute category according to the vehicle attribute query information;

[0277] a fourth vehicle attribute acquisition module, configured to determine the third vehicle attribute corresponding to the target vehicle attribute category from at least one of the third vehicle attributes, and use the third vehicle attribute corresponding to the target vehicle attribute category as the fourth vehicle attribute;

[0278] an attribute confirmation information acquisition module, used for combining the fourth vehicle attribute with the vehicle attribute inquiry information to form attribute confirmation information;

[0279] The second target vehicle attribute acquisition submodule is used to input the vehicle image and the attribute confirmation information into the pre-trained multimodal model for processing, and output the target vehicle attributes corresponding to the vehicle to be identified.

[0280] In an optional embodiment of the present application, the image recognition model is trained using the following device:

[0281] A first training sample acquisition module is used to acquire a picture recognition model training sample and a picture recognition model verification sample; the picture recognition model training sample includes a sample vehicle image and a sample vehicle standard attribute; the picture recognition model verification sample includes a verification vehicle image;

[0282] A picture recognition model training module, used to use the sample vehicle image as the input of the picture recognition model, use the sample vehicle standard attributes as the output of the picture recognition model, and use the picture recognition model training sample to train the picture recognition model to obtain the picture recognition model to be verified; the sample vehicle standard attributes include at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color;

[0283] The image recognition model verification module is used to input the verification vehicle image into the image recognition model to be verified, and obtain the sample vehicle verification attribute output by the image recognition model to be verified; the sample vehicle verification attribute includes at least one of the verification vehicle image area, verification sample vehicle style, verification sample vehicle type, and verification sample vehicle color;

[0284] An image recognition model loss function acquisition module, used to calculate the loss function corresponding to the image recognition model according to the sample vehicle standard attributes and the sample vehicle verification attributes;

[0285] The image recognition model optimization module is used to adjust the model parameters of the image recognition model to be verified by using the loss function corresponding to the image recognition model, and re-execute the steps of taking the sample vehicle image as the input of the image recognition model, taking the sample vehicle standard attributes as the output of the image recognition model, and using the image recognition model training samples to train the image recognition model until a preset first training stop condition is reached to obtain a trained image recognition model.

[0286] In an optional embodiment of the present application, the first training sample acquisition module includes:

[0287] A sample video acquisition module is used to acquire a sample video shot for a sample vehicle;

[0288] A sample video segmentation module, used for segmenting the sample video to obtain a plurality of sample video frames, and taking the sample video frames containing the sample vehicles as the sample vehicle images and the verification vehicle images;

[0289] A first verification sample acquisition module, used to use the verification vehicle image as a verification sample for the image recognition model;

[0290] An image region acquisition module, used for taking an image region containing a sample vehicle in a sample vehicle image as the sample vehicle image region;

[0291] A first vehicle attribute acquisition module, configured to determine at least one of the sample vehicle style, the sample vehicle type, and the sample vehicle color corresponding to the sample vehicle according to the sample vehicle image;

[0292] A first vehicle attribute checking module, used for checking whether the sample vehicle image matches at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, and the sample vehicle color;

[0293] A first vehicle attribute confirmation module, configured to use at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, and the sample vehicle color as the sample vehicle standard attribute if a match is found;

[0294] The first training sample acquisition submodule is used to use the sample vehicle image and the sample vehicle standard attributes as the image recognition model training samples.

[0295] In an optional embodiment of the present application, the image recognition model includes at least one residual structure, a compressed excitation structure, and at least one fully connected layer, and the image recognition model verification module includes:

[0296] An initial image feature acquisition module, used for inputting the verification vehicle image into the residual structure for processing to obtain the initial image features output by the residual structure;

[0297] A target image feature acquisition module, used for inputting the initial image feature into the compression excitation structure for processing, and obtaining the target image feature output by the compression excitation structure;

[0298] The sample vehicle verification attribute acquisition module is used to input the target image features into the fully connected layer for processing to obtain the sample vehicle verification attributes.

[0299] In an optional embodiment of the present application, the multimodal model is trained using the following apparatus:

[0300] A second training sample acquisition module is used to acquire multimodal model training samples and multimodal model verification samples; the multimodal model training samples include sample vehicle images, sample attribute query information, and sample vehicle response attributes; the multimodal model verification samples include verification vehicle images and verification attribute query information;

[0301] A multimodal model training module, used to take the sample vehicle image and the sample attribute query information as inputs of the multimodal model, take the sample vehicle reply attributes as outputs of the multimodal model, and train the multimodal model using the multimodal model training samples to obtain a multimodal model to be verified;

[0302] A multimodal model verification module, used for inputting the verification vehicle image and the verification attribute query information into the multimodal model to be verified, and obtaining the verification vehicle reply attribute output by the multimodal model to be verified;

[0303] A multimodal model loss function acquisition module, used to calculate the loss function corresponding to the multimodal model according to the sample vehicle response attribute and the verification vehicle response attribute;

[0304] A multimodal model optimization module is used to adjust the model parameters of the multimodal model to be verified using the loss function corresponding to the multimodal model, and re-execute the steps of taking the sample vehicle image and the sample attribute query information as the input of the multimodal model, taking the sample vehicle response attributes as the output of the multimodal model, and training the multimodal model using the multimodal model training samples, until a preset second training stop condition is reached, thereby obtaining a trained multimodal model.

[0305] In an optional embodiment of the present application, the second training sample acquisition module includes:

[0306] A vehicle position acquisition module, used to determine the sample vehicle position according to the position of the center point of the sample vehicle image area in the sample video frame;

[0307] A sales evaluation acquisition module, used to determine the sample vehicle sales and sample vehicle evaluation corresponding to the sample vehicle according to the sample vehicle image;

[0308] A query information acquisition module, used to determine the sample attribute query information and the verification attribute query information according to at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle orientation, the sample vehicle sales volume, and the sample vehicle evaluation;

[0309] A second verification sample acquisition module, configured to use the verification vehicle image and the verification attribute query information as the multimodal model verification sample;

[0310] A second vehicle attribute acquisition module is used to use at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle position, the sample vehicle sales volume, and the sample vehicle evaluation corresponding to the sample attribute query information as a sample vehicle response attribute;

[0311] The second training sample acquisition submodule uses the sample vehicle image, the sample attribute query information, and the sample vehicle response attribute as the multimodal model training samples.

[0312] In an optional embodiment of the present application, the multimodal model includes an image encoder, a text encoder, a feature fusion structure, and a decoder, and the second verification sample acquisition module includes:

[0313] An image coding feature acquisition module, used for inputting the verification vehicle image into the image encoder for processing, and obtaining the image coding features output by the image encoder;

[0314] A text encoding feature acquisition module, used for inputting the verification attribute query information into the text encoder for processing, and obtaining the text encoding feature output by the text encoder;

[0315] A splicing feature acquisition module, used for inputting the image coding feature and the text coding feature into the feature fusion structure for processing, and obtaining the text image splicing feature output by the feature fusion structure;

[0316] The reply attribute acquisition module is used to input the text image splicing features into the decoder for processing to obtain the verification vehicle reply attributes.

[0317] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0318] An embodiment of the present application further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the vehicle attribute recognition method as described above when executed by the processor.

[0319] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the vehicle attribute recognition method as described above is implemented.

[0320] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0321] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0322] Those skilled in the art will appreciate that the present application embodiment can be provided as a method, device, or computer program product. Therefore, the present application embodiment can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application embodiment can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0323] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0324] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0325] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0326] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0327] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0328] The vehicle attribute identification method, device, electronic device and medium provided above are introduced in detail. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.< / ai>

Claims

1. A method for identifying vehicle attributes, characterized in that: The method comprises: In response to an attribute recognition instruction issued by a user for a vehicle to be recognized, acquiring a vehicle image of the vehicle to be recognized and vehicle attribute query information issued by the user for the vehicle to be recognized; Inputting the vehicle image and the vehicle attribute query information into a pre-trained multimodal model for processing, and outputting a first vehicle attribute corresponding to the vehicle to be identified; Inputting the vehicle image and the vehicle attribute query information into the pre-trained multimodal model again for processing, and outputting a second vehicle attribute corresponding to the vehicle to be identified; In the case where the first vehicle attribute is different from the second vehicle attribute, inputting the vehicle image into a pre-trained image recognition model for processing, and outputting at least one third vehicle attribute corresponding to the vehicle to be identified; Determining a target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information; The vehicle image and the target vehicle attributes are presented to the user as attribute recognition results of the vehicle to be identified.

2. The method according to claim 1, characterized in that The method further comprises: In the case where the first vehicle attribute is the same as the second vehicle attribute, the first vehicle attribute or the second vehicle attribute is determined as the target vehicle attribute corresponding to the vehicle to be identified.

3. The method according to claim 1, characterized in that The step of determining the target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information includes: Determine the target vehicle attribute category according to the vehicle attribute query information; Determine the third vehicle attribute corresponding to the target vehicle attribute category from at least one of the third vehicle attributes, and use the third vehicle attribute corresponding to the target vehicle attribute category as a fourth vehicle attribute; combining the fourth vehicle attribute with the vehicle attribute inquiry information to form attribute confirmation information; The vehicle image and the attribute confirmation information are input into the pre-trained multimodal model for processing, and the target vehicle attributes corresponding to the vehicle to be identified are output.

4. The method according to claim 1, characterized in that The image recognition model is trained in the following way: Obtaining a picture recognition model training sample and a picture recognition model verification sample; the picture recognition model training sample includes a sample vehicle image and a sample vehicle standard attribute; the picture recognition model verification sample includes a verification vehicle image; The sample vehicle image is used as the input of the image recognition model, the sample vehicle standard attribute is used as the output of the image recognition model, and the image recognition model is trained using the image recognition model training sample to obtain the image recognition model to be verified; the sample vehicle standard attribute includes at least one of the sample vehicle image area, sample vehicle style, sample vehicle type, and sample vehicle color; Input the verification vehicle image into the image recognition model to be verified, and obtain the sample vehicle verification attribute output by the image recognition model to be verified; the sample vehicle verification attribute includes at least one of the verification vehicle image area, verification sample vehicle style, verification sample vehicle type and verification sample vehicle color; Calculating a loss function corresponding to the image recognition model according to the sample vehicle standard attributes and the sample vehicle verification attributes; The loss function corresponding to the image recognition model is used to adjust the model parameters of the image recognition model to be verified, and the steps of taking the sample vehicle image as the input of the image recognition model, taking the sample vehicle standard attributes as the output of the image recognition model, and using the image recognition model training samples to train the image recognition model are re-executed until a preset first training stop condition is reached to obtain a trained image recognition model.

5. The method according to claim 4, characterized in that The step of obtaining the image recognition model training sample and the image recognition model verification sample includes: Obtain a sample video shot for a sample vehicle; Segmenting the sample video to obtain a plurality of sample video frames, and using the sample video frames containing the sample vehicles as the sample vehicle images and the verification vehicle images; Using the verification vehicle image as a verification sample of the picture recognition model; Taking the image region containing the sample vehicle in the sample vehicle image as the sample vehicle image region; Determine at least one of the sample vehicle style, the sample vehicle type and the sample vehicle color corresponding to the sample vehicle according to the sample vehicle image; Checking whether the sample vehicle image matches at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, and the sample vehicle color; If they match, at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type and the sample vehicle color is used as the sample vehicle standard attribute; The sample vehicle image and the sample vehicle standard attributes are used as training samples for the image recognition model.

6. The method according to claim 4, characterized in that The image recognition model includes at least one residual structure, a compressed excitation structure and at least one fully connected layer, and the inputting of the verification vehicle image into the image recognition model to be verified to obtain the sample vehicle verification attribute output by the image recognition model to be verified includes: Inputting the verification vehicle image into the residual structure for processing to obtain initial image features output by the residual structure; Inputting the initial image features into the compression excitation structure for processing to obtain target image features output by the compression excitation structure; The target image features are input into the fully connected layer for processing to obtain the sample vehicle verification attributes.

7. The method according to claim 5, characterized in that The multimodal model is trained in the following way: Acquire a multimodal model training sample and a multimodal model verification sample; the multimodal model training sample includes a sample vehicle image, sample attribute query information, and a sample vehicle response attribute; the multimodal model verification sample includes a verification vehicle image and verification attribute query information; Taking the sample vehicle image and the sample attribute query information as inputs of the multimodal model, taking the sample vehicle reply attributes as outputs of the multimodal model, and training the multimodal model using the multimodal model training samples to obtain a multimodal model to be verified; Inputting the verification vehicle image and the verification attribute query information into the multimodal model to be verified, and obtaining the verification vehicle reply attribute output by the multimodal model to be verified; Calculating a loss function corresponding to the multimodal model according to the sample vehicle response attributes and the verification vehicle response attributes; The loss function corresponding to the multimodal model is used to adjust the model parameters of the multimodal model to be verified, and the steps of taking the sample vehicle image and the sample attribute query information as the input of the multimodal model, taking the sample vehicle response attributes as the output of the multimodal model, and training the multimodal model using the multimodal model training samples are re-executed until a preset second training stop condition is reached to obtain a trained multimodal model.

8. The method according to claim 7, characterized in that The obtaining of multimodal model training samples and multimodal model verification samples includes: Determine the sample vehicle position according to the position of the center point of the sample vehicle image area in the sample video frame; Determining a sample vehicle sales volume and a sample vehicle evaluation corresponding to the sample vehicle according to the sample vehicle image; Determine the sample attribute query information and the verification attribute query information according to at least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle orientation, the sample vehicle sales volume, and the sample vehicle evaluation; Using the verification vehicle image and the verification attribute query information as verification samples of the multimodal model; At least one of the sample vehicle image area, the sample vehicle style, the sample vehicle type, the sample vehicle color, the sample vehicle position, the sample vehicle sales volume and the sample vehicle evaluation corresponding to the sample attribute query information is used as a sample vehicle response attribute; The sample vehicle image, the sample attribute query information, and the sample vehicle response attributes are used as training samples for the multimodal model.

9. The method according to claim 7, characterized in that: The multimodal model includes an image encoder, a text encoder, a feature fusion structure and a decoder. The step of inputting the verification vehicle image and the verification attribute query information into the multimodal model to be verified to obtain the verification vehicle reply attribute output by the multimodal model to be verified includes: Inputting the verification vehicle image into the image encoder for processing to obtain image coding features output by the image encoder; Inputting the verification attribute query information into the text encoder for processing to obtain text encoding features output by the text encoder; Inputting the image coding feature and the text coding feature into the feature fusion structure for processing, and obtaining the text image splicing feature output by the feature fusion structure; The text image splicing features are input into the decoder for processing to obtain the verification vehicle reply attributes.

10. A vehicle attribute recognition device, characterized in that: The device comprises: A data acquisition module, for responding to an attribute recognition instruction issued by a user for a vehicle to be recognized, and acquiring a vehicle image of the vehicle to be recognized and vehicle attribute inquiry information issued by the user for the vehicle to be recognized; A first vehicle attribute acquisition module, configured to input the vehicle image and the vehicle attribute query information into a pre-trained multimodal model for processing, and output a first vehicle attribute corresponding to the vehicle to be identified; A second vehicle attribute acquisition module, used for inputting the vehicle image and the vehicle attribute query information into the pre-trained multimodal model for processing again, and outputting a second vehicle attribute corresponding to the vehicle to be identified; A third vehicle attribute acquisition module, configured to input the vehicle image into a pre-trained image recognition model for processing and output at least one third vehicle attribute corresponding to the vehicle to be identified when the first vehicle attribute is different from the second vehicle attribute; A target vehicle attribute acquisition module, used to determine a target vehicle attribute corresponding to the vehicle to be identified from at least one of the third vehicle attributes according to the vehicle attribute query information; The recognition result acquisition module is used to display the vehicle image and the target vehicle attributes as the attribute recognition result of the vehicle to be identified to the user.

11. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the vehicle attribute recognition method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the vehicle attribute recognition method according to any one of claims 1 to 9 is implemented.

Images