Label extraction model training method and device, vehicle label extraction method and device and electronic equipment

Abstract

The invention relates to a label extraction model training method and device, a vehicle label extraction method and device and electronic equipment, and the method comprises the steps: inputting a current sample vehicle image into a preset multi-modal model, and obtaining a sample description text; performing grammar analysis processing on the sample description text to obtain at least one segmented word, the part-of-speech of the at least one segmented word and the dependency relationship between the at least one segmented word; performing noun merging based on the at least one segmented word, the part of speech of the at least one segmented word and the dependency relationship among the at least one segmented word to obtain a sample description tag; based on a preset filtering word, filtering the sample description label to obtain a real description label; and training a to-be-trained label extraction model based on the current sample vehicle image and the real description label to obtain a label extraction model. According to the embodiment of the invention, the method can meet the multi-dimensional demands of vehicle recognition, is adaptive to a multi-label scene, greatly improves the readability of a vehicle recognition result, and reduces the decision errors.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a label extraction model training method, a vehicle label extraction method, a device, and an electronic device. Background Technology

[0002] With the rapid development of the transportation industry, the types of vehicles are becoming increasingly diverse. In addition to standard vehicles with regular shapes and fixed structures, there is a growing number of special vehicles with irregular shapes and varied structures obtained by modifying standard vehicles. How to accurately output multi-dimensional vehicle prediction information is a challenge faced by autonomous driving systems.

[0003] In related technologies, models are often trained using the natural language corresponding to images. However, training these models relies on a large amount of natural language data, which is costly in terms of data collection and annotation. Furthermore, unstructured semantics are difficult to train directly, making it impossible to meet the multi-dimensional requirements of vehicle recognition, adapt to multi-label scenarios, and result in low readability of vehicle recognition results. This can lead to safety issues such as decision-making errors. Summary of the Invention

[0004] This disclosure provides a label extraction model training method, a vehicle label extraction method, a device, and an electronic device to at least solve the security problems in related technologies, such as the inability to meet the multi-dimensional needs of vehicle recognition, the inability to adapt to multi-label scenarios, the low readability of vehicle recognition results, and the resulting decision-making errors.

[0005] According to a first aspect of the present disclosure, a method for training a label extraction model is provided, comprising: Obtain the image of the current sample vehicle; Input the current sample vehicle image into a preset multimodal model to obtain the sample description text corresponding to the current sample vehicle image; The sample description text is parsed to obtain at least one word segment corresponding to the sample description text, the part-of-speech of the at least one word segment, and the dependency relationship between the at least one word segment; Based on the at least one word segment, the part-of-speech of the at least one word segment, and the dependency relationship between the at least one word segment, nouns are merged to obtain the sample description tags corresponding to the sample description text; Based on preset filter words, the sample description tags are filtered to obtain the real description tags corresponding to the current sample vehicle image. The label extraction model is trained based on the current sample vehicle image and the real description label to obtain the label extraction model.

[0006] In an optional embodiment, training the label extraction model based on the current sample vehicle image and the real description label to obtain the label extraction model includes: The current sample vehicle image is input into the feature extraction layer of the label extraction model to be trained for feature extraction, and the current sample vehicle features corresponding to the current sample vehicle image are obtained. The current sample vehicle features are input into the cross-attention layer of the label extraction model to be trained. Cross-attention calculation is performed based on at least one preset description label and the current sample vehicle features to obtain sample label indication data of the at least one preset description label. The sample label indication data is used to indicate the probability that the current sample vehicle image includes the feature corresponding to each preset description label in the at least one preset description label. Based on the at least one sample label indication data and a preset indication data threshold, the predicted description label corresponding to the current sample vehicle image is obtained; The label extraction model is trained based on the predicted description labels and the true description labels to obtain the label extraction model.

[0007] In an optional embodiment, the current sample vehicle image is any one of at least one sample image, and the method further includes: Based on the real description labels corresponding to each of the at least one sample image, the at least one preset description label is obtained.

[0008] In an optional embodiment, the step of merging nouns based on the at least one word segment, the part-of-speech tag of the at least one word segment, and the dependency relationship between the at least one word segment to obtain the sample description tag corresponding to the sample description text includes: Based on the part-of-speech tag of the at least one word segment, noun extraction processing is performed on the at least one word segment to obtain at least one noun corresponding to the sample description text; Based on the dependency relationship between the at least one word segment, the at least one noun is merged to obtain the sample description tag corresponding to the sample description text.

[0009] According to a second aspect of the present disclosure, a vehicle tag extraction method based on the tag extraction model described in any one of the first aspects is provided, characterized in that it includes: Acquire an image to be identified; the image to be identified includes at least one object to be identified; The image to be identified is input into a preset recognition model, and recognition is performed based on at least one preset vehicle category to determine at least one vehicle object to be identified. Based on the bounding box information corresponding to each of the at least one vehicle objects to be identified and the image to be identified, determine the vehicle image to be identified corresponding to each vehicle object to be identified; Each vehicle image to be identified is input into the label extraction model for label extraction, thereby obtaining the target description label corresponding to each vehicle object to be identified.

[0010] In an optional embodiment, the step of inputting the image to be identified into a preset recognition model, and identifying at least one vehicle object based on at least one preset vehicle category includes: The image to be identified is input into a preset recognition model, and recognition is performed based on at least one preset vehicle category to obtain the bounding box information corresponding to each of the at least one objects to be identified and the confidence level of each object to be identified belonging to the at least one preset vehicle category. Based on at least one confidence level corresponding to each object to be identified, the at least one vehicle object to be identified is determined from the at least one object to be identified.

[0011] In an optional embodiment, the step of inputting each vehicle image to be identified into the label extraction model for label extraction to obtain the target description label corresponding to each vehicle object to be identified includes: Each vehicle image to be identified is input into the feature extraction layer of the label extraction model for feature extraction, thereby obtaining the vehicle features corresponding to each vehicle object to be identified. Each feature of the vehicle to be identified is input into the cross-attention layer of the label extraction model. Cross-attention calculation is performed based on at least one preset descriptive label and each feature of the vehicle to be identified to obtain label indication data for at least one preset descriptive label corresponding to each image of the vehicle to be identified. The label indication data is used to indicate the probability that the image of the vehicle to be identified includes the feature corresponding to each preset descriptive label in the at least one preset descriptive label. Based on at least one label indication data corresponding to each vehicle image to be identified and a preset indication data threshold, a target description label corresponding to each vehicle object to be identified is obtained.

[0012] According to a third aspect of the present disclosure, a label extraction model training apparatus is provided, comprising: The current sample vehicle image acquisition module is used to acquire the current sample vehicle image; The sample description text acquisition module is used to input the current sample vehicle image into a preset multimodal model to obtain the sample description text corresponding to the current sample vehicle image; The syntax parsing processing module is used to perform syntax parsing processing on the sample description text to obtain at least one word corresponding to the sample description text, the part-of-speech of the at least one word, and the dependency relationship between the at least one word; The noun merging module is used to merge nouns based on the at least one word segment, the part-of-speech of the at least one word segment, and the dependency relationship between the at least one word segment, to obtain the sample description tag corresponding to the sample description text; The filtering module is used to filter the sample description tags based on preset filter words to obtain the real description tags corresponding to the current sample vehicle image. The model training module is used to train the label extraction model to be trained based on the current sample vehicle image and the real description label, so as to obtain the label extraction model.

[0013] In an optional embodiment, the model training module includes: The first feature extraction unit is used to input the current sample vehicle image into the feature extraction layer of the label extraction model to be trained for feature extraction, and obtain the current sample vehicle features corresponding to the current sample vehicle image. The first cross-attention calculation unit is used to input the current sample vehicle features into the cross-attention layer of the label extraction model to be trained, and perform cross-attention calculation based on at least one preset description label and the current sample vehicle features to obtain sample label indication data of the at least one preset description label; the sample label indication data is used to indicate the probability that the current sample vehicle image includes the feature corresponding to each preset description label in the at least one preset description label; The predictive description label acquisition unit is used to obtain the predictive description label corresponding to the current sample vehicle image based on the at least one sample label indication data and a preset indication data threshold; The model training unit is used to train the label extraction model to be trained based on the predicted description labels and the real description labels, so as to obtain the label extraction model.

[0014] In an optional embodiment, the current sample vehicle image is any one of at least one sample image, and the device further includes: The preset description label acquisition module is used to obtain the at least one preset description label based on the real description label corresponding to each of the at least one sample image.

[0015] In an optional embodiment, the noun merging module includes: The noun extraction processing unit is used to perform noun extraction processing on the at least one word segment based on the part-of-speech of the at least one word segment to obtain at least one noun corresponding to the sample description text; The noun merging processing unit is used to merge the at least one noun based on the dependency relationship between the at least one word segmentation to obtain the sample description tag corresponding to the sample description text.

[0016] According to a fourth aspect of the present disclosure, a vehicle tag extraction device is provided, comprising: An image acquisition module is used to acquire an image to be identified; the image to be identified includes at least one object to be identified. The vehicle object determination module is used to input the image to be identified into a preset recognition model, perform recognition based on at least one preset vehicle category, and determine at least one vehicle object to be identified. The vehicle image determination module is used to determine the vehicle image corresponding to each vehicle object based on the bounding box information corresponding to each vehicle object in the at least one vehicle object to be identified and the image to be identified. The label extraction module is used to input each vehicle image to be identified into the label extraction model to extract labels and obtain the target description label corresponding to each vehicle object to be identified.

[0017] In an optional embodiment, the vehicle object determination module includes: The vehicle category recognition unit is used to input the image to be recognized into a preset recognition model, perform recognition based on at least one preset vehicle category, and obtain the bounding box information corresponding to each of the at least one objects to be recognized and the confidence level of each object to be recognized belonging to the at least one preset vehicle category. The vehicle object determination unit is used to determine the at least one vehicle object to be identified from the at least one vehicle object to be identified based on at least one confidence level corresponding to each vehicle object to be identified.

[0018] In an optional embodiment, the tag extraction module includes: The second feature extraction unit is used to input each vehicle image to be identified into the feature extraction layer in the label extraction model to extract features, and obtain the vehicle features corresponding to each vehicle object to be identified. The second cross-attention calculation unit is used to input each vehicle feature to be identified into the cross-attention layer in the label extraction model, and perform cross-attention calculation based on at least one preset description label and each vehicle feature to be identified to obtain label indication data of at least one preset description label corresponding to each vehicle image to be identified; the label indication data is used to indicate the probability that the vehicle image to be identified includes the feature corresponding to each preset description label in the at least one preset description label. The target description label acquisition unit is used to obtain the target description label corresponding to each vehicle object to be identified based on at least one label indication data corresponding to each vehicle image to be identified and a preset indication data threshold.

[0019] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in any one of the first aspects above.

[0020] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided such that, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the first aspects of the present disclosure.

[0021] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: The process involves: acquiring the current sample vehicle image; inputting the current sample vehicle image into a preset multimodal model to obtain the sample description text corresponding to the current sample vehicle image; performing syntactic parsing on the sample description text to obtain at least one word segment, the part-of-speech tag of at least one word segment, and at least one dependency relationship between word segments; merging nouns based on at least one word segment, the part-of-speech tag of at least one word segment, and at least one dependency relationship between word segments to obtain the sample description tag corresponding to the sample description text; filtering the sample description tag based on preset filter words to obtain the real description tag corresponding to the current sample vehicle image; training the tag extraction model based on the current sample vehicle image and the real description tag to obtain the tag extraction model, which can meet the multi-dimensional needs of vehicle recognition, adapt to multi-label scenarios, greatly improve the readability of vehicle recognition results, and thus reduce decision-making errors.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0024] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a label extraction model training method according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating a current sample vehicle image according to an exemplary embodiment; Figure 4 This is a schematic diagram illustrating a predictive description label according to an exemplary embodiment; Figure 5 This is a flowchart illustrating a vehicle tag extraction method according to an exemplary embodiment; Figure 6 This is a block diagram of a label extraction model training device according to an exemplary embodiment; Figure 7 This is a block diagram illustrating a vehicle tag extraction device according to an exemplary embodiment; Figure 8 This is a block diagram illustrating an electronic device for training a label extraction model according to an exemplary embodiment; Figure 9 This is a block diagram illustrating an electronic device for training a label extraction model according to an exemplary embodiment. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar different contents and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0027] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0028] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include terminal 100 and server 200.

[0029] In an optional embodiment, terminal 100 can be used to provide vehicle tag extraction services. Specifically, terminal 100 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, or software running on the aforementioned electronic devices, such as applications. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, Windows, etc.

[0030] In an optional embodiment, server 200 can provide background services to terminal 100 and perform vehicle tag extraction processing. Optionally, server 200 can pre-train a tag extraction model. Accordingly, vehicle tag extraction processing can be performed in conjunction with the tag extraction model. Specifically, server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0031] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure; in practical applications, other application environments may also be included.

[0032] In the embodiments described in this specification, the terminal 100 and the server 200 can be directly or indirectly connected through wired or wireless communication, and this disclosure does not impose any restrictions.

[0033] Figure 2This is a flowchart illustrating a label extraction model training method according to an exemplary embodiment. This label extraction model training method is used in a server, such as... Figure 2 As shown, it includes the following steps.

[0034] In step S201, the image of the current sample vehicle is acquired; In one specific embodiment, the current sample vehicle image can be the training data of the current training cycle. Specifically, the current sample vehicle image can be an image containing vehicles. Optionally, the current sample vehicle image can be randomly determined from at least one sample vehicle image, or a portion of the sample vehicle images that did not participate in the model training can be randomly selected from at least one sample vehicle image as the current sample vehicle image.

[0035] In step S203, the current sample vehicle image is input into a preset multimodal model to obtain the sample description text corresponding to the current sample vehicle image; In a specific embodiment, a preset multimodal model can be used to generate descriptive information corresponding to an image. The preset multimodal model can be a pre-trained multimodal large model. Specifically, the preset multimodal model can be MLLM (Multimodal Large Language Model), GPT-4V (GPT-4 with Vision), Qwen2.5-VL, etc.

[0036] In one specific embodiment, the sample description text can be the natural language description text corresponding to the current sample vehicle image. Optionally, the current sample vehicle image can be input into a preset multimodal model, and the sample description text corresponding to the current sample vehicle image can be obtained based on preset guiding words. For example, the current sample vehicle image can be as follows: Figure 3As shown, specifically, the preset guiding words can be: "1. First, determine whether there are traffic participants such as vehicles, non-motorized vehicles, and irregularly shaped vehicles in the picture; if so, further determine the type; 2. Please answer the following questions respectively: vehicle type (e.g., crane, pump truck, dump truck, concrete mixer truck, aerial work platform, ordinary truck, etc.); load information (e.g., sand and gravel, steel, concrete, equipment, empty, etc.); status information (e.g., driving, working, etc.); environmental information (e.g., highway, urban road, construction site, rural road, mining area, etc.); 3. Each piece of information should be described using a single sentence of natural language, fully describing the relevant information. Please output in JSON format, for example: {"exists":, "target information": {"vehicle type": "", "load information": "", "status information": "", "environmental information": "",}}", and the corresponding sample description text can be: "{"exists": true, "target information": {"vehicle type": "This is an ordinary truck used to transport various goods." "Load Information": "The truck bed is loaded with a large amount of timber (branches), stacked high and extending beyond the edge of the bed." "Status Information": "The vehicle is stationary, possibly loading or unloading timber or temporarily parked." "Environment Information": "This scene is located beside a rural road, surrounded by trees and land, belonging to a rural road environment."

[0037] In step S205, the sample description text is parsed to obtain at least one word corresponding to the sample description text, the part-of-speech of at least one word, and the dependency relationship between at least one word. In a specific embodiment, at least one word segment can be at least one independent word corresponding to the sample description text. Taking the sample description text as "There are two workers in the basket of the aerial work platform, carrying out wire repair or installation work" as an example, optionally, the words that cannot be further divided in the sample description text can be identified by the natural language processing library spaCy, and part-of-speech tagging (POS tagging) and dependency parsing can be performed to obtain at least one word segment, the part-of-speech of at least one word segment, and the dependency relationship between at least one word segment. Specifically, at least one word segment can be [aerial work platform, of, basket, inside, have, two, people, workers, are, carrying out, wire, repair, or, installation, work].

[0038] In a specific embodiment, the part of speech of at least one word can be the grammatical attribute corresponding to at least one word, and the dependency relationship between at least one word can be the grammatical relationship between at least one word. For example, at least one word can be [aerial work vehicle, of, suspended basket, inside, have, two, name, staff, is, is, is, carrying out, wire, maintenance, or, installation, operation]. Specifically, the part of speech of "aerial work vehicle" is a noun, the part of speech of "maintenance" is a verb, and the dependency relationship between at least one word can be ["maintenance" is the object of "wire"], ["installation" is the object of "operation"], ["suspended basket" is the modifier of "aerial work vehicle"], etc.

[0039] In step S207, nouns are merged based on at least one word segment, the part-of-speech of at least one word segment, and the dependency relationship between at least one word segment to obtain the sample description tag corresponding to the sample description text; In a specific embodiment, the sample description tag can be a set of structured tags corresponding to the sample description text. Specifically, the sample description tag can be obtained by merging nouns from at least one word segment corresponding to the sample description text.

[0040] In an optional embodiment, the above-mentioned noun merging based on at least one word segment, the part-of-speech tag of at least one word segment, and the dependency relationship between at least one word segment to obtain the sample description tag corresponding to the sample description text may include: Based on the part-of-speech tag of at least one word, perform noun extraction processing on at least one word to obtain at least one noun corresponding to the sample description text; Based on the dependency relationship between at least one word segment, at least one noun is merged to obtain the sample description tag corresponding to the sample description text.

[0041] In a specific embodiment, at least one noun can be a noun word corresponding to at least one segment. For example, at least one segment can be [aerial work vehicle, of, suspended basket, inside, have, two, name, staff, is, carrying out, wire, maintenance, or, installation, operation]. Correspondingly, at least one noun can be [aerial work vehicle, suspended basket, name, staff, wire, maintenance, installation, operation].

[0042] In a specific embodiment, the merging of at least one noun can be achieved by combining dependency syntactic parsing. Specifically, it can be based on the dependency relationship between at least one segmentation to merge semantically related "verb + object" or "object + (nominalizable) verb" compound concepts. For at least one segmentation [aerial work vehicle, of, suspended basket, inside, have, two, name, staff, is, is, carrying out, wire, maintenance, or, installation, operation], the corresponding at least one noun can be [aerial work vehicle, suspended basket, inside, name, staff, wire, maintenance, installation, operation]. The object of "maintenance" is "wire", and "maintenance" itself can be nominalized. The object of "installation" is "operation", which can be merged into "wire maintenance" and "installation operation". Correspondingly, the sample description label can be [aerial work vehicle, suspended basket, inside, name, staff, wire maintenance, installation operation].

[0043] In the above embodiments, by extracting nominal components, worthless semantic information can be eliminated, and composite concepts can be merged to reduce redundant labels, improve label quality and model training efficiency. Furthermore, the labels of composite concepts can cover more business scenarios, and the model can identify more types of vehicle operation scenarios, greatly improving generalization ability.

[0044] In step S209, the sample description tags are filtered based on preset filter words to obtain the real description tags corresponding to the current sample vehicle image. In a specific embodiment, preset filter words can be used to filter sample description tags. The preset filter words can be pre-defined. Specifically, the preset filter words can be nouns without actual meaning, such as inside, outside, above, middle, name, individual, some, etc.

[0045] In a specific embodiment, the true description label can be a set of structured labels corresponding to the current sample vehicle image. The true description label can be obtained by filtering the sample description labels. Specifically, after filtering the sample description labels based on preset filter words, the true description labels can be obtained by sorting them based on the label length. Taking the sample description label as [aerial work vehicle, suspended platform, inside, name, worker, wire repair, installation work] as an example, the true description label can be [aerial work vehicle, suspended platform, worker, wire repair, installation work].

[0046] In step S2011, the label extraction model is trained based on the current sample vehicle image and the real description label to obtain the label extraction model; In one specific embodiment, the label extraction model to be trained can be a label extraction model to be trained, and the predicted description label can be used to indicate the description label predicted by the current label extraction model to be trained.

[0047] In an optional embodiment, the above-mentioned training of the label extraction model based on the current sample vehicle image and the real descriptive labels to obtain the label extraction model may include: The current sample vehicle image is input into the feature extraction layer of the label extraction model to be trained for feature extraction, and the current sample vehicle features corresponding to the current sample vehicle image are obtained. Input the current sample vehicle features into the cross-attention layer in the label extraction model to be trained, and perform cross-attention calculation based on at least one preset description label and the current sample vehicle features to obtain sample label indication data with at least one preset description label. Based on at least one sample label indication data and a preset indication data threshold, the predicted description label corresponding to the current sample vehicle image is obtained; The label extraction model is trained based on the predicted and true description labels to obtain the label extraction model.

[0048] In a specific embodiment, the feature extraction layer can be used to extract image features from the current sample vehicle image. The feature extraction layer can be a ViT (VisionTransformer). Specifically, the image size is first adjusted to 224×224, and then divided into 196 independent image blocks using 16×16 pixels. Each image block is first converted into a 256-dimensional vector, then input into the ViT encoder, and finally outputs 196×768-dimensional features. The feature vector of the j-th image block can be represented as... .

[0049] In a specific embodiment, such as Figure 4 As shown, the predicted description label can include four types of labels: vehicle body label, load label, status label, and environment label. For the i-th label... The labeled text is input into the pre-trained multimodal model CLIP (Contrastive Language-Image Pre-Training) to generate a d=768-dimensional feature vector. .

[0050] In a specific embodiment, the cross-attention layer can be used to perform cross-attention calculation on the current sample vehicle features and at least one preset descriptive label. The at least one preset label can be a preset label corresponding to the label extraction process. Optionally, the specific structure of the cross-attention layer can be a cross-attention mechanism. Specifically, the at least one preset descriptive label can be a query feature in the cross-attention mechanism, and the current sample vehicle features can be key features and value features. Correspondingly, decoding can be performed through the cross-attention mechanism decoding layer, outputting sample label indication data through a fully connected layer and a sigmoid layer. The formula for the cross-attention decoding layer is as follows: .

[0051] In a specific embodiment, the sample label indication data can be used to indicate the probability of each feature corresponding to a preset description label in the current sample vehicle image. Optionally, at least one sample label indication data can be obtained through a fully connected layer and a sigmoid. Specifically, taking at least one preset description label as [ordinary truck, aerial work platform, crane] and a preset indication data threshold of 0.9 as an example, the first output number can be preset to correspond to the preset description label as ordinary truck, the second output number to correspond to the preset description label as aerial work platform, and the third output number to correspond to the preset description label as crane. At least one sample label indication data can be [0.92, 0.23, 0.96], and the corresponding predicted description label can be [1, 0, 1].

[0052] In one specific embodiment, the label extraction model can be a model that has been trained in advance based on at least one sample image and at least one real descriptive label corresponding to the sample image. The current sample vehicle image is any one of the at least one sample image.

[0053] In one specific embodiment, training the label extraction model based on predicted and true description labels may include training with a preset loss function. Specifically, the preset loss function may be set as a binary cross-entropy loss function.

[0054] In the above embodiments, based on cross-attention, accurate semantic matching of vehicle images and descriptive labels is achieved, adapting to multi-label scenarios, improving the ability to identify special vehicle models not appearing in the training data, and enhancing generalization ability.

[0055] In an optional embodiment, the aforementioned current sample vehicle image can be any image from at least one sample image, and the method may further include: At least one preset description label is obtained based on the real description label corresponding to each of the at least one sample image.

[0056] In a specific embodiment, at least one preset description label may be a preset description label corresponding to the label extraction process, and at least one preset description label may be a set of real description labels corresponding to at least one sample image. Specifically, at least one preset description label can be obtained by deduplicating the real description labels corresponding to at least one sample image. Optionally, after deduplicating the real description labels corresponding to at least one sample image, manual screening can also be performed to obtain at least one preset description label.

[0057] In the above embodiments, by deduplication, redundant and noisy labels can be removed, simplifying the label data, avoiding redundant learning, reducing the amount of computation during training, accelerating the convergence speed, improving the training efficiency of the model, and reducing the training cost of the model.

[0058] In an optional embodiment, Figure 5 This is a flowchart illustrating a vehicle tag extraction method according to an exemplary embodiment. This vehicle tag extraction method is executed based on any of the aforementioned tag extraction models and is used in a server, such as... Figure 5 As shown, it includes the following steps: In step S501, the image to be identified is acquired; In one specific embodiment, the image to be identified may include at least one object to be identified. Specifically, the image to be identified may be an environmental image of the vehicle's surroundings in a driving scenario, and correspondingly, the vehicle object in the environmental image needs to be identified.

[0059] In step S503, the image to be identified is input into a preset recognition model, and recognition is performed based on at least one preset vehicle category to determine at least one vehicle object to be identified. In one specific embodiment, a preset recognition model can be used to identify vehicle objects in an image to be recognized, at least one preset vehicle category can be a preset vehicle category corresponding to the recognition process, and at least one vehicle object to be recognized can be a vehicle object to be recognized among at least one object to be recognized.

[0060] In step S505, based on the bounding box information and the image to be identified for each vehicle object in at least one vehicle object to be identified, the image to be identified for each vehicle object is determined. In a specific embodiment, the bounding box information can characterize the position and range of the vehicle object to be identified in the image to be identified. The image of the vehicle to be identified can be the image corresponding to the vehicle object in the image to be identified. Specifically, the image of the vehicle to be identified can be obtained by combining the bounding box information.

[0061] In step S507, each vehicle image to be identified is input into the label extraction model for label extraction to obtain the target description label corresponding to each vehicle object to be identified.

[0062] In one specific embodiment, the target description label may be a set of structured labels corresponding to the vehicle object to be identified, and the target description label may be used to describe the characteristics of the vehicle object to be identified.

[0063] In the above embodiments, by identifying vehicle objects from the image to be identified and processing the cropped single vehicle image, interference from non-vehicle objects can be reduced, the amount of computation can be reduced, and the accuracy of label extraction can be improved.

[0064] In an optional embodiment, the above-described input of the image to be identified into a preset recognition model, and the identification based on at least one preset vehicle category, to determine at least one vehicle object to be identified may include: The image to be identified is input into a preset recognition model, and recognition is performed based on at least one preset vehicle category to obtain the bounding box information corresponding to each object to be identified and the confidence level of each object to be identified belonging to at least one preset vehicle category. Based on at least one confidence level corresponding to each object to be identified, at least one vehicle object to be identified is determined from at least one object to be identified.

[0065] In a specific embodiment, the confidence level of each object to be identified belonging to at least one preset vehicle category can characterize the credibility of each object to be identified belonging to each preset vehicle category. At least one vehicle to be identified can be identified through open set object detection. Specifically, based on at least one confidence level corresponding to each object to be identified, an upper limit value of the confidence level corresponding to each object to be identified can be obtained. Based on a preset confidence threshold and the upper limit value of the confidence level corresponding to each object to be identified, at least one vehicle object to be identified can be determined from at least one object to be identified. The preset confidence threshold can be preset. Optionally, a lower threshold can be set so that only objects to be identified whose confidence level reaches the preset confidence threshold are identified as vehicles to be identified.

[0066] In the above embodiments, by setting a threshold, invalid noise is filtered out, potential vehicles in open set detection scenarios are identified, the problem of poor generalization ability of closed set detection is solved, and a high recall rate is achieved, without missing any potential vehicles.

[0067] In an optional embodiment, the above-described method of inputting each vehicle image to be identified into a label extraction model for label extraction, to obtain a target description label corresponding to each vehicle object to be identified, may include: Each vehicle image to be identified is input into the feature extraction layer of the label extraction model for feature extraction, and the vehicle features corresponding to each vehicle object to be identified are obtained. Each feature of the vehicle to be identified is input into the cross-attention layer of the label extraction model. Cross-attention calculation is performed based on at least one preset description label and each feature of the vehicle to be identified to obtain label indication data for at least one preset description label corresponding to each image of the vehicle to be identified. Based on at least one label indication data corresponding to each vehicle image to be identified and a preset indication data threshold, a target description label corresponding to each vehicle object to be identified is obtained.

[0068] In one specific embodiment, the vehicle feature to be identified can be the image feature corresponding to the image of the vehicle to be identified.

[0069] In a specific embodiment, the cross-attention layer can be used to perform cross-attention calculation on the vehicle features to be identified and at least one preset descriptive label. Optionally, the specific structure of the cross-attention layer can be a cross-attention mechanism. Specifically, the at least one preset descriptive label can be a query feature in the cross-attention mechanism, and the vehicle features to be identified can be key features and value features. Correspondingly, the cross-attention mechanism can be decoded through a decoding layer, and the label indication data can be output through a fully connected layer and a sigmoid layer.

[0070] In one specific embodiment, the label indication data can be used to indicate the probability that the image of the vehicle to be identified includes a feature corresponding to each of the at least one preset description label. Optionally, at least one label indication data can be obtained through a fully connected layer and a Sigmoid.

[0071] In the above embodiments, based on cross-attention, the accurate matching of the features of the vehicle to be identified and the preset descriptive labels is achieved, which is suitable for multi-label scenarios and can identify special vehicle models that do not appear in the training data, thereby improving the accuracy of vehicle identification.

[0072] As can be seen from the technical solutions provided in the embodiments of this specification above, this specification acquires a current sample vehicle image; inputs the current sample vehicle image into a preset multimodal model to obtain sample description text corresponding to the current sample vehicle image; performs syntactic parsing processing on the sample description text to obtain at least one word segment, the part-of-speech tag of at least one word segment, and the dependency relationship between at least one word segment; merges nouns based on at least one word segment, the part-of-speech tag of at least one word segment, and the dependency relationship between at least one word segment to obtain sample description tags corresponding to the sample description text; filters the sample description tags based on preset filter words to obtain the real description tags corresponding to the current sample vehicle image; and trains the tag extraction model to be trained based on the current sample vehicle image and the real description tags to obtain the tag extraction model. This model can meet the multi-dimensional needs of vehicle recognition, adapt to multi-tag scenarios, greatly improve the readability of vehicle recognition results, and thus reduce decision-making errors.

[0073] Figure 6 This is a block diagram illustrating a label extraction model training apparatus according to an exemplary embodiment. (Refer to...) Figure 6 The device includes: The current sample vehicle image acquisition module 610 is used to acquire the current sample vehicle image; The sample description text acquisition module 630 is used to input the current sample vehicle image into a preset multimodal model to obtain the sample description text corresponding to the current sample vehicle image; The syntax parsing processing module 650 is used to perform syntax parsing processing on the sample description text to obtain at least one word corresponding to the sample description text, the part-of-speech of at least one word, and the dependency relationship between at least one word. The noun merging module 670 is used to merge nouns based on at least one word segment, the part-of-speech of at least one word segment, and the dependency relationship between at least one word segment to obtain the sample description tags corresponding to the sample description text. The filtering processing module 690 is used to filter the sample description tags based on preset filter words to obtain the real description tags corresponding to the current sample vehicle image. The model training module 6110 is used to train the label extraction model to be trained based on the current sample vehicle image and the real description label, so as to obtain the label extraction model.

[0074] In an optional embodiment, the model training module 6110 includes: The first feature extraction unit is used to input the current sample vehicle image into the feature extraction layer of the label extraction model to be trained for feature extraction, and obtain the current sample vehicle features corresponding to the current sample vehicle image. The first cross-attention calculation unit is used to input the current sample vehicle features into the cross-attention layer of the label extraction model to be trained, and to perform cross-attention calculation based on at least one preset description label and the current sample vehicle features to obtain sample label indication data of at least one preset description label; the sample label indication data is used to indicate the probability that the current sample vehicle image includes the feature corresponding to each preset description label in at least one preset description label. The predictive description label acquisition unit is used to obtain the predictive description label corresponding to the current sample vehicle image based on at least one sample label indication data and a preset indication data threshold. The model training unit is used to train the label extraction model to be trained based on the predicted description labels and the real description labels, so as to obtain the label extraction model.

[0075] In an optional embodiment, the current sample vehicle image is any one of at least one sample image, and the above-described apparatus further includes: The preset description label acquisition module is used to obtain at least one preset description label based on the real description label corresponding to at least one sample image.

[0076] In an optional embodiment, the noun merging module 670 includes: The noun extraction processing unit is used to extract nouns from at least one word based on the part-of-speech tag of at least one word segment to obtain at least one noun corresponding to the sample description text. The noun merging processing unit is used to merge at least one noun based on the dependency relationship between at least one word segment to obtain the sample description tag corresponding to the sample description text.

[0077] Figure 7 This is a block diagram illustrating a vehicle tag extraction device according to an exemplary embodiment. (Refer to...) Figure 7 The device includes: The image acquisition module 710 is used to acquire the image to be identified; the image to be identified includes at least one object to be identified. The vehicle object determination module 730 is used to input the image to be identified into a preset recognition model, perform recognition based on at least one preset vehicle category, and determine at least one vehicle object to be identified. The vehicle image determination module 750 is used to determine the vehicle image corresponding to each vehicle object based on the bounding box information and the image to be identified for each vehicle object in at least one vehicle object to be identified. The label extraction module 770 is used to input each vehicle image to be identified into the label extraction model for label extraction, and obtain the target description label corresponding to each vehicle object to be identified.

[0078] In an optional embodiment, the vehicle object determination module 730 includes: The vehicle category recognition unit is used to input the image to be recognized into a preset recognition model, recognize it based on at least one preset vehicle category, and obtain the bounding box information corresponding to each object to be recognized and the confidence level of each object to be recognized belonging to at least one preset vehicle category. The vehicle object determination unit is used to determine at least one vehicle object from at least one object to be identified based on at least one confidence level corresponding to each object to be identified.

[0079] In an optional embodiment, the tag extraction module 770 includes: The second feature extraction unit is used to input each vehicle image to be identified into the feature extraction layer in the label extraction model for feature extraction, and obtain the vehicle features corresponding to each vehicle object to be identified. The second cross-attention calculation unit is used to input each vehicle feature to be identified into the cross-attention layer in the label extraction model, and to perform cross-attention calculation based on at least one preset description label and each vehicle feature to be identified, to obtain label indication data of at least one preset description label corresponding to each vehicle image to be identified; the label indication data is used to indicate the probability that the vehicle image to be identified includes the feature corresponding to each preset description label in at least one preset description label. The target description label acquisition unit is used to obtain the target description label corresponding to each vehicle object to be identified based on at least one label indication data corresponding to each vehicle image to be identified and a preset indication data threshold.

[0080] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0081] Figure 8 This is a block diagram illustrating an electronic device for training a label extraction model according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a label extraction model training method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse. Figure 9 This is a block diagram illustrating an electronic device for training a label extraction model according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a label extraction model training method. Those skilled in the art will understand that Figure 8 or Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements. In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the label extraction model training method as described in the embodiments of this disclosure.

[0082] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the label extraction model training method of the present disclosure embodiments.

[0083] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0084] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training a label extraction model, characterized in that, include: Obtain the image of the current sample vehicle; Input the current sample vehicle image into a preset multimodal model to obtain the sample description text corresponding to the current sample vehicle image; The sample description text is parsed to obtain at least one word segment corresponding to the sample description text, the part-of-speech of the at least one word segment, and the dependency relationship between the at least one word segment; Based on the at least one word segment, the part-of-speech of the at least one word segment, and the dependency relationship between the at least one word segment, nouns are merged to obtain the sample description tags corresponding to the sample description text; Based on preset filter words, the sample description tags are filtered to obtain the real description tags corresponding to the current sample vehicle image. The label extraction model is trained based on the current sample vehicle image and the real description label to obtain the label extraction model.

2. The label extraction model training method according to claim 1, characterized in that, The process of training the label extraction model based on the current sample vehicle image and the real description label to obtain the label extraction model includes: The current sample vehicle image is input into the feature extraction layer of the label extraction model to be trained for feature extraction, and the current sample vehicle features corresponding to the current sample vehicle image are obtained. The current sample vehicle features are input into the cross-attention layer of the label extraction model to be trained. Cross-attention calculation is performed based on at least one preset description label and the current sample vehicle features to obtain sample label indication data of the at least one preset description label. The sample label indication data is used to indicate the probability that the current sample vehicle image includes the feature corresponding to each preset description label in the at least one preset description label. Based on the at least one sample label indication data and a preset indication data threshold, the predicted description label corresponding to the current sample vehicle image is obtained; The label extraction model is trained based on the predicted description labels and the true description labels to obtain the label extraction model.

3. The label extraction model training method according to claim 2, characterized in that, The current sample vehicle image is any one of at least one sample image, and the method further includes: Based on the real description labels corresponding to each of the at least one sample image, the at least one preset description label is obtained.

4. The label extraction model training method according to claim 1, characterized in that, The process of merging nouns based on the at least one word segment, the part-of-speech tag of the at least one word segment, and the dependency relationship between the at least one word segment to obtain the sample description tag corresponding to the sample description text includes: Based on the part-of-speech tag of the at least one word segment, noun extraction processing is performed on the at least one word segment to obtain at least one noun corresponding to the sample description text; Based on the dependency relationship between the at least one word segment, the at least one noun is merged to obtain the sample description tag corresponding to the sample description text.

5. A vehicle tag extraction method based on the tag extraction model according to any one of claims 1 to 4, characterized in that, include: Acquire the image to be recognized; The image to be identified includes at least one object to be identified; The image to be identified is input into a preset recognition model, and recognition is performed based on at least one preset vehicle category to determine at least one vehicle object to be identified. Based on the bounding box information corresponding to each of the at least one vehicle objects to be identified and the image to be identified, determine the vehicle image to be identified corresponding to each vehicle object to be identified; Each vehicle image to be identified is input into the label extraction model for label extraction, thereby obtaining the target description label corresponding to each vehicle object to be identified.

6. The vehicle tag extraction method according to claim 5, characterized in that, The step of inputting the image to be identified into a preset recognition model, and identifying at least one vehicle object based on at least one preset vehicle category includes: The image to be identified is input into a preset recognition model, and recognition is performed based on at least one preset vehicle category to obtain the bounding box information corresponding to each of the at least one objects to be identified and the confidence level of each object to be identified belonging to the at least one preset vehicle category. Based on at least one confidence level corresponding to each object to be identified, the at least one vehicle object to be identified is determined from the at least one object to be identified.

7. The vehicle tag extraction method according to claim 5, characterized in that, The step of inputting each vehicle image to be identified into the label extraction model for label extraction to obtain the target description label corresponding to each vehicle object to be identified includes: Each vehicle image to be identified is input into the feature extraction layer of the label extraction model for feature extraction, thereby obtaining the vehicle features corresponding to each vehicle object to be identified. Each feature of the vehicle to be identified is input into the cross-attention layer of the label extraction model. Cross-attention calculation is performed based on at least one preset descriptive label and each feature of the vehicle to be identified to obtain label indication data for at least one preset descriptive label corresponding to each image of the vehicle to be identified. The label indication data is used to indicate the probability that the image of the vehicle to be identified includes the feature corresponding to each preset descriptive label in the at least one preset descriptive label. Based on at least one label indication data corresponding to each vehicle image to be identified and a preset indication data threshold, a target description label corresponding to each vehicle object to be identified is obtained.

8. A label extraction model training device, characterized in that, include: The current sample vehicle image acquisition module is used to acquire the current sample vehicle image; The sample description text acquisition module is used to input the current sample vehicle image into a preset multimodal model to obtain the sample description text corresponding to the current sample vehicle image; The syntax parsing processing module is used to perform syntax parsing processing on the sample description text to obtain at least one word corresponding to the sample description text, the part-of-speech of the at least one word, and the dependency relationship between the at least one word; The noun merging module is used to merge nouns based on the at least one word segment, the part-of-speech of the at least one word segment, and the dependency relationship between the at least one word segment, to obtain the sample description tag corresponding to the sample description text; The filtering module is used to filter the sample description tags based on preset filter words to obtain the real description tags corresponding to the current sample vehicle image. The model training module is used to train the label extraction model to be trained based on the current sample vehicle image and the real description label, so as to obtain the label extraction model.

9. A vehicle tag extraction device, characterized in that, include: The image acquisition module is used to acquire the image to be recognized. The image to be identified includes at least one object to be identified; The vehicle object determination module is used to input the image to be identified into a preset recognition model, perform recognition based on at least one preset vehicle category, and determine at least one vehicle object to be identified. The vehicle image determination module is used to determine the vehicle image corresponding to each vehicle object based on the bounding box information corresponding to each vehicle object in the at least one vehicle object to be identified and the image to be identified. The label extraction module is used to input each vehicle image to be identified into the label extraction model to extract labels and obtain the target description label corresponding to each vehicle object to be identified.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the label extraction model training method as described in any one of claims 1 to 7.

Images