Fake-licensed vehicle identification method and device, model training method and device and electronic equipment

Abstract

The embodiment of the invention provides a fake-licensed vehicle identification method and device, a model training method and device and electronic equipment. According to the fake-licensed vehicle identification method, multiple vehicle task features are extracted from a vehicle image by using a multi-task feature extraction model; converting the vehicle text information to obtain vehicle text features; fusing the various vehicle task features with the vehicle text features by using a feature fusion model to generate multi-modal fusion features; and determining whether there is a fake-licensed relationship between the two vehicles based on a comparison result of the two multi-modal fusion features. According to the method, various vehicle task features extracted from the vehicle image are utilized, and the vehicle text features are introduced, so that the deficiency of a pure vision model in semantic understanding is made up, and the recognition precision is improved. In the training process of the multi-task feature extraction model, when center loss variants are calculated, inter-class separation loss is introduced to constrain feature mapping, interference and redundancy between tasks are reduced, and the discrimination ability of feature expression and task specificity are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a method for identifying vehicles using counterfeit license plates, a model training method, a device, and electronic equipment. Background Technology

[0002] A vehicle license plate number is a vehicle's unique identifier. Vehicle counterfeiting, such as forging, altering, or using someone else's license plate, is a common traffic violation. Traditional methods for identifying counterfeit vehicles rely on on-site checks by traffic police or manual comparison of surveillance data, which suffers from low efficiency and data silos.

[0003] With the widespread adoption of intelligent transportation systems, infrastructure such as cameras, checkpoints, and ETC (Electronic Toll Collection) provides comprehensive coverage and all-weather data collection capabilities. Simultaneously, the development of deep learning has driven improvements in the accuracy of computer vision technologies, such as license plate OCR (Optical Character Recognition) and vehicle feature extraction, as well as enhanced big data analysis capabilities. Against this backdrop, cloned license plate detection technology has developed rapidly, significantly improving accuracy and efficiency. However, some challenges still exist in certain specialized scenarios.

[0004] A key characteristic of deep learning-based vehicle clone detection technology is feature comparison, whose performance is highly dependent on the richness and diversity of the training data. However, in reality, due to limitations in data collection, variations in lighting conditions across different scenes, differences in vehicle posture, and occlusion, the training data often fails to cover all possible complex situations, which may lead to a decline in the model's generalization ability in real-world environments. Furthermore, cloned vehicles themselves are highly concealed and deceptive, employing methods such as forged license plates, similar fonts, or color imitations, further increasing the difficulty of identification and requiring further improvement in accuracy.

[0005] In some related technologies, image processing, pattern recognition, and machine learning techniques are used to extract vehicle features such as license plate numbers and vehicle models from vehicle images. The similarity of these features is then compared to identify vehicles using counterfeit license plates. However, this approach relies on the independent use of different techniques for extracting vehicle features from images, resulting in poor comparison results.

[0006] In other related technologies, AI is used to supplement missing data and then perform Re-Identification. However, the features generated by AI generative technology differ from the real features, and the noise generated can affect the high accuracy and generalization ability of the model, making it difficult to guarantee recognition accuracy. Summary of the Invention

[0007] The purpose of this application is to provide a method, model training method, device, and electronic device for identifying cloned vehicles, so as to improve the recognition accuracy. The specific technical solution is as follows:

[0008] This application provides a method for identifying vehicles using counterfeit license plates, including:

[0009] Obtain two vehicle images to be identified and two corresponding vehicle text information; each vehicle text information includes at least: a license plate number; and the license plate numbers in the two vehicle text information are the same;

[0010] The two vehicle images and their corresponding two vehicle text information are processed separately, including: inputting the vehicle images into a multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; converting the vehicle text information corresponding to the vehicle images to obtain vehicle text features; concatenating the multiple vehicle task features and vehicle text features, and then inputting them into a feature fusion model for fusion to generate multimodal fusion features;

[0011] The multimodal fusion features corresponding to the two vehicle images to be identified are compared. If the multimodal fusion features are different, it is determined that the vehicles in the two vehicle images are not the same vehicle and there is a license plate clone relationship between the two vehicles.

[0012] In some embodiments, the multi-task feature extraction model includes: a multi-scale feature extraction sub-network and multiple task feature mapping sub-networks, wherein each task feature mapping sub-network corresponds to a task;

[0013] The process involves inputting vehicle images into a multi-task feature extraction model to obtain various vehicle task features, including:

[0014] Input the vehicle image into the multi-scale feature extraction subnetwork;

[0015] The multi-scale feature extraction subnetwork extracts features from vehicle images and outputs multi-scale features to the feature mapping subnetworks for each task.

[0016] Each task feature mapping subnetwork maps multi-scale features to their corresponding task features.

[0017] In some embodiments, the feature fusion model is obtained by training the multi-task feature extraction model after it has been trained.

[0018] In some embodiments, the multi-task feature extraction model is implemented by adding multiple fully connected layers to the CNN+FPN visual model; each fully connected layer is used to map a task feature.

[0019] The feature fusion model is a ReID model with Transformer as its core architecture.

[0020] This application also provides a training method for a multi-task feature extraction model, including:

[0021] Obtain the original model of the multi-task feature extraction model, the original model including: a multi-scale feature extraction sub-network, multiple task feature mapping sub-networks and multiple task heads corresponding one-to-one with the multiple task feature mapping sub-networks, wherein each task feature mapping sub-network corresponds to a task.

[0022] Obtain multiple sample vehicle images and the ground truth value of each task feature for each sample vehicle image; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features;

[0023] Vehicle images are input into the original model for processing to obtain recognition results for various tasks, including: inputting sample vehicle images into a multi-scale feature extraction sub-network; the multi-scale feature extraction sub-network extracts features from the sample vehicle images and outputs sample multi-scale features to various task feature mapping sub-networks; each task feature mapping sub-network maps the multi-scale features to their corresponding task features and outputs them to the corresponding task head; each task head outputs the recognition result for its task based on the task features.

[0024] The calculation of the total model loss includes: calculating the task loss of each task based on the recognition results of each task and the ground truth of each task; calculating the inter-class separation loss of each task center; calculating the center loss variant based on the inter-class separation loss; and calculating the total model loss based on the center loss variant and the loss of each task.

[0025] Based on the total loss of the model, the network parameters of each task feature mapping sub-network are adjusted, and the task feature mapping sub-networks are jointly trained.

[0026] If the total loss of the model satisfies the convergence condition, each task head in the original model is deleted to obtain the trained multi-task feature extraction model.

[0027] In some embodiments, the calculation of the center loss variant based on inter-class separation loss includes: calculating the center loss variant based on the following preset feature map center loss variant formula. :

[0028]

[0029]

[0030]

[0031] in, It is a multi-scale feature mapping to the task Feature vectors in the feature space It is a task The corresponding task center, It's the number of tasks. It is a central loss. It is the inter-class separation loss; These are preset hyperparameters.

[0032] In some embodiments, calculating the total model loss based on the center loss variant and the losses of each task includes: calculating the total model loss based on the following preset formula for total model loss. :

[0033]

[0034] in, The task loss for the i-th task; The task loss weight for the k-th task is calculated based on the following formula:

[0035]

[0036] in, This represents the loss of the k-th task; The rate of change of the task loss; Represents multi-scale feature vectors; The norm for multi-scale features; This refers to temperature hyperparameters. Let L2 norm represent the feature after the k-th branch mapping; Let L2 norm represent the feature after the j-th branch is mapped; As the reference norm, the mean of the norms of all branches is taken; and These are the preset hyperparameters.

[0037] In some embodiments, the multi-task feature extraction model is implemented by adding multiple fully connected layers to the CNN+FPN visual model; each fully connected layer is used to map a task feature.

[0038] This application also provides a method for training a feature fusion model, including:

[0039] Obtain the original feature fusion model;

[0040] Multiple sets of training samples are obtained. Each set of training samples includes two sample vehicle images and corresponding text information for the two sample vehicles. The text information for the sample vehicles includes at least one license plate number. In some sets of training samples, the two sample vehicle images are not of the same vehicle. In other sets of training samples, the two sample vehicle images are of the same vehicle or a vehicle with a fake license plate, and the license plate numbers in the text information of the two sample vehicles are the same. The result of whether the two sample vehicle images are of the same vehicle is taken as the ground truth of that set of training samples.

[0041] Each set of samples is processed separately, including: inputting the sample vehicle images into a trained multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; converting the vehicle text information to obtain vehicle text features; concatenating the multiple vehicle task features with the vehicle text features and inputting them into the original feature fusion model for fusion to generate multimodal fusion features;

[0042] By comparing the multimodal fusion features corresponding to the two sample vehicle images in this set of samples, the sample comparison results of whether the vehicles in the two sample vehicle images are the same vehicle are obtained.

[0043] The sample comparison results of this set of samples are compared with the ground truth. If the comparison results are the same, the feature fusion model training is completed. If the comparison results are different, the network parameters of the feature fusion model are adjusted and training continues.

[0044] In some embodiments, the feature fusion model is a ReID model with Transformer as its core architecture.

[0045] This application also provides a device for identifying vehicles using counterfeit license plates, including:

[0046] The acquisition module is used to acquire two vehicle images to be identified and two corresponding vehicle text information; each vehicle text information includes at least: a license plate number; and the license plate numbers in the two vehicle text information are the same;

[0047] The multimodal feature generation module is used to process two vehicle images and their corresponding two vehicle text information separately: the vehicle images are input into a multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; the vehicle text information corresponding to the vehicle images is converted to obtain vehicle text features; the multiple vehicle task features and vehicle text features are concatenated and then input into a feature fusion model for fusion to generate multimodal fusion features;

[0048] The feature comparison module is used to compare the multimodal fusion features corresponding to the two vehicle images to be identified. If the multimodal fusion features are different, it is determined that the vehicles in the two vehicle images are not the same vehicle and there is a license plate clone relationship between the two vehicles.

[0049] This application also provides a training apparatus for a multi-task feature extraction model, including:

[0050] The original model acquisition module is used to obtain the original model of the multi-task feature extraction model. The original model includes: a multi-scale feature extraction sub-network, multiple task feature mapping sub-networks, and multiple task heads that correspond one-to-one with the multiple task feature mapping sub-networks, wherein each task feature mapping sub-network corresponds to a task.

[0051] The sample image acquisition module is used to acquire multiple sample vehicle images and the ground truth value of each task feature of each sample vehicle image; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features;

[0052] The original model processing module is used to input vehicle images into the original model and obtain recognition results for various tasks, including: inputting sample vehicle images into a multi-scale feature extraction sub-network; the multi-scale feature extraction sub-network extracts features from the sample vehicle images and outputs sample multi-scale features to various task feature mapping sub-networks; each task feature mapping sub-network maps the multi-scale features to their corresponding task features and outputs them to the corresponding task head; each task head outputs the recognition result for this task based on the task features.

[0053] The model total loss calculation module is used to calculate the model total loss, including: calculating the task loss of each task based on the recognition results of each task and the ground truth of each task; calculating the inter-class separation loss of each task center; calculating the center loss variant based on the inter-class separation loss; and calculating the model total loss based on the center loss variant and the loss of each task.

[0054] The parameter adjustment module is used to adjust the network parameters of each task feature mapping sub-network based on the total loss of the model, and to jointly train each task feature mapping sub-network.

[0055] The task head deletion module is used to delete each task head in the original model when the total loss of the model meets the convergence condition, so as to obtain the trained multi-task feature extraction model.

[0056] This application embodiment also provides a training device for a feature fusion model, including:

[0057] The original feature fusion model acquisition module is used to obtain the original feature fusion model;

[0058] The training sample acquisition module is used to acquire multiple sets of training samples. Each set of training samples includes: two sample vehicle images and corresponding text information for two sample vehicles; the text information for the sample vehicles includes at least: license plate number; in the multiple sets of training samples, the two sample vehicle images in some training samples are not of the same vehicle; in other training samples, the two sample vehicle images are of the same vehicle or a vehicle with a cloned license plate, and the license plate number in the text information of the two sample vehicles is the same; the result of whether the vehicles in the two sample vehicle images are the same vehicle is taken as the ground truth of that set of training samples;

[0059] The sample processing module is used to perform the following processing for each group of samples: inputting the sample vehicle image into a trained multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; converting the vehicle text information to obtain vehicle text features; concatenating the multiple vehicle task features with the vehicle text features and inputting them into the original feature fusion model for fusion to generate multimodal fusion features;

[0060] The sample feature comparison module compares the multimodal fusion features corresponding to the two sample vehicle images in the group of samples to obtain the sample comparison result of whether the vehicles in the two sample vehicle images are the same vehicle.

[0061] The parameter adjustment module is used to compare the sample comparison results with the ground truth based on the sample set. If the comparison results are the same, the feature fusion model training is complete. If the comparison results are different, the network parameters of the feature fusion model are adjusted and training continues.

[0062] This application also provides an electronic device, including:

[0063] Memory, used to store computer programs;

[0064] When a processor executes a program stored in memory, it implements the above-mentioned method for identifying cloned vehicles; or implements the above-mentioned training method for the multi-task feature extraction model; or implements the above-mentioned training method for the feature fusion model.

[0065] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method for identifying cloned vehicles; or implements the above-described method for training a multi-task feature extraction model; or implements the above-described method for training a feature fusion model.

[0066] Beneficial effects of the embodiments in this application:

[0067] This application provides a method, model training method, apparatus, and electronic device for identifying cloned license plates. The cloned license plate identification method processes two vehicle images and their corresponding text information separately: a multi-task feature extraction model is used to extract various vehicle task features from the vehicle images; the vehicle text information is transformed to obtain vehicle text features; and a feature fusion model is used to fuse the various vehicle task features with the vehicle text features to generate a multimodal fusion feature. If the comparison results of the two multimodal fusion features are different, it is determined that the vehicles in the two vehicle images are not the same vehicle, and a cloned license plate relationship exists between the two vehicles. In other words, in the process of identifying cloned license plates, not only are various vehicle task features extracted from the vehicle images utilized, but vehicle text features are also introduced, compensating for the shortcomings of pure visual models in semantic understanding and improving the accuracy of identification.

[0068] Furthermore, during the training process of the multi-task feature extraction model in this application embodiment, when calculating the center loss variant, inter-class separation loss is introduced to constrain the feature mapping, which reduces interference and redundancy between tasks and improves the discriminative ability and task specificity of feature representation.

[0069] In addition, in some embodiments, a dynamic weight adjustment mechanism is proposed during the training process of the multi-task feature extraction model. Based on the preset task loss weight adjustment formula, the weights of different tasks in training are adaptively adjusted according to the real-time loss or uncertainty of each task, thereby effectively alleviating the conflict and imbalance between multiple tasks and improving the generalization performance and convergence stability of the overall model.

[0070] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0072] Figure 1 This is a flowchart illustrating a method for identifying cloned vehicles, as provided in this application embodiment.

[0073] Figure 2 This is a schematic diagram illustrating the principle of a parallel processing example of a method for identifying cloned vehicles, as provided in the embodiments of this application.

[0074] Figure 3 This is a schematic diagram of an example feature fusion model in the embodiments of this application.

[0075] Figure 4 This is a flowchart illustrating the overall process of training the multi-task feature extraction model and feature fusion model in the embodiments of this application.

[0076] Figure 5 This is a flowchart illustrating the training method for the multi-task feature extraction model provided in this application embodiment.

[0077] Figure 6 This is a schematic diagram of the original model of the multi-task feature extraction model in the embodiments of this application.

[0078] Figure 7 This is a flowchart illustrating the training method for the feature fusion model provided in this application embodiment.

[0079] Figure 8 This is a schematic diagram of the device for identifying cloned vehicles provided in an embodiment of this application.

[0080] Figure 9 This is a schematic diagram of the structure of the training device for the multi-task feature extraction model provided in the embodiments of this application.

[0081] Figure 10 This is a schematic diagram of the structure of the training device for the feature fusion model provided in the embodiments of this application.

[0082] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0083] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0084] To improve the accuracy of identifying cloned vehicles, embodiments of this application provide a method for identifying cloned vehicles, a model training method, a device, and an electronic device. These will be described in detail below.

[0085] See Figure 1 , Figure 1 This is a flowchart illustrating the method for identifying cloned vehicles provided in an embodiment of this application. Figure 1 As shown, the process includes:

[0086] Step S100: Obtain two vehicle images to be identified and two corresponding vehicle text information; the vehicle text information includes at least: license plate number; and the license plate number in the two vehicle text information is the same;

[0087] Step S110 involves processing the two vehicle images and their corresponding text information, including:

[0088] Step S111: Input the vehicle image into the multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features;

[0089] Step S112: Convert the vehicle text information to obtain vehicle text features;

[0090] Step S113: After concatenating multiple vehicle task features with vehicle text features, input them into the feature fusion model for fusion to generate multimodal fusion features;

[0091] Step S120: Compare the multimodal fusion features corresponding to the two vehicle images to be identified. If the multimodal fusion features are different, determine that the vehicles in the two vehicle images are not the same vehicle and that there is a license plate clone relationship between the two vehicles.

[0092] Depend on Figure 1 As can be seen from the embodiments shown, the cloned vehicle identification method provided in this application not only utilizes various vehicle task features extracted from vehicle images during the cloned vehicle identification process, but also introduces vehicle text features, which makes up for the shortcomings of pure visual models in semantic understanding, enhances the distinguishability and comparison ability of vehicle features, effectively improves the robustness of the model under noise interference and abnormal conditions, and improves the accuracy of identification.

[0093] The application scenario of this application can be an intelligent transportation system. The vehicle image can be a vehicle image captured by a camera in the intelligent transportation system. The vehicle text information can be the vehicle text information directly identified from the vehicle image after image processing and structured processing, mainly including the license plate number, and may also include the vehicle brand name. Due to limitations in data acquisition, changes in lighting in different scenarios, differences in vehicle posture, and occlusion, the vehicle text information directly identified from the vehicle image may not be completely accurate. Therefore, in cases where the license plate number identified from two vehicle images is the same, the information can be further analyzed by... Figure 1 The process shown determines whether the vehicles in the two vehicle images are the same vehicle, and thus determines whether there is a license plate clone relationship between the two vehicles.

[0094] In this embodiment of the application, the two vehicle images and the corresponding two vehicle text information are processed separately, which can be done in parallel or in sequence.

[0095] The following section uses parallel processing as an example to explain the above method for identifying cloned vehicles in detail.

[0096] See Figure 2 This is a schematic diagram illustrating the principle of a parallel processing example for a method to identify cloned vehicles, as provided in this application. Figure 2 As shown, the parallel processing process includes:

[0097] Vehicle image 1 is input into multi-task feature extraction model 1; multi-task feature extraction model 1 extracts features and outputs multiple vehicle task features 1 to feature concatenation module 1; vehicle text information 1 is input into text information conversion module 1; text information conversion module 1 converts the text information into discrete symbol sequences (tokens) and maps them into dense vector representations through an embedding layer, which are used as vehicle text features 1 and output to feature concatenation module 1; feature concatenation module 1 concatenates multiple vehicle task features 1 with vehicle text features 1 and outputs the concatenated features to feature fusion model 1; feature fusion model 1 fuses the concatenated features to generate multimodal fusion features 1 and outputs them to feature comparison module.

[0098] After undergoing the same processing, vehicle image 2 and vehicle text information 2 are fused by feature fusion model 2 to generate multimodal fusion feature 2, which is then output to feature comparison module.

[0099] The feature comparison module compares multimodal fusion feature 1 and multimodal fusion feature 2. If the two multimodal fusion features are different, it determines that the vehicles in the two vehicle images are not the same vehicle and that there is a license plate clone relationship between the two vehicles.

[0100] like Figure 2 As shown, in this embodiment, the multi-task feature extraction model 1 and the multi-task feature extraction model 2 have the same structure, the feature splicing module 1 and the feature splicing module 2 are the same, and the feature fusion model 1 and the feature fusion model 2 are the same.

[0101] The following explanation uses multi-task feature extraction model 1 as an example. Figure 2 As shown, the multi-task feature extraction model 1 includes a multi-scale feature extraction sub-network and multiple task feature mapping sub-networks, wherein each task feature mapping sub-network corresponds to a task.

[0102] Specifically, vehicle image 1 is input into a multi-scale feature extraction subnetwork; the multi-scale feature extraction subnetwork extracts features from the vehicle image and outputs multi-scale features to various task feature mapping subnetworks; each task feature mapping subnetwork maps the multi-scale features to its corresponding task features and outputs them to feature stitching module 1.

[0103] like Figure 2As shown, the multi-scale feature extraction sub-network in this embodiment may include a backbone network, such as a CNN (Convolutional Neural Network) and an FPN (Feature Pyramid Network). The CNN may be a ResNet (Residual Neural Network). Multiple task feature mapping sub-networks may be implemented using multiple fully connected layers.

[0104] First, the backbone network performs multi-level feature extraction on the input vehicle image, generating a basic feature map with rich semantic information. The basic features are then fed into a Feature Pyramid Network (FPN), which fuses multi-scale information through a top-down and laterally connected structure, generating a series of multi-scale feature maps with decreasing resolution, thereby simultaneously capturing the image's detailed features and global contextual information. In this embodiment, multiple fully connected layers are used, employing a center loss variant feature mapping to map the multi-scale features of the shared backbone network to sub-feature spaces for different tasks. The center loss variant introduces inter-class separation loss, preventing class centers from diverging infinitely, constraining them to a unit sphere, and maximizing the cosine distance. The specific formula for the center loss variant will be explained in detail during the subsequent model training process.

[0105] in addition, Figure 2 The feature fusion model in this context can be a ReID model with Transformer as its core architecture. See [link to ReID documentation] for the specific structure. Figure 3 , Figure 3 This is a schematic diagram illustrating an example of a feature fusion model in an embodiment of this application. Figure 3 As shown, this ReID model includes multiple layers of Transformer decoders (such as...). Figure 3 The ReID model uses Transformer 1 to Transformer n as inputs, concatenated features, and fused features as outputs. It leverages the powerful self-attention mechanism of the Transformer decoder to model the interaction relationships between different features, achieving deep semantic alignment and complementarity between visual features extracted from vehicle images and text features converted from text information. The output features, processed by multiple Transformer encoders, serve as the final multimodal fusion representation, exhibiting strong identity discrimination capabilities.

[0106] In some embodiments, a serial processing approach can be adopted, requiring only a multi-task feature extraction model, a text information conversion module, a feature concatenation module, a feature fusion model, and a feature comparison module.

[0107] Specifically, vehicle image 1 can be input into a multi-task feature extraction model to obtain multiple vehicle task features 1; vehicle text information 1 corresponding to vehicle image 1 can be input into a text information conversion module to obtain vehicle text features 1; multiple vehicle task features 1 and vehicle text features 1 are concatenated in a feature concatenation module, and the concatenated features are fused by a feature fusion model to obtain multimodal fusion features 1, which are then temporarily stored.

[0108] The vehicle image 2 is then input into the multi-task feature extraction model to obtain multiple vehicle task features 2; the vehicle text information 2 corresponding to the vehicle image 2 is input into the text information conversion module to obtain vehicle text features 2. The multiple vehicle task features 2 and vehicle text features 2 are then concatenated in the feature concatenation module. The concatenated features are then fused by the feature fusion model to obtain multimodal fusion features 2.

[0109] Finally, the temporarily stored multimodal fusion feature 1 and multimodal fusion feature 2 are input into the feature comparison module for feature comparison to obtain the comparison result of whether the two multimodal features are the same. This leads to the identification result of whether the vehicles in the two vehicle images are the same vehicle, and whether there is a license plate clone relationship between the two vehicles.

[0110] In this embodiment, the feature comparison module can calculate the Euclidean distance between multimodal fusion feature 1 and multimodal fusion feature 2. If the Euclidean distance is less than a preset threshold, the two multimodal fusion features are determined to be the same. Alternatively, the feature comparison module can calculate the cosine similarity between multimodal fusion feature 1 and multimodal fusion feature 2. If the cosine similarity is less than a preset threshold, the two multimodal fusion features are determined to be the same. In other embodiments, other algorithms can also be used for comparison, and this application embodiment does not impose any limitations.

[0111] In this embodiment, the multi-task feature extraction model employs a multi-task learning framework for feature extraction. Based on the backbone network sharing multi-scale features, the model performs subspace feature mapping and introduces a center loss variant to constrain feature mapping during training. This reduces interference and redundancy between tasks, improving the discriminative power and task specificity of feature representation.

[0112] See Figure 4 , Figure 4 This is a flowchart illustrating the overall training process for the multi-task feature extraction model and feature fusion model in the embodiments of this application. Figure 4 As shown, the overall process includes two main steps: training the multi-task feature extraction model and training the multimodal ReID model (i.e., feature fusion model).

[0113] First, the multi-task feature extraction model is trained.

[0114] like Figure 4As shown, in this embodiment, multi-task learning training adopts an end-to-end multi-task deep learning framework. First, the vehicle image is input into the backbone network. The backbone network performs multi-level feature extraction on the input vehicle image, generating a basic feature map with rich semantic information. Subsequently, the basic feature map is fed into the Feature Pyramid Network (FPN), which fuses multi-scale information through a top-down and lateral connection structure, generating a series of feature maps with decreasing resolution, thereby simultaneously capturing the image's detailed features and global contextual information. Feature mapping using a center loss variant maps the shared backbone's multi-scale features to the sub-feature spaces of different tasks. Features from different tasks are input into the multi-task learning module for joint training.

[0115] like Figure 4 As shown, compared with traditional multi-task learning, in this embodiment, the multi-task learning module uses the recognition results output by each task head for training. In addition to the task heads and loss functions of each task, a dynamic weight adjustment mechanism is introduced to adaptively adjust the weights w of different tasks in training according to the real-time loss or uncertainty of each task (such as gradient normalization or probability weighting). This effectively alleviates the conflict and imbalance between multiple tasks and improves the generalization performance and convergence stability of the overall model.

[0116] Then, after the multi-task feature extraction model is trained, the feature fusion model is trained.

[0117] In this embodiment, the feature fusion model can be a multimodal vehicle re-identification model, such as... Figure 4 The ReID model shown above. Its specific structure is described in the previous section. Figure 3 This will not be repeated here.

[0118] like Figure 4 As shown, the input to the feature fusion model is multimodal concatenated features, which consist of two parts: one part is various vehicle task features extracted by the multi-task feature extraction model, i.e., visual features, such as color features, license plate location features, etc., in addition to vehicle model features and brand features. The vehicle model can be further refined to specify which series and model. These features are set according to actual needs, and this embodiment does not impose any limitations. The other part is structured text information, such as license plate number and vehicle brand name. This text information is first converted into discrete symbol sequences (tokens) and mapped into dense vector representations through an embedding layer to capture its semantic information. Then, the text embedding vectors are concatenated with the multi-task visual features to obtain the multimodal concatenated features, which are then input into the ReID model for feature fusion.

[0119] As mentioned earlier, the feature fusion model can use Transformer as the core architecture of ReID, leveraging its powerful self-attention mechanism to model the interaction relationships between different features, achieving deep semantic alignment and complementarity between visual and textual information. The output features processed by multiple Transformer encoders serve as the final multimodal fusion representation, possessing strong identity discrimination capabilities.

[0120] When training the feature fusion model, it is necessary to extract vehicle multi-task features and vehicle text features from two sample vehicle images and their corresponding text information, respectively. After concatenation, the feature fusion model generates two multimodal fusion features. Based on the comparison results of whether the two multimodal fusion features are the same and the preset ground truth, the feature fusion model is trained.

[0121] In this embodiment, a multi-task deep learning framework is used to collaboratively optimize multiple related tasks and extract more discriminative vehicle subspace features. Then, multi-modal information such as vehicle multi-task features and vehicle text features are fused to construct a robust multi-modal vehicle re-identification model, which improves the accuracy of vehicle identity recognition and cross-modal retrieval performance. Finally, efficient and accurate vehicle identification and comparison in complex scenarios are achieved.

[0122] The training methods for the multi-task feature extraction model and the feature fusion model will be explained in detail below.

[0123] See Figure 5 , Figure 5 This is a flowchart illustrating the training method of the multi-task feature extraction model provided in an embodiment of this application. Figure 5 As shown, the training process includes:

[0124] Step S500: Obtain the original model of the multi-task feature extraction model.

[0125] See Figure 6 , Figure 6 This is a schematic diagram of the original model of the multi-task feature extraction model in this application embodiment. The original model includes: a multi-scale feature extraction sub-network, multiple task feature mapping sub-networks, and multiple task heads corresponding one-to-one with the multiple task feature mapping sub-networks. Each task feature mapping sub-network corresponds to one task. For example... Figure 6 As shown, in this embodiment, the multi-scale feature extraction sub-network includes CNN and FPN as the backbone network, and the multiple task feature mapping sub-networks are multiple fully connected layers.

[0126] like Figure 6As shown, in order to complete the training in the original model of this embodiment, in addition to the multi-scale feature extraction subnetwork and multiple task feature mapping subnetworks, multiple task heads are added. The model is trained based on the task recognition results output by each task head and the preset ground truth.

[0127] Step S510: Obtain multiple sample vehicle images and the ground truth value of each task feature for each sample vehicle image.

[0128] Among them, the various vehicle task characteristics include at least: vehicle model task characteristics and brand task characteristics.

[0129] Step S520: Input the vehicle image into the original model for processing to obtain recognition results for various tasks, including:

[0130] Step S521: Input the sample vehicle images into the multi-scale feature extraction subnetwork;

[0131] Step S522: Multi-scale feature extraction sub-network extracts features from sample vehicle images and outputs sample multi-scale features to each task feature mapping sub-network.

[0132] Step S523: Each of the task feature mapping sub-networks maps the multi-scale features to its corresponding task features and outputs them to the corresponding task header.

[0133] Step S524: For each task head, output the task identification result based on the task features;

[0134] Step S530, calculate the total model loss, including: calculating the task loss of each task based on the recognition results of each task and the ground truth of each task; calculating the inter-class separation loss of each task center; calculating the center loss variant based on the inter-class separation loss; and calculating the total model loss based on the center loss variant and the loss of each task.

[0135] Step S540: Based on the total loss of the model, adjust the network parameters of each task feature mapping sub-network and perform joint training on each task feature mapping sub-network.

[0136] Step S550: If the total loss of the model satisfies the convergence condition, delete each task head in the original model to obtain the trained multi-task feature extraction model.

[0137] The training method for the multi-task feature extraction model provided in this application introduces inter-class separation loss to constrain feature mapping when calculating the loss, thereby reducing interference and redundancy between tasks and improving the discriminative ability and task specificity of feature representation.

[0138] In some embodiments, a dynamic weight adjustment mechanism can be used to adaptively adjust the weights of different tasks during training based on the real-time loss or uncertainty of each task (such as gradient normalization or probability weighting).

[0139] In this embodiment, the center loss variant can be calculated based on the following preset feature mapping center variant loss formula. :

[0140]

[0141]

[0142]

[0143] in, It is a multi-scale feature mapping to the task Feature vectors in the feature space It is a task The corresponding task center, It's the number of tasks. It is a central loss. It is the inter-class separation loss; These are preset hyperparameters. Among them, The feature mapping matrix derived from multi-scale feature mapping.

[0144] Among them, the central loss This is used to constrain task features to be close to the class center (i.e., compact within the class), and the inter-class separation loss. The λ hyperparameter is used to constrain the feature class centers of different tasks to be as far apart as possible (i.e., inter-class divergence), and it is used to adjust the balance between intra-class compactness and inter-class divergence during training. Therefore, by introducing a regularization constraint of a "center loss variant" during the mapping process, interference and redundancy between tasks are reduced, and the discriminative power and task specificity of feature representation are improved.

[0145] In this embodiment, the total model loss can be calculated based on the following preset formula for total model loss. :

[0146] in, The task loss for the i-th task; This is the task loss weight for the k-th task, which can be calculated based on the following dynamic weight adjustment formula:

[0147]

[0148] in, This represents the loss of the k-th task; The rate of change of the task loss; Represents multi-scale feature vectors; The norm for multi-scale features; This is a temperature hyperparameter used to control the smoothness of the weight distribution; Let L2 norm represent the feature after the k-th branch mapping; Let L2 norm represent the feature after the j-th branch is mapped; As a reference norm, the mean of the norms of all branches is taken to avoid numerical instability; and These are preset hyperparameters used to control the contribution ratios of the gradient term and the feature norm term.

[0149] In this embodiment, based on the dynamic weight adjustment formula, the weights of different tasks in training can be adaptively adjusted according to the real-time loss or uncertainty of each task (such as gradient normalization or probability weighting), thereby effectively alleviating the conflict and imbalance between multiple tasks and improving the generalization performance and convergence stability of the overall model.

[0150] In this embodiment, This represents the task loss for the i-th task. In practical applications, the aforementioned tasks, besides categorized tasks such as vehicle model tasks and brand tasks, can also include detection tasks such as detecting license plate locations.

[0151] For classification tasks, the cross-entropy loss formula can be used to calculate the loss. :

[0152]

[0153] Where N is the total number of categories for the classification task. One-hot encoding of the real label. Predict the probability that the sample belongs to the i-th category for the model.

[0154] For detection tasks, the loss can be calculated using the mean squared error loss formula. :

[0155]

[0156] Where M is the number of values ​​that need to be regressed. The actual value of the i-th value. Let be the i-th value predicted by the model.

[0157] See Figure 7 , Figure 7 A flowchart illustrating the training method for the feature fusion model provided in this application embodiment. The process includes:

[0158] Step S700: Obtain the original feature fusion model;

[0159] Step S710: Obtain multiple sets of training samples. Each set of training samples includes: two sample vehicle images and corresponding two sample vehicle text information; the sample vehicle text information includes at least: license plate number.

[0160] In multiple training samples, some training samples have two vehicle images that are not the same vehicle; in other training samples, two vehicle images are the same vehicle or a vehicle with a fake license plate, and the license plate numbers in the text information of the two vehicle images are the same. The result of whether the two vehicle images are the same vehicle is taken as the ground truth of that training sample.

[0161] Step S720 involves processing each group of samples separately, including:

[0162] Step S721: Input the sample vehicle image into the trained multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features;

[0163] Step S722: Convert the vehicle text information to obtain vehicle text features;

[0164] Step S723: After concatenating multiple vehicle task features with vehicle text features, input them into the original feature fusion model for fusion to generate multimodal fusion features;

[0165] Step S730: Compare the multimodal fusion features corresponding to the two sample vehicle images in the group of samples to obtain the sample comparison result of whether the vehicles in the two sample vehicle images are the same vehicle.

[0166] Step S740: Based on the comparison results of the sample group, compare them with the ground truth. If the comparison results are the same, the feature fusion model training is complete. If the comparison results are different, adjust the network parameters of the feature fusion model and continue training.

[0167] In this embodiment, the feature fusion model is trained using the two different training samples mentioned above. This not only trains the model's ability to distinguish and identify the same vehicle, but also trains the model's ability to distinguish different vehicles.

[0168] In this embodiment, the feature fusion model can be a ReID model with Transformer as its core architecture. In this embodiment, a trained multi-task feature extraction model is used to train the feature fusion model based on the process of identifying cloned vehicles. This enables the feature fusion model to effectively fuse vehicle task features and vehicle text features, resulting in a multimodal fusion feature with strong discriminative power, thereby improving the accuracy of cloned vehicle identification.

[0169] This application also provides a device for identifying vehicles using counterfeit license plates, see [link to relevant documentation]. Figure 8 The device includes:

[0170] The acquisition module 801 is used to acquire two vehicle images to be identified and two corresponding vehicle text information; each vehicle text information includes at least: a license plate number; and the license plate numbers in the two vehicle text information are the same;

[0171] The multimodal feature generation module 802 is used to process two vehicle images and their corresponding two vehicle text information separately: inputting the vehicle images into a multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; converting the vehicle text information corresponding to the vehicle images to obtain vehicle text features; concatenating the multiple vehicle task features and vehicle text features, and then inputting them into a feature fusion model for fusion to generate multimodal fusion features;

[0172] The feature comparison module 803 is used to compare the multimodal fusion features corresponding to the two vehicle images to be identified. If the multimodal fusion features are different, it is determined that the vehicles in the two vehicle images are not the same vehicle and there is a license plate clone relationship between the two vehicles.

[0173] By applying the embodiments of this application, in the process of identifying cloned vehicles, not only are various vehicle task features extracted from vehicle images utilized, but vehicle text features are also introduced, which makes up for the shortcomings of pure visual models in semantic understanding and improves the accuracy of recognition.

[0174] This application also provides a training device for a multi-task feature extraction model, see [link to relevant documentation]. Figure 9 The device includes:

[0175] The original model acquisition module 901 is used to obtain the original model of the multi-task feature extraction model. The original model includes: a multi-scale feature extraction sub-network, multiple task feature mapping sub-networks, and multiple task heads that correspond one-to-one with the multiple task feature mapping sub-networks, wherein each task feature mapping sub-network corresponds to a task.

[0176] The sample image acquisition module 902 is used to acquire multiple sample vehicle images and the ground truth value of each task feature of each sample vehicle image; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features;

[0177] The original model processing module 903 is used to input vehicle images into the original model to obtain recognition results for various tasks, including: inputting sample vehicle images into a multi-scale feature extraction sub-network; the multi-scale feature extraction sub-network extracts features from the sample vehicle images and outputs sample multi-scale features to various task feature mapping sub-networks; each task feature mapping sub-network maps the multi-scale features to their corresponding task features and outputs them to the corresponding task head; each task head outputs the recognition result for this task based on the task features.

[0178] The model total loss calculation module 904 is used to calculate the model total loss, including: calculating the task loss of each task based on the recognition results of each task and the ground truth of each task; calculating the inter-class separation loss of each task center; calculating the center loss variant based on the inter-class separation loss; and calculating the model total loss based on the center loss variant and the loss of each task.

[0179] The parameter adjustment module 905 is used to adjust the network parameters of each task feature mapping sub-network based on the total loss of the model, and to jointly train each task feature mapping sub-network.

[0180] The task head deletion module 906 is used to delete each task head in the original model when the total loss of the model satisfies the convergence condition, so as to obtain the trained multi-task feature extraction model.

[0181] By applying the embodiments of this application, when calculating the central loss variant, inter-class separation loss is introduced to constrain the feature mapping, which reduces interference and redundancy between tasks and improves the discriminative ability and task specificity of feature representation.

[0182] This application also provides a training device for a feature fusion model, see [link to relevant documentation]. Figure 10 The device includes:

[0183] Original feature fusion model acquisition module 1001 is used to obtain the original feature fusion model;

[0184] The training sample acquisition module 1002 is used to acquire multiple sets of training samples. Each set of training samples includes: two sample vehicle images and corresponding text information for two sample vehicles; the text information for the sample vehicles includes at least: license plate number; in the multiple sets of training samples, the two sample vehicle images in some training samples are not of the same vehicle; the two sample vehicle images in other training samples are of the same vehicle or a vehicle with a fake license plate, and the license plate number in the text information of the two sample vehicles is the same; the result of whether the vehicles in the two sample vehicle images are the same vehicle is taken as the ground truth of the set of training samples;

[0185] The sample processing module 1003 is used to perform the following processing for each group of samples: inputting the sample vehicle image into a trained multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; converting the vehicle text information to obtain vehicle text features; concatenating the multiple vehicle task features with the vehicle text features and inputting them into the original feature fusion model for fusion to generate multimodal fusion features;

[0186] The sample feature comparison module 1004 compares the multimodal fusion features corresponding to the two sample vehicle images in the group of samples to obtain the sample comparison result of whether the vehicles in the two sample vehicle images are the same vehicle.

[0187] The parameter adjustment module 1005 is used to compare the sample comparison results and the true values ​​based on the sample group. If the comparison results are the same, the feature fusion model training is completed. If the comparison results are different, the network parameters of the feature fusion model are adjusted and training continues.

[0188] In this embodiment, a trained multi-task feature extraction model is used to train a feature fusion model based on the process of identifying cloned vehicles. This enables the feature fusion model to effectively fuse vehicle task features and vehicle text features. The resulting multimodal fusion features have strong discriminative power, thereby improving the accuracy of cloned vehicle identification.

[0189] This application also provides an electronic device, such as... Figure 11 As shown, it includes:

[0190] Memory 1101 is used to store computer programs;

[0191] When the processor 1102 executes the program stored in the memory 1101, it implements the following steps of the method for identifying cloned vehicles: obtaining two vehicle images to be identified and two corresponding vehicle text information; each vehicle text information includes at least: a license plate number; and the license plate numbers in the two vehicle text information are the same;

[0192] The two vehicle images and their corresponding two vehicle text information are processed separately, including: inputting the vehicle images into a multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; converting the vehicle text information corresponding to the vehicle images to obtain vehicle text features; concatenating the multiple vehicle task features and vehicle text features, and then inputting them into a feature fusion model for fusion to generate multimodal fusion features;

[0193] The multimodal fusion features corresponding to the two vehicle images to be identified are compared. If the multimodal fusion features are different, it is determined that the vehicles in the two vehicle images are not the same vehicle, and there is a license plate clone relationship between the two vehicles; or,

[0194] The steps to implement the training method for the following multi-task feature extraction model are as follows:

[0195] Obtain the original model of the multi-task feature extraction model, the original model including: a multi-scale feature extraction sub-network, multiple task feature mapping sub-networks and multiple task heads corresponding one-to-one with the multiple task feature mapping sub-networks, wherein each task feature mapping sub-network corresponds to a task.

[0196] Obtain multiple sample vehicle images and the ground truth value of each task feature for each sample vehicle image; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features;

[0197] Vehicle images are input into the original model for processing to obtain recognition results for various tasks, including: inputting sample vehicle images into a multi-scale feature extraction sub-network; the multi-scale feature extraction sub-network extracts features from the sample vehicle images and outputs sample multi-scale features to various task feature mapping sub-networks; each task feature mapping sub-network maps the multi-scale features to their corresponding task features and outputs them to the corresponding task head; each task head outputs the recognition result for its task based on the task features.

[0198] The calculation of the total model loss includes: calculating the task loss of each task based on the recognition results of each task and the ground truth of each task; calculating the inter-class separation loss of each task center; calculating the center loss variant based on the inter-class separation loss; and calculating the total model loss based on the center loss variant and the loss of each task.

[0199] Based on the total loss of the model, the network parameters of each task feature mapping sub-network are adjusted, and the task feature mapping sub-networks are jointly trained.

[0200] If the total loss of the model satisfies the convergence condition, each task head in the original model is deleted to obtain the trained multi-task feature extraction model; or,

[0201] The steps to implement the training method for the following feature fusion model are as follows:

[0202] Obtain the original feature fusion model;

[0203] Multiple sets of training samples are obtained. Each set of training samples includes two sample vehicle images and corresponding text information for the two sample vehicles. The text information for the sample vehicles includes at least one license plate number. In some sets of training samples, the two sample vehicle images are not of the same vehicle. In other sets of training samples, the two sample vehicle images are of the same vehicle or a vehicle with a fake license plate, and the license plate numbers in the text information of the two sample vehicles are the same. The result of whether the two sample vehicle images are of the same vehicle is taken as the ground truth of that set of training samples.

[0204] Each set of samples is processed separately, including: inputting the sample vehicle images into a trained multi-task feature extraction model to obtain multiple vehicle task features; wherein, the multiple vehicle task features include at least: vehicle model task features and brand task features; converting the vehicle text information to obtain vehicle text features; concatenating the multiple vehicle task features with the vehicle text features and inputting them into the original feature fusion model for fusion to generate multimodal fusion features;

[0205] By comparing the multimodal fusion features corresponding to the two sample vehicle images in this set of samples, the sample comparison results of whether the vehicles in the two sample vehicle images are the same vehicle are obtained.

[0206] The sample comparison results of this set of samples are compared with the ground truth. If the comparison results are the same, the feature fusion model training is completed. If the comparison results are different, the network parameters of the feature fusion model are adjusted and training continues.

[0207] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1102, the communication interface, and the memory 1101 communicating with each other via the communication bus.

[0208] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0209] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0210] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0211] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0212] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described methods for identifying counterfeit license plates; or, the steps of a training method for a multi-task feature extraction model; or, the steps of a training method for a feature fusion model.

[0213] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the license plate recognition methods described above; or, a training method for a multi-task feature extraction model; or, a training method for a feature fusion model.

[0214] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0215] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0216] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0217] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for identifying a suit car, characterized by, The method comprises the following steps: obtaining two vehicle images to be identified and two corresponding vehicle text information respectively; each of the vehicle text information at least includes a license plate number; and the license plate numbers in the two vehicle text information are the same; processing the two vehicle images and the corresponding two vehicle text information respectively, comprising: inputting the vehicle image into a multi-task feature extraction model to obtain multiple vehicle task features; wherein the multiple vehicle task features at least include vehicle model task features and brand task features; converting the vehicle text information corresponding to the vehicle image to obtain vehicle text features; and inputting the multiple vehicle task features and the vehicle text features after splicing into a feature fusion model for fusion to generate multi-modal fusion features; comparing the multi-modal fusion features corresponding to the two vehicle images to be identified, and determining that the vehicles in the two vehicle images are not the same vehicle and that there is a plate covering relationship between the two vehicles in the case of different multi-modal fusion features.

2. The method of claim 1, wherein the multi-task feature extraction model comprises a multi-scale feature extraction subnetwork and a plurality of task feature mapping subnetworks, wherein each task feature mapping subnetwork corresponds to a task. The method comprises the following steps: inputting the vehicle image into the multi-scale feature extraction subnetwork; the multi-scale feature extraction subnetwork extracts features from the vehicle image and outputs multi-scale features to each task feature mapping subnetwork; each task feature mapping subnetwork maps the multi-scale features to the task features corresponding thereto.

3. The method of claim 1, wherein the feature fusion model is obtained by further training after the multi-task feature extraction model is trained.

4. The method of claim 1, wherein the multi-task feature extraction model is realized by adding a plurality of fully connected layers to a CNN+FPN visual model; each fully connected layer is used to map a task feature. The feature fusion model is a ReID model with a Transformer as the core architecture. The method comprises the following steps: obtaining an original model of a multi-task feature extraction model, wherein the original model comprises a multi-scale feature extraction subnetwork, a plurality of task feature mapping subnetworks, and a plurality of task heads corresponding to the plurality of task feature mapping subnetworks, wherein each task feature mapping subnetwork corresponds to a task; obtaining a plurality of sample vehicle images and the true value of each task feature of each sample vehicle image; wherein the multiple vehicle task features at least include vehicle model task features and brand task features; 5. A method for training a multitask feature extraction model, the method comprising: ​ ​ ​ The vehicle image is input into the original model for processing to obtain a plurality of task recognition results, including: inputting a sample vehicle image into a multi-scale feature extraction subnetwork; the multi-scale feature extraction subnetwork extracts features from the sample vehicle image and outputs sample multi-scale features to each task feature mapping subnetwork; each task feature mapping subnetwork maps the multi-scale features to corresponding task features and outputs the task features to a corresponding task head; and each task head outputs a recognition result of the task based on the task features. The total loss of the model is calculated, including: calculating the task loss of each task based on the task recognition result and the true value of each task; calculating the inter-class separation loss of each task center; calculating the center loss variant based on the inter-class separation loss; and calculating the total loss of the model based on the center loss variant and the task loss of each task. Based on the total loss of the model, the network parameters of each task feature mapping subnetwork are adjusted, and each task feature mapping subnetwork is jointly trained. In the case where the total loss of the model meets the convergence condition, each task head in the original model is deleted to obtain a trained multi-task feature extraction model.

6. The training method of the multi-task feature extraction model according to claim 5, wherein The center loss variant based on the inter-class separation loss calculation comprises: calculating the center loss variant based on a preset feature mapping center loss variant formula : ; ; ; wherein, is a multi-scale feature mapping to a task feature space, is a task corresponding task center, is a number of tasks, is a center loss, is an inter-class separation loss; is a preset hyper-parameter.

7. The training method of the multi-task feature extraction model according to claim 6, wherein The center loss-based variant and the respective task loss are used to calculate a total loss of the model, including: calculating the total loss of the model based on a preset total loss formula of the model : ; wherein, is the task loss for the i-th task; is the task loss weight for the k-th task, calculated based on the following equation: ; wherein, represents the loss of the kth task; is the change rate of the task loss; represents the multi-scale feature vector; is the norm of the multi-scale feature; is the temperature hyper-parameter; represents the L2 norm of the mapped feature of the kth branch; represents the L2 norm of the mapped feature of the jth branch; is the reference norm, which takes the mean of all branch feature norms; and is a preset hyper-parameter.

8. The training method of the multi-task feature extraction model according to claim 5, wherein The multi-task feature extraction model is realized by adding a plurality of fully connected layers on the basis of a CNN+FPN visual model; and each fully connected layer is used for mapping a task feature. 9.A method for training a feature fusion model, the method comprising: including: obtaining an original feature fusion model; obtaining a plurality of groups of training samples, each group of training samples including: two sample vehicle images and corresponding two sample vehicle text information; the sample vehicle text information at least includes: a license plate number; in the plurality of groups of training samples, a part of the training samples are two sample vehicle images of different vehicles; another part of the training samples are two sample vehicle images of the same vehicle or a set of license plates, and the license plate numbers in the two sample vehicle text information are the same; and the result of whether the vehicles in the two sample vehicle images are the same vehicle is taken as the true value of the group of training samples; processing each group of samples respectively, including: inputting the sample vehicle image into the trained multi-task feature extraction model to obtain a plurality of vehicle task features; wherein the plurality of vehicle task features at least include: a vehicle model task feature and a vehicle brand task feature; converting the vehicle text information to obtain a vehicle text feature; and inputting the plurality of vehicle task features and the vehicle text feature after splicing into the original feature fusion model for fusion to generate a multi-modal fusion feature; comparing the multi-modal fusion features corresponding to the two sample vehicle images of the group of samples to obtain a sample comparison result of whether the vehicles in the two sample vehicle images are the same vehicle; comparing the sample comparison result and the true value of the group of samples, in the case where the comparison result is the same, the feature fusion model is trained, and in the case where the comparison result is not the same, the network parameters of the feature fusion model are adjusted for further training.

10. The feature fusion model training method according to claim 9, wherein the feature fusion model is a ReID model with a Transformer as a core architecture. The feature fusion model comprises:

11. A device for identifying a suit car, characterized by, an obtaining module configured to obtain two vehicle images to be identified and two vehicle text information corresponding to the two vehicle images respectively; each of the vehicle text information comprises at least a license plate number; and the license plate numbers in the two vehicle text information are the same; a multi-modal feature generation module configured to process the two vehicle images and the two vehicle text information respectively: inputting the vehicle images into a multi-task feature extraction model to obtain a plurality of vehicle task features, wherein the plurality of vehicle task features comprise at least a vehicle model task feature and a vehicle brand task feature; converting the vehicle text information corresponding to the vehicle images to obtain a vehicle text feature; and inputting the plurality of vehicle task features and the vehicle text feature into a feature fusion model after splicing to generate multi-modal fusion features; a feature comparison module configured to compare the multi-modal fusion features corresponding to the two vehicle images to be identified, and determine that the vehicles in the two vehicle images are not the same vehicle and that there is a plate covering relationship between the two vehicles when the multi-modal fusion features are different. The feature fusion model comprises: 12.A device for training a multi-task feature extraction model, characterized in that, an original model obtaining module configured to obtain an original model of the multi-task feature extraction model, wherein the original model comprises a multi-scale feature extraction subnetwork, a plurality of task feature mapping subnetworks, and a plurality of task heads corresponding to the plurality of task feature mapping subnetworks, and each task feature mapping subnetwork corresponds to one task; a sample image obtaining module configured to obtain a plurality of sample vehicle images and true values of each task feature of each sample vehicle image, wherein the plurality of vehicle task features comprise at least a vehicle model task feature and a vehicle brand task feature; an original model processing module configured to input the vehicle images into the original model to obtain a plurality of task recognition results, comprising: inputting the sample vehicle images into the multi-scale feature extraction subnetwork; the multi-scale feature extraction subnetwork extracts features from the sample vehicle images and outputs sample multi-scale features to each task feature mapping subnetwork; each task feature mapping subnetwork maps the multi-scale features to the corresponding task feature and outputs the task feature to the corresponding task head; and each task head outputs the recognition result of the task based on the task feature; a model total loss calculation module configured to calculate a model total loss, comprising: calculating a task loss of each task based on the true value of each task and the task recognition result; calculating an inter-class separation loss of each task center; calculating a center loss variant based on the inter-class separation loss; and calculating the model total loss based on the center loss variant and the task loss; a parameter adjustment module configured to adjust the network parameters of each task feature mapping subnetwork based on the model total loss and jointly train each task feature mapping subnetwork; a task head deletion module configured to delete each task head in the original model to obtain a trained multi-task feature extraction model when the model total loss meets a convergence condition. The feature fusion model comprises: 13.A device for training a feature fusion model, characterized in that, ​ The original feature fusion model obtaining module is configured to obtain an original feature fusion model. The training sample obtaining module is configured to obtain a plurality of groups of training samples, each group of training samples including two sample vehicle images and corresponding two sample vehicle text information; the sample vehicle text information at least includes a license plate number; in the plurality of groups of training samples, a part of the training samples have two sample vehicle images of different vehicles; another part of the training samples have two sample vehicle images of the same vehicle or a fake license plate, and the license plate numbers in the two sample vehicle text information are the same; and a result of whether the vehicles in the two sample vehicle images are the same vehicle is taken as a true value of the group of training samples. The sample processing module is configured to perform the following processing on each group of samples respectively: inputting the sample vehicle images into the trained multi-task feature extraction model to obtain a plurality of vehicle task features; the plurality of vehicle task features at least include a vehicle model task feature and a vehicle brand task feature; converting the vehicle text information to obtain a vehicle text feature; and inputting the plurality of vehicle task features and the vehicle text feature after splicing into the original feature fusion model for fusion to generate a multi-modal fusion feature. The sample feature comparison module compares the multi-modal fusion features corresponding to the two sample vehicle images of the group of samples to obtain a sample comparison result of whether the vehicles in the two sample vehicle images are the same vehicle. The parameter adjustment module is configured to compare the sample comparison result and the true value of the group of samples, and in a case where the comparison result is the same, the feature fusion model training is completed, and in a case where the comparison result is not the same, the network parameters of the feature fusion model are adjusted for further training.

14. An electronic device, comprising: The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the fake license plate vehicle identification method of any one of claims 1-4; or implement the multi-task feature extraction model training method of any one of claims 5-8; or implement the feature fusion model training method of any one of claims 9-10. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the fake license plate vehicle identification method of any one of claims 1-4; or implement the multi-task feature extraction model training method of any one of claims 5-8; or implement the feature fusion model training method of any one of claims 9-10.

15. A computer-readable storage medium, characterized in that, ​

Images