Vehicle identification code photo comparison method based on character identification network

Abstract

According to the character recognition network-based vehicle identification code photo comparison method provided by the invention, the noise prediction network is constructed, the image features of all existing VIN codes are learned through the noise prediction network, and the trained noise prediction network is set in the diffusion model; in the diffusion model, inputting a preset standard Gaussian noise picture into the trained noise prediction network, predicting noise distribution through the noise prediction network by taking the vehicle fusion information code Vout as guide information, then denoising the preset standard Gaussian random noise picture, and finally generating a pseudo reference picture for the VIN code lacking the reference picture; and comparing the VIN code picture to be compared with the reference picture for comparison, and positioning a forgery suspicion VIN code.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically to a method for comparing vehicle identification number photos based on a character recognition network. Background Technology

[0002] The Vehicle Identification Number (VIN) is a unique identifier for a vehicle. It records not only the vehicle's production information and configuration details but also serves as a crucial basis for various stages of vehicle management, maintenance, and insurance claims. Because the graphic characteristics of VINs differ between manufacturers, batches, and models, VIN verification is necessary in many scenarios, such as used car transactions, to prevent issues like counterfeiting. The typical VIN verification process involves: staff collecting an image of the vehicle's VIN as the image to be verified; and then comparing it to the VIN image stored in the database when the vehicle was first registered. The comparison focuses on verifying the consistency of various visual features displayed in the VIN images to ensure that any forged VINs can be detected. If they match, the VIN verification is successful, and further verification can proceed.

[0003] Traditional methods mostly rely on manual visual comparison of VIN codes for consistency, which is not only time-consuming but also prone to errors, failing to meet the growing demand for accurate identification and digitization of vehicle information such as VIN codes. With technological advancements, automatic identification technologies based on neural network models and other methods have been introduced into the field of VIN code recognition.

[0004] However, in practical applications, VIN code feature comparison has revealed significant challenges, including missing historical data and limited verification methods. In scenarios such as used car transactions and cross-regional vehicle inspections, inspectors cannot obtain original factory images of the vehicle's VIN for comparison. If forged information or re-engraved VINs are found, they are difficult to detect. Furthermore, VIN comparison involves character-level micro-feature comparison, requiring high similarity in font and character shape. Current text analysis and recognition technologies are largely limited to natural scenes, offering little focus on VIN photo recognition and lacking end-to-end solutions for precise character-level localization and recognition. Moreover, due to the difficulty in collecting complete VIN photo data samples, it is challenging to comprehensively cover all character shape subcategories and maintain sample balance. Therefore, data-driven deep learning algorithms cannot guarantee high accuracy in highly similar forged cases and struggle to handle rare samples. If deep learning methods are used for VIN code recognition, continuous updates are needed to maintain the algorithm's effectiveness and robustness, placing extremely high demands on human and material resources. Summary of the Invention

[0005] To address the issue of existing automatic VIN identification methods failing to guarantee accuracy in identifying counterfeit VINs, this invention provides a vehicle identification number photo comparison method based on a character recognition network. Even if a reference photo for the VIN to be identified is missing, a reference photo can be generated for comparison. After comparison, suspicious characters present in the original vehicle identification number photo are found, and the suspected counterfeit VIN is located, providing a reference for inspectors.

[0006] The technical solution of this invention is as follows: a method for comparing vehicle identification number photos based on a character recognition network, characterized in that it includes the following steps: S1: Construct a feature extraction model; the feature extraction model includes: a VIN semantic information encoder, a vehicle basic information encoder, and a fusion coding layer; The input to the VIN semantic information encoder is VIN code text data, and the output is the global VIN semantic information encoding V. global ; The input to the vehicle basic information encoder is text data of vehicle basic information, and the output is a feature vector corresponding to each vehicle basic information. The basic vehicle information includes: vehicle manufacturer, vehicle type, brand and model, and year of manufacture; Encode the global VIN semantic information V globalAll vehicle basic information feature vectors are input into the fusion coding layer. The fusion coding layer concatenates all vehicle basic information feature vectors to obtain concatenated vehicle information features. Then, based on cross-attention, the VIN semantic information features and the concatenated vehicle information features are fused to output the vehicle fusion information code V. out ; S2: Construct a diffusion model; The input to the diffusion model is the vehicle fusion information code V. out and initialize standard Gaussian random noise images; The diffusion model uses vehicle fusion information encoding V out To guide the information, a trained noise prediction network is used to denoise the initial standard Gaussian random noise image, and finally a pseudo benchmark image is generated and output. S3: Construct a character recognition network; The character recognition network outputs the type and coordinates of each character in the input image. S4: Obtain the image of the VIN code to be compared, and identify the vehicle identification number in the image, which is recorded as: VIN code to be confirmed; S5: Retrieve the corresponding baseline photo from the historical database based on the VIN code to be confirmed; If a reference image is found in the database, and the reference image is used as a reference image for comparison, then proceed to step S7; If the reference image cannot be retrieved, proceed to step S6; S6: Using the pre-built diffusion model, generate a pseudo-reference photo for the VIN code image to be compared; Use the pseudo-reference photograph as a reference image for comparison, and proceed to step S7; S7: By comparing the VIN code image to be compared with the reference image for comparison, the VIN code suspected of being counterfeited is located.

[0007] Its further features are: The VIN semantic information encoder includes: a character encoding layer, an embedding layer, a Transformer encoding layer, and a global pooling layer; The character encoding layer performs one-hot encoding on the VIN information to obtain the VIN character encoding set V, with a tensor format of B×L×N; Where B is the batch size, L is the length of the vehicle identification number, and N is the total number of digits and letters in the mapped vehicle identification number; The embedding layer comprises two independent fully connected MLP layers: a character embedding encoding layer and a positional embedding encoding layer, which perform character and positional embedding encoding respectively. For each input VIN text information, after processing by the character embedding encoding layer and the positional embedding encoding layer, a character embedding tensor V is obtained. char and position embedding tensor V pos ; The TransFormer coding layer contains n multi-head attention heads of dimension dim, a feedforward neural network consisting of two fully connected layers, and an LN normalization layer. The character embedding tensor V char and the position embedding tensor V pos After concat processing, it is used as the input V of the TransFormer encoding layer. con We obtain the TransFormer tensor V. trans The calculation formula is as follows: ; In the formula, W Q T W K T and W V T These represent the linear transformation parameters of the query, key, and value, respectively; D is the input V. con The number of channels; FFN represents a feedforward neural network; The global pooling layer performs average pooling on all character information and then obtains the final global VIN semantic information encoding V through linear mapping. global ; The vehicle basic information encoder includes: a vehicle manufacturer encoding layer, a vehicle type encoding layer, a brand and model encoding layer, and a manufacturing year encoding layer; the vehicle manufacturer encoding layer assigns an index value to each manufacturer, and uses an embedding layer to map the index value to a vehicle manufacturer feature vector Veh1; the vehicle type encoding layer assigns an index value to each vehicle type, and uses an embedding layer to map the index value to a vehicle type feature vector Veh2. The brand model encoding layer uses the Word2Vec model to calculate the brand model vector, and then adjusts the dimensions through an MLP layer to obtain the brand model feature vector Veh3. The manufacturing year encoding layer uses the year as a tensor value for normalization, and finally obtains the vehicle year feature vector Veh4 through an MLP layer. The noise prediction network is constructed based on the U-Net network model; The noise prediction network includes a feature extraction module, an image super-resolution module, and an attention alignment layer; The feature extraction module includes a basic convolutional module 1, a residual fusion downsampling layer, and a depth downsampling unit connected in sequence. The image super-resolution module includes a depth upsampling unit, a residual fusion upsampling layer and a basic convolution module 2 connected in sequence. The feature extraction module and the image super-resolution module are arranged symmetrically on the left and right, and the number of residual fusion downsampling layers and residual fusion upsampling layers are equal and correspond one-to-one. The basic convolution module 1 is located at the top of the feature extraction module as an input module, and the basic convolution module 2 is located at the top of the image super-resolution module as an output module. The depth downsampling unit is located at the bottom of the feature extraction module, and the depth upsampling unit is located at the bottom of the image super-resolution module. The output of the depth downsampling unit is processed by the attention alignment layer and then sent to the depth upsampling unit. At the same time, the feature map from the downsampling process of the depth downsampling unit is fused during the upsampling process of the depth upsampling unit. The residual fusion downsampling layer includes: a VIN residual module and a convolutional downsampling layer; The residual fusion upsampling layer includes: a VIN residual module and a convolutional upsampling layer. The convolutional upsampling layer in the residual fusion upsampling layer fuses the feature maps from the same level of the convolutional downsampling layer during the upsampling process. The VIN residual module includes a time-step fusion module and a cross-attention calculation module; the time-step fusion module converts the time step t into conditional vector weights (scale) and biases (bias) on the input feature map z. t The time step and VIN semantic information are fused across all channels; the calculation process of the VIN residual module is as follows: x'=Conv(x)•(1+scale)+bias; x”=conv(x')+CrossAtt(x,V out ); x out = x”+x; In the formula, Conv represents the basic convolution module operation, CrossAtt represents the cross-attention computation unit, the weight scale and bias value are obtained by chunking after time step encoding, and x out This indicates the output of the VIN residual module; During the training process, the noise prediction network randomly samples the original vehicle identification code electronic data image based on the pre-constructed training dataset, then uniformly and randomly samples the time step t, and generates standard Gaussian noise. The diffusion model includes: a noise addition process and a noise reduction process; The noise addition process is as follows: ; In the formula, z t Represents the image with added noise at time t; α and β are hyperparameters; ε t The noise added at time t follows a Gaussian distribution; α t 2 +β t 2 =1; T is the preset maximum time step for the noise addition process; The noise reduction process is as follows: ; In the formula, σ t Let t represent the variance, and k represent the sampled values ​​from the same distribution; when t=1, k=0. ε θ (z t ,t) represents the noise distribution output by the noise prediction network at time step t, and θ is the parameter of the noise prediction network; ; In step S3, the character recognition network includes: a feature extraction network and a character segmentation network connected in sequence; The feature extraction network is built based on the HRNetV2 model. After extracting features from the input image, the feature extraction network sends the extracted features to the character segmentation network. The character segmentation network includes three parallel task branches: a discrimination task branch, a recognition task branch, and a detection task branch. The discrimination task branch includes: K k1×k1 convolutional layers and one k2×k2 convolutional layer; the discrimination task branch uses a binary mask image as supervision, outputs a two-channel text instance segmentation feature map, and predicts the probability of text presence or absence at each pixel position; The recognition task branch includes: K k1×k1 convolutional layers and one k2×k2 convolutional layer; the recognition task branch outputs a C-channel category prediction feature map, where C represents all possible categories of the vehicle identification code character, and predicts the probability that the character at each pixel is a character. The detection task branch includes: K+1 k1×k1 convolutional layers and one k2×k2 convolutional layer; the detection task branch outputs a coordinate position feature map with 4 channels to describe the distance from the pixel position to the four sides of the predicted character rectangle. Step S7 specifically includes the following steps: a1: Input the VIN code image to be compared into a pre-trained character recognition network to obtain the text data and coordinates of the character to be compared; a2: The reference image is input into a pre-trained character recognition network to obtain the text data of the reference character and the reference coordinates. a3: Confirm whether the character content included in the character to be compared and the comparison reference character is consistent; If they match, proceed to step a4; Otherwise, if the character content of the two is inconsistent, proceed to step a5; a4: Perform a multi-dimensional comparison between the VIN code image to be compared and the reference image used for comparison; The multi-dimensional features include: structural features, texture features, geometric features, and edge features; If the two are similar, it means that the VIN code image to be compared has passed this comparison operation; Otherwise, if the two are not similar, it indicates that the VIN code image to be compared is suspected of being forged, and then step a5 is executed; a5: Submit the VIN code images suspected of being forged for further manual processing; Step a4 specifically includes the following steps: b1: The VIN code image to be compared and the reference image for comparison are respectively sent as input images to the character recognition network; b2: For each input image, obtain the output of the discrimination task branch of the character recognition network, use the non-maximum suppression criterion to select the classification results of the default box, predict the character category corresponding to each character coordinate region, and obtain the character text data corresponding to the input image; b3: Extract the character image set {Z1, Z2, ..., Z...} from the VIN code image to be compared, by cropping all the characters included in the image. n}; On the reference image used for comparison, all characters included in it are cropped to obtain a character image set {W1, W2, ..., W}. n}; b4: Pair the two character image sets one by one in front-to-back order to obtain {Z k W k The paired character image group is binarized to obtain the binarized character image group {Z'}. k W' k}; b5: For each group of binary character images {Z' k W' kExtract multi-dimensional feature values ​​from the images in the table. b6: Based on each group of binary character images {Z' k W' k The similarity calculation of the corresponding multi-dimensional feature values ​​yields a comprehensive consistency score; b7: Constructing a partial anomaly detection model based on isolated forests; The input to the partial anomaly detection model is the group of all binary character images {Z'} corresponding to the VIN code image to be compared. k W' k The output of the combined score of all the multi-dimensional feature values ​​and consistency is the character image group that is different from other binarized character image groups based on similarity features. b8: Convert each group of binary character images {Z' k W' k The corresponding consistency score is compared with the preset similarity threshold. If the similarity threshold is less than 1, then this group of characters is identified as a suspected image group, and the VIN code image to be compared is taken as a suspected image, and step a5 is executed. Otherwise, if the overall consistency score is greater than or equal to the similarity threshold, proceed to step b9; b9: Confirm whether the overall consistency score corresponding to the VIN code images to be compared is greater than or equal to the similarity threshold; If so, proceed to step b10; If not, stop this operation; b10: Group all binarized character images {Z' corresponding to the VIN code image to be compared. k W' k All the multi-dimensional feature values ​​and consistency scores mentioned above are simultaneously fed into the trained partial anomaly detection model. If the output of the partial anomaly detection model matches the abnormal character image group, then the VIN code image to be compared is taken as the suspect image, and step a5 is executed.

[0008] This application provides a method for comparing vehicle identification numbers (VINs) photos based on a character recognition network. It constructs a noise prediction network, learns the image features of all existing VIN codes, and sets the trained noise prediction network in a diffusion model. In the diffusion model, a preset standard Gaussian noise image is input into the trained noise prediction network. Using the vehicle fusion information code Vout as guiding information, the noise prediction network predicts the noise distribution. Then, the preset standard Gaussian random noise image is denoised, ultimately generating a pseudo-reference image for VINs lacking a reference image. By comparing the VIN image to be compared with the reference image, suspected counterfeit VINs are located. Based on this method, even if a reference image is missing for the VIN to be identified, the pseudo-reference image can effectively locate suspected counterfeit VINs, providing a reference for inspectors. This method can solve the problems of incomplete coverage of character shapes and stamped photo samples for vehicle identification numbers in the traffic management industry, low efficiency of manual comparison, and susceptibility to errors. Attached Figure Description

[0009] Figure 1 This is a flowchart of the vehicle identification number photo comparison method based on diffusion model and character recognition network of this application; Figure 2 A schematic diagram illustrating the steps for encoding VIN semantic information and basic vehicle information; Figure 3 This is a schematic diagram of the diffusion model architecture generated for VIN photos; Figure 4 This is a schematic diagram of the U-Net architecture for noise prediction models; Figure 5 This is a schematic diagram of a character recognition network and a multi-dimensional character comparison process. Detailed Implementation

[0010] like Figure 1 As shown, the present invention includes a method for comparing vehicle identification number photos based on a character recognition network, which includes the following steps.

[0011] S1: Construct a feature extraction model; such as Figure 2 As shown, the feature extraction model includes: a VIN semantic information encoder, a vehicle basic information encoder, and a fusion coding layer; The VIN semantic information encoder takes VIN code text data as input and outputs the global VIN semantic information encoding V. global ; The input to the vehicle basic information encoder is text data of vehicle basic information, and the output is a feature vector corresponding to each vehicle basic information. The vehicle basic information includes: vehicle manufacturer, vehicle type, brand and model, and year of manufacture.

[0012] Encode the global VIN semantic information V global All vehicle basic information feature vectors are input into the fusion coding layer. The fusion coding layer concatenates all vehicle basic information feature vectors to obtain the concatenated vehicle information features. Then, based on cross-attention, the VIN semantic information features and the concatenated vehicle information features are fused to output the vehicle fusion information code V. out .

[0013] To construct the training dataset, high-quality vehicle identification number (VIN) photos were selected from historical inspection data archives, and the specific VINs were labeled to obtain sample data. Based on the sample dataset, the training dataset and the validation dataset were obtained.

[0014] The VIN semantic information encoder includes: a character encoding layer, an embedding layer, a Transformer encoding layer, and a global pooling layer.

[0015] The character encoding layer performs one-hot encoding on the VIN information to obtain the VIN character encoding set V, with a tensor format of B×L×N; where B is the batch size, L is the length of the vehicle identification number, and N is the total number of digits and letters mapped to the vehicle identification number. In this embodiment, B=1, L=17, and N=33.

[0016] The embedding layer comprises two independent fully connected MLP layers: a character embedding encoding layer and a positional embedding encoding layer. These layers perform character and positional embedding encoding, respectively, preserving the rich semantic information and positional information between characters of the vehicle identification number (VIN). For each input VIN text, the character embedding tensor V is obtained after processing by the character embedding encoding layer and the positional embedding encoding layer. char and position embedding tensor V pos ; The TransFormer encoding layer contains n multi-head attention heads of dimension dim, a feedforward neural network consisting of two fully connected layers, and an LN normalization layer; in this embodiment, n=8 and dim=32.

[0017] Character embedding tensor V char and position embedding tensor V pos After concat processing, it is used as the input V of the TransFormer encoding layer. con We obtain the TransFormer tensor V. trans The calculation formula is as follows: ; In the formula, W Q T W K T and W V TThese represent the linear transformation parameters of the query, key, and value, respectively; D is the input V. con The number of channels; FFN represents a feedforward neural network; The global pooling layer performs average pooling on all character information and then obtains the final global VIN semantic information encoding V through linear mapping. global .

[0018] The vehicle basic information encoder includes: a vehicle manufacturer encoding layer, a vehicle type encoding layer, a brand and model encoding layer, and a manufacturing year encoding layer; the vehicle manufacturer encoding layer assigns an index value to each manufacturer, and uses an embedding layer to map the index value to a vehicle manufacturer feature vector Veh1; the vehicle type encoding layer assigns an index value to each vehicle type, and uses an embedding layer to map the index value to a vehicle type feature vector Veh2; The brand and model encoding layer uses the Word2Vec model to calculate the brand and model vector, and then adjusts the dimensions through an MLP layer to obtain the brand and model feature vector Veh3; The manufacturing year encoding layer normalizes the year as a tensor value, and finally obtains the vehicle year feature vector Veh4 through an MLP layer.

[0019] The fusion coding layer concatenates {Veh1,Veh2, Veh3, Veh4} and then uses a cross-attention mechanism to combine it with V... global The vehicle information code is then fused with the basic vehicle information code to obtain the final vehicle fused information code V. out .

[0020] The vehicle basic information encoder constructed in this application encodes the vehicle from four levels: vehicle manufacturer, vehicle type, brand and model, and manufacturing year. Finally, the fusion encoding layer performs cross-attention calculation on the vehicle basic information and VIN semantic information from the four levels to complete the information interaction, which serves as the guiding condition for the diffusion model.

[0021] S2: Construct a diffusion model. For example... Figure 3 As shown, the input to the diffusion model is the vehicle fusion information encoding V. out Initialize standard Gaussian random noise images; use a diffusion model to encode V with vehicle fusion information. out To guide the information, a trained noise prediction network is used to denoise the initial standard Gaussian random noise image, and finally a pseudo benchmark image is generated and output.

[0022] The noise prediction network is built based on the U-Net network model; such as Figure 4 As shown, the U-Net network belongs to the encoder-decoder architecture, so the noise prediction network includes a feature extraction module, an image super-resolution module, and an attention alignment layer; The feature extraction module includes a basic convolutional module 1, a residual fusion downsampling layer, and a depth downsampling unit connected in sequence; The image super-resolution module includes a depth upsampling unit, a residual fusion upsampling layer, and a basic convolutional module 2 connected in sequence; The feature extraction module and the image super-resolution module are set symmetrically on the left and right, and the number of residual fusion downsampling layers and residual fusion upsampling layers are equal and correspond one-to-one; Basic convolution module 1 is located at the top of the feature extraction module as the input module, and basic convolution module 2 is located at the top of the image super-resolution module as the output module; The depth downsampling unit is located at the bottom of the feature extraction module, and the depth upsampling unit is located at the bottom of the image super-resolution module. The output of the depth downsampling unit is processed by the attention alignment layer and then fed into the depth upsampling unit. At the same time, the feature map from the downsampling process of the depth downsampling unit is fused into the upsampling process of the depth upsampling unit. The residual fusion downsampling layer includes: a VIN residual module and a convolutional downsampling layer; The residual fusion upsampling layer includes: VIN residual module and convolutional upsampling layer. The convolutional upsampling layer in the residual fusion upsampling layer fuses the feature map from the same level of convolutional downsampling layer during the upsampling process. The VIN residual module includes a time-step fusion module and a cross-attention calculation module. The time-step fusion module converts the time step t into conditional vector weights (scale) and biases (bias) on the input feature map z. t The time step and VIN semantic information are fused across all channels; the calculation process of the VIN residual module is as follows: x'=Conv(x)•(1+scale)+bias; x”=conv(x')+CrossAtt(x,V out ); x out = x”+x; In the formula, Conv represents the basic convolution module operation, CrossAtt represents the cross-attention computation unit, the weight scale and bias value are obtained by chunking after time step encoding, and x out This indicates the output of the VIN residual module.

[0023] Specifically, the convolutional sampling layer consists of a ker×ker kernel with a stride of stride1 during downsampling and a nearest neighbor interpolation algorithm and a ker2×ker2 kernel with a stride of stride2 during upsampling. In this embodiment, the standard convolution of the downsampling layer uses a 3×3 kernel with a stride of 2 and padding of 1; the standard convolution of the upsampling layer uses a 1×1 kernel with a stride of 1.

[0024] Input image z t First, a standard convolutional layer reduces the spatial fraction to decrease computation in subsequent operations, while continuously converting rich low-level image details into high-level semantic information. Then, after three feature extraction layers and a depth downsampling unit consisting of two basic residual units, the encoder output Uneen is obtained. The encoder output Uneen is then passed through an attention alignment layer consisting of a Transformer layer and two basic residual units, and concatenated with the original encoder output Uneen to serve as the input to the decoder. The decoder consists of a depth upsampling unit, three decoding layers, and a basic convolutional module. The depth upsampling unit includes one basic residual unit and one basic convolutional unit; the basic convolutional module includes a grouping normalization unit, a SiLU activation function, and a basic convolution operation.

[0025] In this application, a diffusion model for generating pseudo-benchmark images is constructed by improving the existing DDPM (Denoising Diffusion Probabilistic Models) model. When constructing the noise prediction network, a VIN residual module is incorporated into both the upsampling and downsampling processes. The VIN residual module utilizes V... out The data-guided noise prediction network learns image features of VIN codes that vary depending on the vehicle manufacturer, vehicle type, brand, model, and year of manufacture.

[0026] To ensure that the diffusion model can generate effective pseudo-reference images, this application uses vehicle fusion information encoding V. out As guiding information, it is fed into the diffusion model. Because the U-Net denoising model learns a large number of real data distribution features during training, in the process of generating pseudo-benchmark photos, the U-Net denoising model comprehensively utilizes the text data of the VIN code to be identified and the V... out The corresponding VIN code graphic features can effectively generate pseudo-reference images that are consistent with standard reference images. Through the learning and inference process of the noise prediction network, it is ensured that the pseudo-reference images generated in this application can be used as comparison references just like real reference images.

[0027] U-Net is used as the noise prediction model. In each training step, based on a pre-built training dataset, original vehicle identification number (VIN) electronic data images are randomly sampled, then uniformly randomized for time step t, and standard Gaussian noise is generated. The model receives the pre-processed VIN image Z with random noise. t Vehicle fusion information coding V out The time step t is used as input; during the inference phase, DDIM is used as a sampler to speed up the model's inference.

[0028] The diffusion model includes a noise addition process and a noise removal process; the noise addition process is as follows: ; In the formula, z t This represents the image with added noise at time t; α and β are hyperparameters; εt is the noise added at time t, which follows a Gaussian distribution; ; α t 2 +β t 2 =1; T is the preset maximum time step for the noise addition process, which is set to 1000 in this embodiment.

[0029] In fact, the diffusion process does not necessarily have to follow a recursive relationship; the state at a specified time step can be obtained in one step. The calculation process is as follows: ; In the formula, ; .

[0030] The noise reduction process is as follows: ; In the formula, σ t Let represent the variance, and k represent the sampled values ​​from the same distribution. When t=1, k=0. ε θ (z t ,t) represents the noise distribution output by the noise prediction network at time step t; θ is the parameter of the noise prediction network, indicating that the predicted noise is related to the model.

[0031] During training, a mean squared error (MSE) loss function is constructed based on the added Gaussian random noise and the noise distribution predicted by U-Net. The Adam optimizer is used to optimize the network parameters using gradient descent, and the loss gradient is calculated through backpropagation to update the network parameters to minimize the difference between the predicted noise and the actual noise. A maximum number of training epochs is set, and the convergence of the loss on the validation set is monitored.

[0032] A mean squared error (MSE) loss function is constructed based on the real-added Gaussian random noise and the noise distribution predicted by U-Net. The Adam optimizer is used to optimize the network parameters using gradient descent. The loss gradient is calculated using the backpropagation algorithm to update the network parameters to minimize the difference between the predicted noise and the real noise. A maximum number of training epochs is set, and the convergence of the validation set loss is monitored. When the validation loss change is less than a threshold or the maximum number of training epochs is reached for 20 consecutive epochs, training is stopped and the optimal model parameters are saved.

[0033] In the inference phase of the diffusion model, DDIM is used as a sampler to accelerate the inference speed. A deterministic sampling strategy using DDIM replaces the random sampling of DDPM. DDIM compresses the original T-step denoising process into T' steps through skip sampling. Given initial standard Gaussian random noise, target VIN code guidance information, and sampling time step sequence, conditional denoising is performed using a trained U-Net noise prediction network. Noise is progressively removed in each sampling step, ultimately generating a high-quality vehicle identification number reference image.

[0034] This application uses a diffusion model to learn the distribution characteristics of a large number of real datasets and combines vehicle identification number (VIN) information encoding as a guiding condition to improve the accuracy of image generation during the denoising process. It also uses the DDIM sampling method to accelerate the inference process, achieving fast and high-quality generation and solving the problem of incomplete historical archive photo databases of VINs, which makes it impossible for inspectors to verify the information.

[0035] S3: Construct a character recognition network; the character recognition network outputs the type and coordinates of each character in the input image.

[0036] like Figure 5 As shown, the character recognition network includes a feature extraction network and a character segmentation network connected in sequence.

[0037] The feature extraction network is built based on the HRNetV2 model. After extracting features from the input image, the extracted features are fed into the character segmentation network.

[0038] This application improves upon the concept of the convolutional character network CharNet to construct a character recognition network. Based on the HRNetV2 model, a feature extraction network is built within the character recognition network. A multi-scale parallel structure is used to extract features from the input image, reducing scale loss and ensuring that the extracted features better represent the original image, thereby improving the accuracy of the final comparison results. HRNetV2 includes a stem downsampling layer and i multi-scale feature fusion layers. Each stage consists of i parallel multi-resolution sub-networks. Each multi-resolution sub-network consists of multiple parallel convolutional branches, each processing a feature map at a specific resolution to obtain the sub-network output M at that resolution. j i M j i Let represent the output of the j-th subnetwork in the i-th stage, where j≤i.

[0039] The parallel convolutional branches in HRNetV2 consist of Conv convolutional layers, Batch Normalization (BN) layers, ReLU activation function layers, Conv convolutional layers, and BN layers in sequence. The input N to the j-th subnetwork in the (i+1)-th stage is... j i+1 The set of outputs of all subnetworks in the i-th stage {M 1 i M 2 i ,…,M i i After cross-resolution fusion, they are aligned and formed separately.

[0040] The character segmentation network consists of three parallel task branches: a discrimination task branch, a recognition task branch, and a detection task branch.

[0041] The discrimination task branch includes: K k1×k1 convolutional layers and one k2×k2 convolutional layer; the discrimination task branch uses a binary mask image as supervision and outputs a two-channel text instance segmentation feature map, predicting the probability of text presence or absence at each pixel; in this embodiment, k1 is 3, k2 is 1, and K is 4. The recognition task branch includes: K k1×k1 convolutional layers and one k2×k2 convolutional layer; the recognition task branch outputs a C-channel category prediction feature map, where C represents all possible categories of the vehicle identification code character, and predicts the probability that the character at that position is each character for each pixel; The detection task branch includes: K+1 k1×k1 convolutional layers and one k2×k2 convolutional layer; the detection task branch adopts the design ideas of one-stage object detection networks such as YOLO and RetinaNet, and outputs a coordinate position feature map with 4 channels to describe the distance from the pixel position to the four sides of the predicted character rectangle.

[0042] In this application, a vehicle identification number (VIN) character recognition network is constructed. The original VIN photo and the reference image for comparison are preprocessed and data augmented to form a standard input image. Based on the output of the character recognition network, the coordinate position and character category of each character on the original image are obtained.

[0043] This application proposes a feature extraction layer for a character recognition network that can extract rich features with strong semantic information. Considering the fine structure of characters, the feature extraction layer can preserve details and spatial information to the maximum extent, maintain high-resolution representation throughout the process, improve the precision of character localization, and provide support for subsequent character position discrimination and recognition.

[0044] S4: Obtain the VIN image to be compared and identify the vehicle identification number in the image, denoted as: VIN code to be confirmed. Specifically, the VIN image to be compared is a 1:1 photograph of the vehicle identification number to be compared, and the VIN image to be compared is obtained using optical character recognition technology.

[0045] S5: Retrieve the corresponding baseline photo from the historical database based on the VIN code to be confirmed; If a reference image is found in the database, and the reference image is used as a reference image for comparison, then proceed to step S7; If the baseline image cannot be retrieved, proceed to step S6.

[0046] S6: Using the pre-built diffusion model, generate a pseudo-reference photo for the VIN code image to be compared; Use the pseudo-reference photograph as a reference image for comparison, and proceed to step S7.

[0047] S7: By comparing the VIN code image to be compared with the reference image, the VIN code suspected of being counterfeited is located.

[0048] Step S7 specifically includes the following steps: a1: Input the VIN code image to be compared into a pre-trained character recognition network to obtain the text data and coordinates of the character to be compared; a2: The reference image is input into the pre-trained character recognition network to obtain the text data of the reference character and the reference coordinates; a3: Confirm whether the character content included in the character to be compared and the comparison reference character are consistent; If they match, proceed to step a4; Otherwise, if the character content of the two is inconsistent, proceed to step a5; a4: Perform multi-dimensional comparison between the VIN code image to be compared and the reference image used for comparison; Multidimensional features include: structural features, texture features, geometric features, and edge features; If the two are similar, it means that the VIN code image to be compared has passed the comparison operation, which means that the original vehicle identification number photo has passed the verification and the vehicle identification number has a low probability of being forged or tampered with. Otherwise, if the two are not similar, it indicates that the VIN code image to be compared is suspected of being forged, and step a5 is executed; a5: Submit VIN code images suspected of being forged for further manual processing.

[0049] Step a4 specifically includes the following steps.

[0050] b1: The VIN code image to be compared and the reference image for comparison are respectively fed into the character recognition network as input images.

[0051] b2: For each input image, obtain the output of the discrimination task branch of the character recognition network, use the non-maximum suppression criterion to select the classification results of the default box, predict the character category corresponding to each character coordinate region, and obtain the character text data corresponding to the input image; Specifically, the output p of the discrimination task branch of the character recognition network is filtered out to remove character bounding boxes with a confidence level less than a set threshold m. The remaining character bounding boxes are then subjected to Non-Maximum Suppression (NMS) post-processing to remove redundant bounding boxes, thus obtaining the coordinates of all characters. Finally, the softmax function is used to predict the character category corresponding to each character coordinate region, thereby obtaining the character text data corresponding to the input image.

[0052] b3: Extract the character image set {Z1, Z2, ..., Z...} from the VIN code image to be compared, by cropping all the characters included in the image. n}; On the reference image used for comparison, all characters included in it are cropped to obtain the character image set {W1, W2, ..., W}. n}

[0053] b4: Pair the two character image sets one by one in front-to-back order to obtain {Z k W k The paired character image group is binarized to obtain the binarized character image group {Z'}. k W' k}

[0054] b5: For each group of binary character images {Z' k W' k Multi-dimensional feature values ​​were extracted from the images in the table.

[0055] b6: Based on each group of binary character images {Z' k W' k The similarity of the corresponding multi-dimensional feature values ​​is used to calculate the overall consistency score.

[0056] The specific similarity calculation method for each dimension of feature value can be implemented based on existing technologies. In this embodiment, structural similarity is represented by the Structure Similarity Index Measure (SSIM), which is an index used to measure the similarity between two images, mainly considering brightness, contrast, and structural information. Texture similarity is represented by the Gray-Level Co-occurrence Matrix (GLCM), a statistical method used in image processing to analyze the spatial relationship between pixel intensities in grayscale images. Cosine similarity is used to compare the texture features of the images to obtain texture similarity values. Geometric similarity is calculated using the SIFT (Scale-Invariant Feature Transform) algorithm. SIFT extracts scale- and rotation-invariant keypoints from the image and generates discriminative descriptive vectors. The SIFT feature extractor detects keypoints in the grayscale image and calculates the corresponding 128-dimensional descriptors. A FLANN matcher is configured, and KNN is used to match each descriptor in the two images. The geometric similarity calculation method is the proportion of the number of matches to the smaller of the number of keypoints in the two images. Edge similarity is calculated based on the Canny algorithm, which extracts edge features from the image and calculates the overall edge feature similarity based on the similarity of edge feature overlap, distance transformation similarity, and orientation similarity.

[0057] Calculation yields {Z' k W' k After scoring the similarity of each feature value in each dimension, a weighted average method is used to calculate the overall consistency score.

[0058] b7: Constructing a partial anomaly detection model based on isolated forests; The input to the partial anomaly detection model is the group of all binary character images {Z'} corresponding to the VIN code image to be compared. k W' kThe output is a group of character images that are different from other binarized character image groups, based on all multi-dimensional feature values ​​and consistency scores.

[0059] b8: Convert each group of binary character images {Z' k W' k The corresponding consistency score is compared with the preset similarity threshold. If the similarity threshold is less than 1, then this group of characters is identified as a suspect image group, the VIN code image to be compared is taken as a suspect image, and step a5 is executed. Otherwise, if the overall consistency score is greater than or equal to the similarity threshold, proceed to step b9.

[0060] b9: Confirm whether the overall consistency score of the VIN code images to be compared is greater than or equal to the similarity threshold; If so, proceed to step b10; If not, then stop this operation.

[0061] b10: Group all binarized character images corresponding to the VIN code image to be compared into {Z' k W' k All multi-dimensional feature values ​​and consistency scores are simultaneously fed into the pre-trained partial anomaly detection model. If the output of the partial anomaly detection model matches the abnormal character image group, then the VIN code image to be compared is taken as the suspect image, and step a5 is executed.

[0062] This application proposes a multi-dimensional character comparison algorithm for the comparison process between VIN code images and reference images. This algorithm is highly interpretable and, compared to deep learning-based algorithms, requires no additional sample collection and training. It can accurately calculate the pixel differences of vehicle identification number characters, achieving precise comparison while maintaining both accuracy and efficiency. In this application, during the comparison process, not only are similarity thresholds used to confirm the features of all characters in the VIN code images, but a partial anomaly detection model is also constructed based on isolated forests. Even if all characters in the VIN code images pass the similarity threshold, if some characters in the VIN code images have abnormal pixels that are inconsistent with the comparison results of other binary character image groups, these can be identified through horizontal comparison using the partial anomaly detection model. That is, if there are some VIN characters with a high degree of forgery and similarity to the reference image, they can also be identified through horizontal comparison using the partial anomaly detection model. The comparison algorithm designed in this application covers more real-world scenarios, effectively improving the detection accuracy.

[0063] By using the technical solution of this invention, the shortcomings of existing technologies, such as the lack of reference photos for comparison, can be effectively avoided. The vehicle identification number (VIN) photo comparison method based on the diffusion model and character recognition network of this application can generate an ideal VIN image as a reference photo for VIN comparison even when a corresponding VIN image does not exist in historical archives. It can also accurately segment the character regions in the VIN photo and identify the characters, and can compare and analyze the similarity between characters and photos from multiple dimensions, pointing out suspicious characters in the original VIN photo and providing a reference for inspectors. This application utilizes popular technical solutions in the field of computer vision to realize a complete generation, recognition, and comparison technology chain for VIN comparison tasks, closely aligning with the actual situation of motor vehicle inspection business and meeting the real-time and accuracy requirements of the application.

Claims

1. A method for comparing vehicle identification number photos based on a character recognition network, characterized in that, It includes the following steps: S1: Construct a feature extraction model; The feature extraction model includes: a VIN semantic information encoder, a vehicle basic information encoder, and a fusion coding layer; The input to the VIN semantic information encoder is VIN code text data, and the output is the global VIN semantic information encoding V. global ; The input to the vehicle basic information encoder is text data of vehicle basic information, and the output is a feature vector corresponding to each vehicle basic information. The basic vehicle information includes: vehicle manufacturer, vehicle type, brand and model, and year of manufacture; Encode the global VIN semantic information V global All vehicle basic information feature vectors are input into the fusion coding layer. The fusion coding layer concatenates all vehicle basic information feature vectors to obtain concatenated vehicle information features. Then, based on cross-attention, the VIN semantic information features and the concatenated vehicle information features are fused to output the vehicle fusion information code V. out ; S2: Construct a diffusion model; The input to the diffusion model is the vehicle fusion information code V. out and initialize standard Gaussian random noise images; The diffusion model uses vehicle fusion information encoding V out To guide the information, a trained noise prediction network is used to denoise the initial standard Gaussian random noise image, and finally a pseudo benchmark image is generated and output. S3: Construct a character recognition network; The character recognition network outputs the type and coordinates of each character in the input image. S4: Obtain the image of the VIN code to be compared, and identify the vehicle identification number in the image, which is recorded as: VIN code to be confirmed; S5: Retrieve the corresponding baseline photo from the historical database based on the VIN code to be confirmed; If a reference image is found in the database, and the reference image is used as a reference image for comparison, then proceed to step S7; If the reference image cannot be retrieved, proceed to step S6; S6: Using the pre-built diffusion model, generate a pseudo-reference photo for the VIN code image to be compared; Use the pseudo-reference photograph as a reference image for comparison, and proceed to step S7; S7: By comparing the VIN code image to be compared with the reference image for comparison, the VIN code suspected of being counterfeited is located.

2. The method for comparing vehicle identification number photos based on a character recognition network according to claim 1, characterized in that: The VIN semantic information encoder includes: a character encoding layer, an embedding layer, a Transformer encoding layer, and a global pooling layer; The character encoding layer performs one-hot encoding on the VIN information to obtain the VIN character encoding set V, with a tensor format of B×L×N; Where B is the batch size, L is the length of the vehicle identification number, and N is the total number of digits and letters in the mapped vehicle identification number; The embedding layer comprises two independent fully connected MLP layers: a character embedding encoding layer and a positional embedding encoding layer, which perform character and positional embedding encoding respectively. For each input VIN text information, after processing by the character embedding encoding layer and the positional embedding encoding layer, a character embedding tensor V is obtained. char and position embedding tensor V pos ; The TransFormer coding layer contains n multi-head attention heads of dimension dim, a feedforward neural network consisting of two fully connected layers, and an LN normalization layer. The character embedding tensor V char and the position embedding tensor V pos After concat processing, it is used as the input V of the TransFormer encoding layer. con We obtain the TransFormer tensor V. trans The calculation formula is as follows: ； In the formula, W Q T W K T and W V T These represent the linear transformation parameters of the query, key, and value, respectively; D is the input V. con The number of channels; FFN represents a feedforward neural network; The global pooling layer performs average pooling on all character information and then obtains the final global VIN semantic information encoding V through linear mapping. global .

3. The method for comparing vehicle identification number photos based on a character recognition network according to claim 1, characterized in that: The vehicle basic information encoder includes: a vehicle manufacturer encoding layer, a vehicle type encoding layer, a brand and model encoding layer, and a manufacturing year encoding layer; the vehicle manufacturer encoding layer assigns an index value to each manufacturer, and uses an embedding layer to map the index value to a vehicle manufacturer feature vector Veh1; the vehicle type encoding layer assigns an index value to each vehicle type, and uses an embedding layer to map the index value to a vehicle type feature vector Veh2. The brand model encoding layer uses the Word2Vec model to calculate the brand model vector, and then adjusts the dimensions through an MLP layer to obtain the brand model feature vector Veh3. The manufacturing year encoding layer normalizes the year as a tensor value and finally obtains the vehicle year feature vector Veh4 through an MLP layer.

4. The method for comparing vehicle identification number photos based on a character recognition network according to claim 1, characterized in that: The noise prediction network is constructed based on the U-Net network model; The noise prediction network includes a feature extraction module, an image super-resolution module, and an attention alignment layer; The feature extraction module includes a basic convolutional module 1, a residual fusion downsampling layer, and a depth downsampling unit connected in sequence. The image super-resolution module includes a depth upsampling unit, a residual fusion upsampling layer and a basic convolution module 2 connected in sequence. The feature extraction module and the image super-resolution module are arranged symmetrically on the left and right, and the number of residual fusion downsampling layers and residual fusion upsampling layers are equal and correspond one-to-one. The basic convolution module 1 is located at the top of the feature extraction module as an input module, and the basic convolution module 2 is located at the top of the image super-resolution module as an output module. The depth downsampling unit is located at the bottom of the feature extraction module, and the depth upsampling unit is located at the bottom of the image super-resolution module. The output of the depth downsampling unit is processed by the attention alignment layer and then sent to the depth upsampling unit. At the same time, the feature map from the downsampling process of the depth downsampling unit is fused during the upsampling process of the depth upsampling unit. The residual fusion downsampling layer includes: a VIN residual module and a convolutional downsampling layer; The residual fusion upsampling layer includes a VIN residual module and a convolutional upsampling layer. The convolutional upsampling layer in the residual fusion upsampling layer fuses the feature maps from the same level of the convolutional downsampling layer during the upsampling process. The VIN residual module includes a time-step fusion module and a cross-attention calculation module; the time-step fusion module converts the time step t into conditional vector weights (scale) and biases (bias) on the input feature map z. t The time step and VIN semantic information are fused across all channels; the calculation process of the VIN residual module is as follows: x'=Conv(x)•(1+scale)+bias; x”=conv(x’)+CrossAtt(x,V out ); x out = x”+x; In the formula, Conv represents the basic convolution module operation, CrossAtt represents the cross-attention computation unit, the weight scale and bias value are obtained by chunking after time step encoding, and x out This indicates the output of the VIN residual module.

5. The method for comparing vehicle identification number photos based on a character recognition network according to claim 4, characterized in that: During the training process, the noise prediction network randomly samples original vehicle identification code electronic data images based on a pre-built training dataset, then uniformly and randomly samples time steps t, and generates standard Gaussian noise.

6. The method for comparing vehicle identification number photos based on a character recognition network according to claim 4, characterized in that: The diffusion model includes: a noise addition process and a noise reduction process; The noise addition process is as follows: ； In the formula, z t Represents the image with added noise at time t; α and β are hyperparameters; ε t The noise added at time t follows a Gaussian distribution; α t 2 +β t 2 =1; T is the preset maximum time step for the noise addition process; The noise reduction process is as follows: ； In the formula, σ t Let t represent the variance, and k represent the sampled values ​​from the same distribution; when t=1, k=0. ε θ (z t ,t) represents the noise distribution output by the noise prediction network at time step t, and θ is the parameter of the noise prediction network; 。 7. The method for comparing vehicle identification number photos based on a character recognition network according to claim 1, characterized in that: In step S3, the character recognition network includes: a feature extraction network and a character segmentation network connected in sequence; The feature extraction network is built based on the HRNetV2 model. After extracting features from the input image, the feature extraction network sends the extracted features to the character segmentation network. The character segmentation network includes three parallel task branches: a discrimination task branch, a recognition task branch, and a detection task branch. The discrimination task branch includes: K k1×k1 convolutional layers and one k2×k2 convolutional layer; the discrimination task branch uses a binary mask image as supervision, outputs a two-channel text instance segmentation feature map, and predicts the probability of text presence or absence at each pixel position; The recognition task branch includes: K k1×k1 convolutional layers and one k2×k2 convolutional layer; the recognition task branch outputs a C-channel category prediction feature map, where C represents all possible categories of the vehicle identification code character, and predicts the probability that the character at each pixel is a character. The detection task branch includes: K+1 k1×k1 convolutional layers and one k2×k2 convolutional layer; the detection task branch outputs a coordinate position feature map with 4 channels to describe the distance from the pixel position to the four sides of the predicted character rectangle.

8. The method for comparing vehicle identification number photos based on a character recognition network according to claim 7, characterized in that: Step S7 specifically includes the following steps: a1: Input the VIN code image to be compared into a pre-trained character recognition network to obtain the text data and coordinates of the character to be compared; a2: The reference image is input into a pre-trained character recognition network to obtain the text data of the reference character and the reference coordinates. a3: Confirm whether the character content included in the character to be compared and the comparison reference character is consistent; If they match, proceed to step a4; Otherwise, if the character content of the two is inconsistent, proceed to step a5; a4: Perform a multi-dimensional comparison between the VIN code image to be compared and the reference image used for comparison; The multi-dimensional features include: structural features, texture features, geometric features, and edge features; If the two are similar, it means that the VIN code image to be compared has passed this comparison operation; Otherwise, if the two are not similar, it indicates that the VIN code image to be compared is suspected of being forged, and then step a5 is executed; a5: Submit VIN code images suspected of being forged for further manual processing.

9. The method for comparing vehicle identification number photos based on a character recognition network according to claim 8, characterized in that: Step a4 specifically includes the following steps: b1: The VIN code image to be compared and the reference image for comparison are respectively sent as input images to the character recognition network; b2: For each input image, obtain the output of the discrimination task branch of the character recognition network, use the non-maximum suppression criterion to select the classification results of the default box, predict the character category corresponding to each character coordinate region, and obtain the character text data corresponding to the input image; b3: Extract the character image set {Z1, Z2, ..., Z...} from the VIN code image to be compared, by cropping all the characters included in the image. n }; On the reference image used for comparison, all characters included in it are cropped to obtain a character image set {W1, W2, ..., W}. n }; b4: Pair the two character image sets one by one in front-to-back order to obtain {Z k W k The paired character image group is binarized to obtain the binarized character image group {Z'}. k W' k }; b5: For each group of binary character images {Z' k W' k Extract multi-dimensional feature values ​​from the images in the table. b6: Based on each group of binary character images {Z' k W' k The similarity calculation of the corresponding multi-dimensional feature values ​​yields a comprehensive consistency score; b7: Constructing a partial anomaly detection model based on isolated forests; The input to the partial anomaly detection model is the group of all binary character images {Z'} corresponding to the VIN code image to be compared. k W' k The output of the combined score of all the multi-dimensional feature values ​​and consistency is the character image group that is different from other binarized character image groups based on similarity features. b8: Convert each group of binary character images {Z' k W' k The corresponding consistency score is compared with the preset similarity threshold. If the similarity threshold is less than 1, then this group of characters is identified as a suspected image group, and the VIN code image to be compared is taken as a suspected image, and step a5 is executed. Otherwise, if the overall consistency score is greater than or equal to the similarity threshold, proceed to step b9; b9: Confirm whether the overall consistency score corresponding to the VIN code images to be compared is greater than or equal to the similarity threshold; If so, proceed to step b10; If not, stop this operation; b10: Group all binarized character images {Z' corresponding to the VIN code image to be compared. k W' k All the multi-dimensional feature values ​​and consistency scores mentioned above are simultaneously fed into the trained partial anomaly detection model. If the output of the partial anomaly detection model matches the abnormal character image group, then the VIN code image to be compared is taken as the suspect image, and step a5 is executed.

Images