License Plate Image Generation Method and System Based on Adaptive Diffusion Prior Variational Autoencoder

Abstract

The present invention discloses a license plate image generation method and system based on an adaptive diffusion prior variational autoencoder. The method includes: S1. Collecting a license plate image data set; S2. Constructing a variational autoencoder network with an adaptive diffusion prior, and determining the character classes and character domains for generating license plate images; S3. Inputting the training data into the encoding network of the variational autoencoder to obtain a posterior distribution; S4. Sampling the data and inputting it into the forward process of the diffusion model, adaptively adjusting the diffusion time to obtain latent variables; S5. Inputting the latent variables into the score network to obtain the scores of the latent variables at different diffusion times; S6. Determining the target character domain and target character classes, and decoding to obtain the image data reconstructed by the variational autoencoder; S7. Solving the reconstruction loss and the KL divergence distance loss between the prior distribution and the posterior distribution, updating and training the parameters of the variational autoencoder and the diffusion model, and testing to generate images; S8. Saving the optimal model weights and generating image data using the trained model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and relates to a method and system for generating license plate images, specifically a method and system for generating license plate images based on an adaptive diffusion prior variational autoencoder. Background Art

[0002] With the rapid development of deep learning technology, end-to-end license plate recognition algorithms based on neural networks have gradually replaced license plate recognition algorithms based on traditional machine learning. License plate recognition algorithms based on neural networks have many advantages, such as good robustness, high accuracy, fast processing speed, etc., but there are also some disadvantages. For example, they need to collect a large number of license plate annotation samples for training, resulting in high labor costs for data annotation and poor data quality. In addition, conventional license plate images are mainly collected by cameras and then annotated. However, the license plate images collected by cameras generally have a single type and uneven distribution of characters, resulting in the license plate recognition model training being prone to bias towards common samples. If such license plate images are used to train the license plate recognition model, the robustness of the license plate recognition model will be very poor.

[0003] In order to obtain license plate annotation data that meets the training requirements, a large amount of time, funds, and manpower are required for collection and annotation work. First, there are many types of license plates, including small license plates, large license plates, large double-layer license plates, new energy license plates, military license plates, double-layer military license plates, etc. Relatively rare license plates increase the difficulty of sample collection work. Second, the license plate content also varies in each province and city, and the same repeated collection needs to be carried out for each region.

[0004] In addition, in the solutions provided by the prior art, in order to improve the recognition rate of the license plate recognition model, the general method is to purposefully increase the training data set. However, the efficiency of manually collecting training data is low and the quantity is very limited, while the data synthesized by software, although meeting the quantity requirements, has a large difference from the real data and is difficult to be used for the training of the license plate recognition model. How to efficiently obtain a large amount of high-quality data has become an urgent problem to be solved. Summary of the Invention

[0005] In view of the above problems existing in the prior art, the present invention proposes a method and system for generating license plate images based on an adaptive diffusion prior variational autoencoder.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for generating license plate images based on an adaptive diffusion prior variational autoencoder, comprising the following steps:

[0008] S1. Collect and organize the original license plate image dataset; preferably, it can be further divided into a training set and a test set;

[0009] S2. Construct a variational autoencoder network with an adaptive diffusion prior, including an encoder network, a decoder network, and an adaptive diffusion prior network, and determine the character classes and character domains for constructing the generated license plate images. The character classes represent the character categories of the characters in the license plate images, and the character domains represent the character positions of the characters in the license plate images; construct and output the generated license plate images based on the image features, character classes, and character domains extracted by the variational autoencoder network;

[0010] S3. Input several batches of set training data into the encoding network of the variational autoencoder to obtain the posterior distribution, where the sample license plate images input into the encoding network contain the original character domains and original character classes;

[0011] S4. Sample data from the obtained posterior distribution and input it into the forward process of the diffusion model. According to the forward sampling adjustment process, adaptively adjust the diffusion time so that the sampled data undergoes adaptive diffusion within the range of τ~1 (0 < τ < 1)) to obtain the noise-added latent variables;

[0012] S5. Input the noise-added latent variables into the score network to obtain the scores of the latent variables at different diffusion times, that is, the gradients after taking the logarithm of the noise-added latent variable distribution;

[0013] S6. Determine the target character domain and target character class, and input the target character domain, target character class, and the sampled data of the posterior distribution into the decoder to decode and obtain the image data reconstructed by the variational autoencoder. The reconstructed image data contains the target character domain and target character class;

[0014] S7. Use the sampled data output by the encoder and the reconstructed image data of the decoder to obtain the reconstruction loss; use the mean square error of the gradients after taking the logarithm of the noise-added latent variable distribution and the output of the score network to obtain the distance loss between the prior distribution and the posterior distribution, and update and train the parameters of the variational autoencoder and the diffusion model and test the generated images;

[0015] S8. Save the optimal model weights in the test, and use the trained model to generate unconditional image data.

[0016] Further, in step S1, collect and organize various license plate image datasets D, and divide them into a training set D train and a test set D test in a certain proportion; the categories of the collected license plate images include but are not limited to the following: license plates of different vehicle models, license plates for different purposes, license plates from different regions, etc.

[0017] Further, in step S2, the parameters in the constructed variational autoencoder network include the network input image size C×H×W, the batch size is B, and the latent variable dimension is D latent , the encoder network parameters are φ, the decoder network parameters are ψ, the diffusion model network parameters are θ, and the latent variable is Z.

[0018] Further, in step S3, the training data set of several batches of license plate images set is input into the encoding network of the variational autoencoder, where the input training data includes the image samples X = {x1, x2, …, x N}, where x i is the i-th sample in the current batch, i = 1, 2, … B.

[0019] Further, the data output by the encoding network is the posterior distribution of the latent variable before noise addition, that is, q φ (z|x).

[0020] Further, in step S4, after obtaining the clean latent variable, the latent variable can be diffused and noise-added according to the rules of the diffusion model. Define the forward diffusion process of the continuous time variable t ∈ [0, 1] Assume that this diffusion process does not start from time 0, but starts to diffuse from a certain τ moment, that is, z τ is the starting variable, z t is the perturbation at time t, and the purpose of the diffusion process is to gradually map z τ to a multi-dimensional normal distribution through a Markov chain, that is

[0021]

[0022] where T is the total number of diffusion steps, and q(z t |z t-1 ) is defined as a Gaussian distribution related to the constant β t I is the identity matrix. Let α t := 1 - β t , It can be obtained that:

[0023]

[0024] From the above formula, the latent variable at the τ moment can be perturbed to obtain the noise-added latent variable at any moment within the range of 0 to 1.

[0025] ​Furthermore, in step S4, when adjusting the forward diffusion process, using the same step size for all samples in a batch of data samples may cause samples that could originally converge faster to converge more slowly. Since the forward diffusion process of each image is independent, different diffusion step sizes are adopted for different samples. By using the KL divergence to measure the distance between the distribution at time τ and the distribution at time 1, the KL divergence at time τ in batch b is denoted as KL b , and the KL divergence at time τ corresponding to batch b + 1 is denoted as KL b+1 :

[0026]

[0027]

[0028] Among them, represents the distribution at time τ corresponding to batch b, represents the distribution at time τ corresponding to batch b + 1.

[0029] Furthermore, when KL b < KL b+1 or KL b > KL b+1 , the corresponding diffusion time update formula is:

[0030] τ b+1 = τ b + α·Δτ

[0031] In the above formula, α represents the diffusion time update step size, Δτ represents the diffusion time change amount. In the range of τ b ∈(0, 1), Δτ is usually taken as 0.01 to update the time τ b+1 . The calculation method of α is as follows:

[0032]

[0033] In the above formula, λ represents the direction control bit for forward or backward diffusion of the diffusion time, and β represents the change speed of α with KL b+1 .

[0034] Furthermore, in step S5, the score of the diffusion latent variable is:

[0035]

[0036] Among them, s represents the score network, θ is the parameter of this network, and the score network is generally parameterized by the Unet model.

[0037] Further, in step S6, determine the target character domain and the target character class, and feed the sampled data, the target character domain, and the target character class into the decoder network for decoding. Denote the decoded data distribution as p θ (x|z), and the reconstructed image data includes the target character domain and the target character class.

[0038] Further, in step S7, using the reconstruction loss and the KL divergence between the prior distribution and the posterior distribution, the total loss of this model can be obtained:

[0039]

[0040] Further, the third term in the above formula is the cross-entropy term between the posterior distribution and the prior distribution. In the diffusion framework, the following can be obtained:

[0041]

[0042] where D is a constant in the above formula.

[0043] Further, according to the L(x, φ, θ, ψ) formula, a variational autoencoder based on an adaptive diffusion prior can be trained, and the model parameters are iteratively updated. Every time a specified number of iterations is reached, an image generation is performed. The test process is as follows:

[0044] During the generation process, add a reverse sampling adjustment process. Through this process, select a suitable prior distribution at the τ moment. According to this prior, perform sampling, and at the same time input the target character domain and the target character class into the decoder for decoding to generate an image.

[0045] Further, during the reverse sampling process, perform reverse sampling following the following formula:

[0046]

[0047] where is a standard Wiener process, dt is an infinitesimal negative time step, is given by the score network.

[0048] Further, according to the above sampling formula, starting from the last update moment in the forward diffusion process, sample the latent variable at the τ moment:

[0049] z τ = f(z1, τ) - g 2 (τ)s θ (z1, τ) + g(τ)ò

[0050] where the distribution of z1 is: p(z1) = N(z1; 0, I), and ò is Gaussian noise.

[0051] Furthermore, based on the gradient change during the τ±Δτ period in the reverse sampling process, the reverse diffusion step size can be dynamically adjusted. Define the latent variable sampling at time τ - Δτ as:

[0052] z τ-Δτ = f(z τ , τ - Δτ) - g 2 (τ - Δτ)s θ (z τ , τ - Δτ) + g(τ - Δτ)ò

[0053] Meanwhile, the latent variable at time τ + Δτ can be obtained according to the forward diffusion process:

[0054] z τ+Δτ = f(z τ , τ + Δτ) + g(z τ , τ + Δτ)ò

[0055] Thus, the change amount near τ±Δτ can be calculated, that is, the first-order difference component and the second-order difference component:

[0056] z′ τ = (z τ+Δτ - z τ ) / Δτ

[0057] z″ τ = (z τ+Δτ - 2z τ + z τ-Δτ ) / Δτ 2

[0058] Furthermore, from the above formula, the mixing tolerance can be calculated to perform time update:

[0059] δ(z′ τ , z′ τ(prev) ) = max(∈ abs , ∈ rel max(|z′ τ |, |z′ τ(prev) |))

[0060] where z′ τ(prev) represents the z′ obtained from the previous loop iteration calculation τ , and thus the local error amount can be calculated:

[0061]

[0062] Furthermore, the above formula calculates the L2 norm, and thus the following diffusion time update strategy can be given:

[0063]

[0064] Among them, θ is the safety factor. In the present invention, r adopts the measured value, and r ∈ [0.5, 1]. Through the above method, an optimal prior distribution can be selected, and the target character domain and target character class are input into the decoder together to decode the image.

[0065] Furthermore, through the above iterative training process, the optimal model weights are saved.

[0066] The present invention also discloses a license plate image generation system based on an adaptive diffusion prior variational autoencoder, which includes the following modules:

[0067] Dataset production module: Collect and organize the original license plate images captured by the camera, and further divide them into a training set and a test set;

[0068] Encoding network module: Input the training set data into the encoder network, encode the image data, and sample the encoded posterior distribution to obtain latent variable data;

[0069] Latent variable diffusion module: Diffuse the latent variable data according to the rules of the diffusion model to obtain the noisy latent variable data;

[0070] Forward diffusion time adjustment module: According to the KL divergence between the distribution at time τ and the distribution at time 1, adaptively adjust the diffusion step size for the next iteration;

[0071] Decoding network module: After sampling the posterior distribution obtained by the encoding network, input it into the decoding network together with the target character domain and target character class to decode and obtain the reconstructed image data;

[0072] Training module: End-to-end train the network model according to the total loss function of the model;

[0073] Optimal training weight retention module: Conduct tests and retain the optimal training weights;

[0074] License plate image generation module: In this module, select the reverse diffusion time according to the reverse diffusion adjustment process, and input the latent variable sampled reversely into the decoder for decoding to obtain the license plate image.

[0075] Compared with the prior art, the present invention is a license plate image generation method and system based on an adaptive diffusion prior variational autoencoder, which can update the existing license plate generation technology, adopt a new technical solution to generate license plate images, and can effectively solve the problems existing in the prior art, such as single license plate images, poor robustness of license plate recognition models, and high manual collection costs. At the same time, by using the adaptive diffusion prior model, the complexity of the prior distribution can be adaptively adjusted according to the posterior distribution, and a better optimal prior distribution can be generated. During the forward diffusion process, a forward diffusion adjustment process is used to adaptively adjust the diffusion step size. During the reverse sampling process, a reverse sampling adjustment process is used. By comparing the gradient changes before and after a certain period of time, an optimal diffusion moment is obtained. Finally, sampling is performed at this moment and input into the decoder for decoding to obtain the final generated license plate image. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 FIG. is a flowchart of a license plate image generation method based on an adaptive diffusion prior variational autoencoder according to Embodiment 1 of the present invention.

[0077] Figure 2 FIG. is a block diagram of an adaptive diffusion model structure in step S13 of Embodiment 1 of the present invention.

[0078] Figure 3 FIG. is a block diagram of a license plate image generation system based on an adaptive diffusion prior variational autoencoder according to Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] The following specifically illustrates the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0080] The purpose of the present invention is to provide a license plate image generation method and system based on an adaptive diffusion prior variational autoencoder for the defects of the prior art.

[0081] Embodiment 1

[0082] This embodiment provides a license plate image generation method based on an adaptive diffusion prior variational autoencoder, and its specific implementation process is as Figure 1 shown, including the following steps:

[0083] S11. Collect and organize the original license plate image data set, and further divide it into a training set and a test set;

[0084] S12. Construct a variational autoencoder with an adaptive diffusion prior, including an encoder network, a decoder network, and an adaptive diffusion prior network. The encoder network and the decoder network are implemented by sampling conventional deep convolutional networks. The adaptive diffusion prior network is implemented by improving a diffusion model, and the character classes and character domains for constructing license plate images are determined. The character classes represent the character categories of the characters in the license plate image, and the character domains represent the character positions of the characters in the license plate image. Construct and output the generated license plate image based on the image features, character classes, and character domains extracted by the variational autoencoder network.

[0085] S13. Input several batches of set training data into the encoder network of the variational autoencoder to obtain the posterior distribution, where the sample license plate images input into the encoding network contain the original character domains and original character classes.

[0086] S14. Sample data from the obtained posterior distribution and input it into the forward process of the diffusion model. As shown in, adaptively adjust the diffusion time according to the forward diffusion adjustment process, so that the sampled data undergoes adaptive diffusion in the range of τ~1 (0 < τ < 1) to obtain the noise-added latent variable data. Figure 2 as shown, adaptively adjust the diffusion time according to the forward diffusion adjustment process, so that the sampled data undergoes adaptive diffusion in the range of τ~1 (0 < τ < 1) to obtain the noise-added latent variable data.

[0087] S15. Input the noise-added latent variable data into the score network to obtain the scores of the latent variables at different diffusion times, that is, the gradients after taking the logarithm of the noise-added latent variable distribution.

[0088] S16. Determine the target character domain and target character class, and input the target character domain, target character class, and the sampled data of the posterior distribution into the decoder to decode and obtain the image data reconstructed by the variational autoencoder.

[0089] S17. Use the sampled data output by the encoder network and the reconstructed image data of the decoder to obtain the reconstruction loss. Use the mean square error of the gradient after taking the logarithm of the noise-added latent variable distribution and the output of the score network to obtain the distance loss between the prior distribution and the posterior distribution, and update and train the parameters of the variational autoencoder and the diffusion model and test the generated images.

[0090] S18. Save the optimal model weights in the model, and use the trained model to generate license plate images unconditionally.

[0091] Specific idea of this embodiment: 1. Collect and organize various license plate images, including license plates of different vehicle models, license plates for different purposes, etc., and then divide various images according to a ratio of 5:1 into a training set and a test set; 2. In the training stage, it is necessary to preprocess the data, including normalization and standardization, as well as image size transformation, image enhancement, etc. for the images; 3. Input several batches of set training data into the forward process of the diffusion model, and adaptively adjust the diffusion time according to the set forward sampling adjustment process, so that the sampled data is adaptively diffused within the range of τ~1 (0<τ<1) to obtain the noisy latent variable; 4. Input the noisy latent variable into the score network to obtain the scores of the latent variables at different diffusion moments, that is, the gradient after taking the logarithm of the noisy latent variable distribution; 5. The sampled data of the posterior distribution, the target character domain, and the image data reconstructed by the variational autoencoder decoded by the target character class are input into the decoder; 6. Use the output sampled data of the encoder network and the reconstructed image data of the decoder to obtain the reconstruction loss; the mean square error of the gradient after taking the logarithm of the noisy latent variable distribution and the output of the score network can be used to obtain the distance loss between the prior distribution and the posterior distribution. Thus, the total loss function of the model can be obtained. According to this loss function, the parameters of the variational autoencoder and the diffusion model can be updated and trained and tested to generate images; 7. After each iteration of the specified number of training sets, the model weights of the current training stage will be used to test in the test set images, and the network weights with the highest test image quality will be saved; 8. The unconditional license plate image generation can be carried out using the already trained network weight model.

[0092] The specific introduction of each step of this embodiment is as follows:

[0093] In step S11, collect and organize the license plate image dataset D of various types, and divide it into a training set D train and a test set D test .

[0094] In step S12, the training set D train and the test set D test are normalized and standardized using preprocessing, and their image sizes are transformed and image enhancement such as random cropping, color jitter, horizontal flipping, etc. The data enhancement method can be adjusted or changed according to specific license plate images.

[0095] In step S13, several batches of set training data are input into the encoder network of the variational autoencoder, and the training data input into the encoder network includes image samples X = {x1, x2, …, x N}, where x i is the i-th sample in the current batch, i = 1, 2, … B.

[0096] In step S14, after obtaining the posterior distribution q φ (z|x) of the unnoised latent variable, data is sampled from this distribution and input into the forward diffusion process of the diffusion model, where the forward diffusion process has a continuous time variable t ∈ [0, 1] And it is assumed that the diffusion start time in the present invention does not start from 0, but starts to diffuse from a certain τ time, that is, z τ is the starting variable, z t is the perturbation at time t, and the purpose of the diffusion process is to gradually map z τ onto a multi-dimensional normal distribution through a Markov chain, that is:

[0097]

[0098] where T is the total number of diffusion steps. To make the diffusion process relatively simple, q(z t |z t-1 ) is defined as a Gaussian distribution related to β t S t : = 1 - β t <000030> Then we can obtain:

[0099]

[0100] S

[0101] In the forward diffusion adjustment process, using the same step size for all samples may cause samples that could converge faster to converge more slowly. Since the forward diffusion process of each image is independent, different diffusion step sizes are sampled for different samples. By using the KL divergence to measure the distance between the distribution at time τ and the distribution at time 1, the KL divergence of the τ time of batch b is denoted as KL b , and the KL divergence of the τ time corresponding to batch b + 1 is denoted as KL b+1 :

[0102]

[0103]

[0104] where represents the distribution corresponding to batch b at time τ, represents the distribution corresponding to batch b + 1 at time τ. ​​

[0105] When KL b <KL b+1 or KL b >KL b+1 the corresponding diffusion time update formula is as follows:

[0106] τ b+1 =τ b +α·Δτ

[0107] In the above formula, α represents the diffusion time update step size, Δτ represents the diffusion time change amount. In the range of τ b ∈(0,1), Δτ is usually taken as 0.01, and τ b+1 is updated at this moment. The calculation method of α is as follows:

[0108]

[0109] In the above formula, λ represents the direction control bit for forward or backward diffusion of the diffusion time, and β represents the change speed of α with KL b+1

[0110] In step S15, the score of the diffusion latent variable can be obtained:

[0111]

[0112] where s represents the score network, θ is the parameter of this network, and the score network is generally parameterized by the Unet model.

[0113] In step S16, the data obtained by sampling, the target character domain, and the target character class are input into the decoder network for decoding, and the decoded data distribution is denoted as p θ (x∣z).

[0114] In step S17, using the reconstruction loss and the KL divergence between the prior distribution and the posterior distribution, the total loss of this model can be obtained as follows:

[0115]

[0116] Among them, the third term in the above formula is the cross-entropy term between the posterior distribution and the prior distribution. In the diffusion framework, the following can be obtained:

[0117]

[0118] Among them, in the above formula, D is a constant.

[0119] According to the L(x,φ,θ,ψ) formula, the variational autoencoder model based on the adaptive diffusion prior can be trained, and the model parameters are iteratively updated. Every time a specified number of iterations is performed, an image generation test is carried out. The test process is as follows:​

[0120] During the generation process, an inverse sampling adjustment process is added. Through this process, a prior distribution at an appropriate τ moment is selected. According to this prior, sampling is performed and input into the decoder for decoding to generate an image.

[0121] During the inverse sampling process, sampling is performed according to the following formula:

[0122]

[0123] where is a standard Wiener process, and dt is an infinitesimal negative time step. is given by the score network.

[0124] According to the above sampling formula, starting from the last update moment in the forward diffusion process, the latent variable at the τ moment is sampled:

[0125] z τ = f(z1, τ) - g 2 (τ)s θ (z1, τ) + g(τ)ò

[0126] where the distribution of z1 is: p(z1) = N(z1; 0, I), and ò is Gaussian noise.

[0127] According to the gradient change in the τ ± Δτ time period during the inverse sampling process, the inverse diffusion step size is dynamically adjusted. Thus, the latent variable sampling at the τ - Δτ moment can be given as:

[0128] z τ-Δτ = f(z τ , τ - Δτ) - g 2 (τ - Δτ)s θ (z τ , τ - Δτ) + g(τ - Δτ)ò

[0129] Meanwhile, the latent variable at the τ + Δτ moment can be obtained according to the forward diffusion process:

[0130] z τ+Δτ = f(z τ , τ + Δτ) + g(z τ , τ + Δτ)ò

[0131] Thus, the change amount near τ ± Δτ can be calculated, that is, the first-order difference component and the second-order difference component:

[0132] z′ τ = (z τ+Δτ - z τ ) / Δτ

[0133] z″ τ= (z τ+Δτ - 2z τ + z τ-Δτ ) / Δτ 2

[0134] The mixing tolerance can be calculated therefrom, and then the diffusion time is updated:

[0135] δ(z′ τ , z′ τ(prer) ) = max(∈ abs , ∈ rel max(|z′ τ |, |z′ τ(prev) |))

[0136] where z′ τ(prev) represents z′ obtained from the previous loop calculation τ , and the local error amount can be calculated therefrom:

[0137]

[0138] The above formula calculates the L2 norm, and the following diffusion time update strategy can be given:

[0139]

[0140] where θ is the safety factor. In the present invention, r adopts the measured value, and r ∈ [0.5, 1]. Through the above method, an optimal prior distribution can be selected and input into the decoder to decode the image.

[0141] In step S18, the optimal model weights are saved.

[0142] In step S17, after the complete model network result is constructed, the initial learning rate used by the diffusion model is set to 2×10 -4 and trained using the stochastic gradient descent optimizer, and the initial learning rate used by the variational autoencoder is set to 1×10 -4 and trained using the stochastic gradient descent optimizer, and the total number of diffusion steps used is 1000. During the training process, every 20 iterations of the training data set, a random selection is made from the test data set for testing, and common image quality evaluation metrics are selected for evaluation.

[0143] This embodiment proposes a license plate image generation method based on an adaptive diffusion prior variational autoencoder. First, various license plate datasets need to be collected and sorted to ensure the comprehensiveness and integrity of license plates for different vehicle models, different purposes, etc., and they are divided into a training set and a test set according to a specified ratio. Secondly, a network model is constructed. Using a conventional deep convolutional network to construct an encoder and a decoder, the existing diffusion model is improved by adding a forward diffusion adjustment process to adaptively adjust the forward diffusion time. During the reverse sampling process, by comparing the gradient changes in a specified time period, the optimal diffusion moment is selected for sampling, and the sampling result is input into the decoder to decode the image data. The method adopted by the present invention transfers the data to the latent space, which can make the training of the model more expressive, improve the sampling speed of the diffusion model, and generate higher-quality image samples.

[0144] Embodiment 2

[0145] As Figure 3 shown, a license plate image generation system based on an adaptive diffusion prior variational autoencoder in this embodiment includes the following modules:

[0146] Dataset production module: Collect and sort the original license plate images captured by the camera, and further divide them into a training set and a test set;

[0147] Encoding network module: Input the training set data into the encoder network to encode the image data and sample the posterior distribution obtained by encoding to obtain latent variable data;

[0148] Latent variable diffusion module: Diffuse the latent variable data according to the rules of the diffusion model to obtain the noise-added latent variable data;

[0149] Forward diffusion time adjustment module: Adaptively adjust the diffusion step size for the next iteration according to the KL divergence between the distribution at time τ and the distribution at time 1;

[0150] Decoding network module: After sampling the posterior distribution obtained by the encoding network, input it into the decoding network together with the target character domain and target character class to decode the reconstructed image data;

[0151] Training module: End-to-end train the network model according to the total loss function of the model;

[0152] Optimal training weight retention module: Conduct tests and retain the optimal training weights;

[0153] License plate image generation module: In this module, select the reverse diffusion moment according to the reverse diffusion adjustment process, and input the latent variable sampled reversely into the decoder for decoding to obtain the license plate image.

[0154] In the dataset creation module, various license plate images are collected and sorted, including license plates of different models and different uses, and then the license plate image dataset D is sorted out and divided into training set D and training set D according to the ratio of 5:1. train and the test set D test .

[0155] In the encoding network module, before the data is input into the module, the training set and the test set need to be preprocessed, including normalization, standardization, image size transformation, and image enhancement (such as random cropping, color jittering, horizontal flipping, etc.). The data enhancement method can adjust or change parameters according to the specific license plate image data.

[0156] The common encoding network is implemented by deep convolution module. Through the encoding network, the data space is transferred to the latent space. The training data of the input encoder network includes image samples X = {x1, x2, ..., x N}, where x i For the i-th sample in the current batch, i = 1, 2, ... B, the output distribution of the encoder network is recorded as q φ (z|x), which is the posterior distribution of the latent variable before noise is added.

[0157] In the latent variable diffusion module, stochastic differential equations and forward diffusion regulation modules are needed to realize the forward diffusion of latent variables.

[0158] Forward diffusion process in continuous time variable t∈[0,1] In the diffusion model, it is assumed that the diffusion does not start from time 0, but from a certain time τ, that is, z τ is the starting variable, z t is the disturbance at time t. The purpose of the diffusion process is to transform z τ Gradually mapped to a multidimensional normal distribution, that is:

[0159]

[0160] Where T is the total number of diffusion steps. In order to simplify the diffusion model, q(z t ∣z t-1 ) is set to the sum constant β t The Gaussian distribution I is the unit matrix. To get the hidden variables from any moment to the specified moment, let α t :=1-β t , You can get:

[0161]

[0162] In the forward diffusion time adjustment module, using the same step size for all samples in a batch of data samples may cause samples that could converge faster to converge more slowly. Since the forward diffusion process of each image is independent, different diffusion step sizes are adopted for different samples. By using the KL divergence to measure the distance between the distribution at time τ and the distribution at time 1, the KL divergence at time τ of batch b is denoted as KL b , and the KL divergence at time τ corresponding to batch b+1 is denoted as KL b+1 :

[0163]

[0164]

[0165] Among them, represents the distribution at time τ corresponding to batch b, represents the distribution at time τ corresponding to batch b+1.

[0166] When KL b < KL b+1 (or KL b > KL b+1 ), the corresponding diffusion time update formula is:

[0167] τ b+1 = τ b + α·Δτ

[0168] In the above formula, α represents the diffusion time update step size, Δτ represents the diffusion time change amount. In the change range of τ b ∈(0,1), Δτ is usually taken as 0.01 to update the time of τ b+1 , and the calculation method of α is as follows:

[0169]

[0170] In the above formula, λ represents the direction control bit for forward (or backward) diffusion of the diffusion time, and β represents the change speed of α with KL b+1 .

[0171] In the decoding network module, the data sampled from the posterior distribution output by the encoding network, the target character domain, and the target character class are input into the decoder network for decoding. The decoded data distribution is denoted as p θ (x∣z).

[0172] In the model training module, the score of the diffusion latent variable can be obtained according to the score network:

[0173]

[0174] Among them, s represents the scoring network, θ is the parameter of this network, and the scoring network is generally parameterized by the Unet model.

[0175] Using the reconstruction loss and the KL divergence between the prior distribution and the posterior distribution, the total loss of this model can be obtained:

[0176]

[0177] Among them, the third term in the above formula is the cross-entropy between the posterior distribution and the prior distribution. In the framework of the diffusion model, the following can be obtained:

[0178]

[0179] Among them, in the above formula, D is a constant.

[0180] According to the L(x, φ, θ, ψ) formula, the variational autoencoder model based on the adaptive diffusion prior can be trained, and the model parameters are iteratively updated. Every 20 iterations, an image generation test is performed. The test process is as follows:

[0181] During the image generation process, an inverse sampling adjustment process is added. Through this process, a prior distribution at a suitable τ moment is selected. According to this prior distribution, sampling is performed and input into the decoder for decoding to generate an image.

[0182] During the inverse sampling process, sampling is performed according to the following formula:

[0183]

[0184] Among them, is the standard Wiener process, dt is an infinitesimal negative time step, is given by the scoring network.

[0185] According to the above sampling formula, starting from the last update moment in the forward diffusion process, sampling is performed on the latent variable at the τ moment:

[0186] z τ = f(z1, τ) - g 2 (τ)s θ (z1, τ) + g(τ)ò

[0187] Among them, the distribution of z1 is: p(z1) = N(z1; 0, I), and ò is Gaussian noise.

[0188] According to the gradient change during the τ ± Δτ period in the inverse sampling process, the inverse diffusion step size is dynamically adjusted. Define the latent variable sampling at the τ - Δτ moment as:

[0189] z τ-Δτ = f(zτ , τ - Δτ) - g 2 (τ - Δτ)s θ (z τ , τ - Δτ) + g(τ - Δτ)ò

[0190] Meanwhile, the latent variable at time τ + Δτ can be obtained according to the forward diffusion process:

[0191] z τ+Δτ = f(z τ , τ + Δτ) + g(z τ , τ + Δτ)ò

[0192] Thus, the change amount near τ ± Δτ can be calculated, that is, the first-order difference component and the second-order difference component:

[0193] z′ τ = (z τ+Δτ - z τ ) / Δτ

[0194] z″ τ = (z τ+Δτ - 2z τ + z τΔτ) / Δτ 2

[0195] From this, the mixing tolerance can be calculated, and then the diffusion time can be updated:

[0196] δ(z′ τ , z′ τ(prev) ) = max(∈ abs , ∈ rel max(|z′ τ |, |z′ τ(prev) |))

[0197] where z′ τ(prev) represents the z′ obtained from the previous loop calculation τ , and from this, the local error amount can be calculated:

[0198]

[0199] The above formula calculates the L2 norm, and thus the following diffusion time update strategy can be given:

[0200]

[0201] where θ is the safety factor. In the present invention, r adopts the measured value, and r ∈ [0.5, 1]. Through the above method, an optimal prior distribution can be selected and input into the decoder to decode the reconstructed image.

[0202] In the optimal training weight retention module, after the complete model network result is constructed, the initial learning rate used by the diffusion model is set to 2×10 -4 and it is trained using the stochastic gradient descent optimizer, and the initial learning rate used by the variational autoencoder is set to 1×10 -4 and it is trained using the stochastic gradient descent optimizer, and the total number of diffusion steps used is 1000. During the training process, every 20 iterations of the training dataset, a random selection is made from the test dataset for testing, and common image quality evaluation metrics are selected for evaluation. During the testing process, the optimal model weights can be saved.

[0203] In this embodiment, through modular design, the usability and flexibility of the model are maximally ensured.

[0204] Compared with the prior art, the license plate image generation method and system based on the adaptive diffusion prior variational autoencoder of the present invention effectively solve the problems existing in the current license plate recognition model, such as poor image quality of manually collected images, uneven image types, and poor robustness of the license plate recognition model, etc. The present invention proposes a license plate image generation method and system based on the adaptive diffusion prior variational autoencoder. By constructing an encoding network and a decoding network, and introducing a diffusion model, an adaptive prior distribution is obtained and improved, the forward diffusion process is adaptively adjusted, and during the reverse sampling process, by comparing the gradients at specified time periods, the optimal reverse sampling moment is selected and input into the decoder for decoding to obtain the final generated image. The present invention adopts a modular design, which maximally ensures the usability and flexibility of the model.

[0205] The above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described here. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A license plate image generation method based on an adaptive diffusion prior variational autoencoder, characterized in that It includes the following steps: S1. Collect and organize the original license plate image dataset; S2. Construct a variational autoencoder network with an adaptive diffusion prior, and determine the character classes and character domains used to construct the generated license plate images; construct and output the generated license plate images based on the image features, character classes, and character domains extracted by the variational autoencoder network; S3. Input several batches of set training data into the encoding network of the variational autoencoder to obtain the posterior distribution, where the sample license plate images fed into the encoding network contain the original character domains and original character classes; S4. Sample data from the obtained posterior distribution and input it into the forward process of the diffusion model. According to the forward sampling adjustment process, adaptively adjust the diffusion time so that the sampled data undergoes adaptive diffusion within the range of τ~1 to obtain the noise-added latent variables, 0 < τ < 1; S5. Input the noise-added latent variables into the score network to obtain the scores of the latent variables at different diffusion times, that is, the gradients after taking the logarithm of the noise-added latent variable distribution; S6. Determine the target character domains and target character classes, and send the target character domains, target character classes, and the sampled data of the posterior distribution into the decoder for decoding to obtain the image data reconstructed by the variational autoencoder. The reconstructed image data contains the target character domains and target character classes; S7. Use the output sampled data of the encoder and the reconstructed image data of the decoder to obtain the reconstruction loss; use the mean square error of the gradient after taking the logarithm of the noise-added latent variable distribution and the output of the score network to obtain the KL divergence distance loss between the prior distribution and the posterior distribution, and update and train the parameters of the variational autoencoder and the diffusion model and test the generated images; S8. Save the optimal model weights in the test, and use the trained model to generate image data.

2. The license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 1, wherein In step S1, collect and organize the license plate image dataset and divide it into a training set according to a ratio and a test set 3. A license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 2, characterized in that, In step S2, the parameters in the constructed variational autoencoder network include the network input image size C×H×W, where C = 3, H = 136, W = 36, the batch size is B, and the latent variable dimension is D latent , the encoder network parameters are φ, the decoder network parameters are ψ, the diffusion model network parameters are θ, and the latent variable is Z.

4. A license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 3, characterized in that, In step S3, several batches of set training data are input into the encoding network of the variational autoencoder, where the training data input into the encoder includes image samples X = {x1, x2, …, x N}, where x i is the i-th sample in the current batch, and i = 1, 2, …, B.

5. A license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 4, characterized in that, In step S3, the posterior distribution of the latent variable before adding noise is q φ (z|x).

6. A license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 5, characterized in that Step S4 specifically includes: S41: Define a forward diffusion process for the continuous-time variable t ∈ [0, 1] Suppose the diffusion process starts to diffuse at time τ, τ ≠ 0, i.e., z τ is the starting variable, z t is the perturbation at time t, and the purpose of the diffusion process is to gradually map z τ to a multi-dimensional normal distribution through a Markov chain, i.e., where T is the total number of diffusion steps, q(z t |z t-1 ) is defined as a Gaussian distribution related to the constant β t ; I is the identity matrix; let α :=1 - β t , t and we get: ​ The above formula perturbs the latent variable at time τ to obtain the noise-added latent variable at any time within the range of 0~1; S42: Forward diffusion adjustment process Use different diffusion step sizes for different samples. By using the KL divergence to measure the distance between the distribution at time τ and the distribution at time 1, denote the KL divergence at time τ of batch b as KL b , and denote the KL divergence at time τ corresponding to batch b + 1 as KL b+1 : Among them, represents the distribution corresponding to the τ moment of batch b, represents the distribution corresponding to the τ moment of batch b + 1; When K L b <K L b+1 or K L b >K L b+1 The corresponding diffusion time update formula is: τ b+1 = τ b + α·Δτ In the above formula, α represents the update step size of the diffusion time, and Δτ represents the change amount of the diffusion time. In the range of τ b ∈(0,1), Δτ is taken as 0.01, and τ b+1 is updated at the moment. The calculation method of α is as follows: In the above formula, λ represents the direction control bit for forward or backward diffusion of the diffusion time, and β represents the change rate of α with respect to KL. b+1 ​ 7. A license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 6, characterized in that In step S5, obtain the score of the diffusion latent variable: Among them, s represents the score network, θ is the parameter of this network, and the score network is parameterized by the Unet model.

8. A license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 7, characterized in that In step S6, the target character field and the target character class are determined, and the sampled data, the target character field, and the target character class are input into the decoder network for decoding. The decoded data distribution is denoted as p θ (x|z), and the reconstructed image data includes the target character field and the target character class.

9. A license plate image generation method based on an adaptive diffusion prior variational autoencoder according to claim 8, characterized in that, In step S7, use the reconstruction loss and the KL divergence between the prior distribution and the posterior distribution to obtain the total loss of this model: Among them, the third term in the above formula is the cross-entropy term between the posterior distribution and the prior distribution. In the framework of the diffusion model, it is as follows: Among them, D is a constant; According to the formula, train a variational autoencoder model based on an adaptive diffusion prior and iteratively update the model parameters. For each specified number of iterations, perform an image generation test. The test process is as follows: During the generation process, add a reverse sampling adjustment process. Through this process, select a suitable prior distribution at time τ, and according to this prior, perform sampling and input it into the decoder for decoding to generate images; During the reverse sampling process, sample according to the following formula: where, is a standard Wiener process, dt is an infinitesimal negative time step, is given by the scoring network; According to the above sampling formula, start from the last update time in the forward diffusion process and sample the latent variable at time τ: z τ = f(z1, τ) - g 2 (τ)s θ (z1, τ) + g(τ) ∈ Among them, the distribution of z1 is: p(z1) = N(z1; 0, I), ∈ is Gaussian noise; According to the gradient change in the τ±Δτ period during the reverse sampling process, dynamically adjust the reverse diffusion step size, and define the sampling of the latent variable at time τ-Δτ as: z τ-Δτ = f(z τ , τ - Δτ) - g 2 (τ - Δτ)s θ (z τ , τ - Δτ) + g(τ - Δτ) ∈ At the same time, the latent variable at time τ+Δτ is obtained according to the forward diffusion process: z τ+Δτ = f(z τ , τ + Δτ) + g(z τ , τ + Δτ) ∈ Thus, the change amount near τ±Δτ is calculated, namely the first-order difference component and the second-order difference component: z′ τ = (z τ+Δτ - z τ ) / Δτ z τ ″=(z τ+Δτ - 2z τ + z τ-Δτ ) / Δτ 2 From this, the mixed tolerance is calculated, and then the time update is carried out: δ(z′ τ ,z′ τ(prev) ) = max(∈ abs , ∈ rel max(|z′ τ |, |Z′ τ(pev) |)) where z′ τ(prev) represents the z′ obtained from the previous loop calculation τ , and the local error amount is calculated therefrom: The L2 norm is calculated by the above formula, and the following diffusion time update strategy is given accordingly: Δτ opt = θ·Δτ·(E2) -r Where θ is the safety factor, r adopts the measured value, and r ∈ [0.5, 1].

10. A license plate image generation system based on an adaptive diffusion prior variational autoencoder for performing the method as described in claim 1, characterized in that It includes the following modules: Dataset production module: Collect and organize the original license plate images captured by the camera; Encoding network module: Input the training set data into the encoder network, encode the license plate image data, and sample the posterior distribution encoded to obtain the latent variable data; Latent variable diffusion module: Diffuse the latent variable data according to the rules of the diffusion model to obtain the noise-added latent variable data; Forward diffusion time adjustment module: According to the KL divergence between the distribution at τ time and the distribution at 1 time, adaptively adjust the diffusion step size for the next iteration; Decoding network module: After sampling the posterior distribution obtained by the encoding network, together with the target character domain and the target character class, input them into the decoding network for decoding to obtain the reconstructed image data; Training module: Carry out end-to-end training on the network model according to the total loss function of the model; Optimal training weight retention module: Conduct tests and retain the optimal training weights; License plate image generation module: Select the reverse diffusion time according to the reverse diffusion adjustment process, and send the latent variable sampled reversely into the decoder for decoding to obtain the license plate image.

Images