License Plate Tampering Data Generation Method Based on Perspective Transformation and Multi-Fusion Post-Processing Network

Abstract

The present invention discloses a method for generating license plate tampering data based on perspective transformation and a multi-fusion post-processing network. The method of the present invention inputs randomly selected vehicle pictures into a pre-trained license plate target detection model, obtains the license plate area based on the returned license plate vertex coordinates, designs a license plate replacement strategy based on image perspective transformation to achieve license plate replacement under different perspectives and different environments, designs and trains a multi-fusion post-processing network on this basis to ensure the compatibility between the replaced license plate and the background picture, enhance the authenticity of the forged data, and finally batch generate tampered license plates and their annotation data to form a license plate tampering dataset. The present invention realizes a method for expanding a license plate tampering image dataset with high efficiency and high authenticity without manual tampering.

Description

Technical Field

[0001] The invention belongs to the technical field of image data generation and computer model, and relates to a method for generating license plate image tampering data based on perspective transformation and multiple fusion post-processing networks. Background Art

[0002] Digital image tampering refers to the process of manipulating, modifying or forging digital images using image editing software such as Photoshop and Meitu Xiuxiu. Most traditional methods rely on experienced manual identification or use manually designed directions to generate tampered areas such as features such as histogram of gradients (HOG). These methods are often not only inefficient but also have high identification error rates, and the results are not robust and accurate enough. At present, with the development of deep learning, deep learning-based methods have dominated many computer vision tasks. It uses large-scale training data sets and powerful representation extraction capabilities to obtain robust algorithm models. However, there are still many problems in the research of tampered license plate recognition algorithms: 1) There is no public standard data set, which makes it difficult to train and obtain effective models for promotion and application; 2) The differences in license plates under different scenes, weather and angles make it difficult to guarantee the performance of the detection algorithm after data processing such as noise interference and contrast adjustment of the original image; 3) The time and money cost of manual annotation of large-scale license plate tampering data sets is very expensive.

[0003] Tampered license plate images may be used to conduct adversarial attacks on driving record videos, vehicle insurance policy checks, and surveillance cameras. If they cannot be accurately detected or identified, they may bring great security and economic risks to society and the country. Existing research on tampered data detection lacks research on license plate tampering. Simulating common tampering methods to actively generate adversarial samples for license plate tampering detection algorithms helps to actively and efficiently generate a large number of high-fidelity tampering samples, forming a large-scale data set that can be used for license plate tampering detection, and then helping to train a robust data tampering detection model to prevent problems before they occur. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating license plate tampering data based on perspective transformation and multiple fusion post-processing networks in response to the deficiencies and shortcomings of current license plate counterfeiting and tampering detection.

[0005] The object of the present invention is achieved through the following technical solution: a method for generating license plate tampering data based on perspective transformation and multiple fusion post-processing network, the method comprising the following steps:

[0006] (1) Obtain the license plate dataset for traversal and retrieval, initialize the pre-trained license plate object detection model and load the pre-trained weights; take two license plate images in each group obtained by traversal and input them into the license plate object detection model in combination. The input of the license plate object detection model is a three-channel two-dimensional RGB image, and the output is the coordinate vector of the four vertices of the license plate corresponding to each vehicle image in the group;

[0007] (2) Establish a perspective transformation matrix based on the coordinates of the license plate vertices detected in step (1). Each group of two images is used as the source vehicle image and the target vehicle image respectively. Calculate and solve the perspective transformation matrix between the two according to the coordinates of the four vertices in their respective images, and map the original vehicle images of the two to the same two-dimensional coordinate system;

[0008] (3) According to the perspective transformation matrix established in step (2), crop and replace the transformed license plate rectangular area in the source vehicle image with the transformed license plate rectangular area in the target vehicle image to obtain an intermediate image, and then perform an inverse transformation on the two vehicle images by solving the inverse matrix of the perspective transformation matrix to restore them to the original image angle, obtaining the preliminary tampered license plate data;

[0009] (4) Input the preliminary tampered license plate data obtained in step (3) into the designed multi-fusion post-processing network, and calculate the gain values of the replaced license plate pixels and background pixels in four indicators: color balance, saturation, resolution, and blur. The calculation formula is as follows:

[0010] Gain = w c (c) + w s (s) + w r g(r) + w b g(b) (1)

[0011] Where w represents the weight of each indicator, c represents color balance, s represents saturation, r represents resolution, b represents blur, g(·) represents the gain calculation module of each indicator, and Gain represents the total gain value;

[0012] (5) Modify the pasted license plate pixel information using the total gain value obtained in step (4). The calculation formula is as follows:

[0013]

[0014] Where Img represents the image of the new / old license plate area, represents pixel-level filtering, use the total gain value as conditional information to guide the randomly initialized convolution kernel to process and optimize the old license plate area to generate a new license plate area image, and ω represents the hyperparameter of the convolution kernel;

[0015] (6) Input the entire vehicle image obtained in step (5) into the tampered image discriminator, and output the tampering probability of the image. Minimize this probability value as the optimization objective to guide the adjustment of the gain calculation module for each index and the parameters in the convolution kernel. Through multiple rounds of iteration, train the multi-fusion post-processing network to make the generated image tend to be realistic until it successfully deceives the discriminator. The loss function loss of the multi-fusion post-processing network is as follows:

[0016]

[0017] where Img i represents the i-th tampered entire vehicle image, Discriminator represents the tampered image discriminator, and N is the number of training images. represents the multi-fusion post-processing network;

[0018] (7) Using the training method in step (6), send a large number of tampered vehicle images into the multi-fusion post-processing network in groups of N, and batch-automatically generate a preset number of images to achieve the generation of a large-scale license plate tampering dataset.

[0019] Furthermore, in step (1), if the license plate of the vehicle image used for license plate tampering data generation is incomplete, or the output of the vehicle image input into the license plate target detection model does not meet the data requirements, discard the vehicle image.

[0020] Furthermore, in step (1), the vehicle image is required to be a three-channel RGB image. If not, channel conversion is required.

[0021] Furthermore, in step (2), the source vehicle image and the target vehicle image are processed as follows:

[0022] (2.1) The four vertex coordinates of the license plate are set as 1, 2, 3, and 4 in clockwise order starting from the upper left vertex. The upper left vertex coordinate of the license plate before perspective transformation is expressed as [x s1 , y s1 , z s1 , and the upper left vertex coordinate of the license plate after perspective transformation is expressed as [x t1 , y t1 , z t1 . Its mapping relationship is shown in formula (4), and the perspective transformation matrix A is shown in formula (5):

[0023]

[0024]

[0025] where represents image linear transformation, represents image perspective transformation, T3 = [a 31 a 32 represents image translation, and [x t1 , y t1 , z t1 is expressed as shown in formula (6). Since the license plate change is performed on a two-dimensional picture, z t = 1;

[0026]

[0027] (2.2) The coordinate system where the matrix after perspective transformation is located is a two-dimensional coordinate system. The upper left vertex coordinate is on the y-axis, x t1 = 0, the lower left vertex coordinate is the origin of the coordinate system, and [x t4 , y t4 = [0, 0]. The lower right vertex coordinate is on the x-axis, y t3 = 0, and then the mapping relationships of the remaining three vertices are obtained.

[0028] Furthermore, in step (3), the perspective transformation matrices of the source vehicle picture and the target vehicle picture are respectively solved using the license plate vertex coordinates of the source vehicle picture and the target vehicle picture. Thus, the source vehicle picture and the target vehicle picture are converted to intermediate pictures at the same angle, the license plate rectangular area is cropped for replacement, and finally the inverse matrix of the perspective transformation matrix is solved to restore the angles of the source vehicle picture and the target vehicle picture. The calculation formula is as follows:

[0029]

[0030] Furthermore, in step (4), the replaced vehicle picture and the vertex coordinates of the license plate area are input into the gain calculation modules of different metrics of the multi-fusion post-processing network, and the pixel-level gain values of the license plate area and the background area under different metrics are respectively calculated. The processing methods of color balance, saturation, and resolution are the same. Using the gain value as conditional information, the randomly initialized convolution module processes the pixels of the license plate area to achieve equalization, and through multiple iterations, the replaced license plate and the background tend to be consistent in each metric. The calculation formula is as follows:

[0031]

[0032] Among them, g(i) respectively represents the gain values of color balance c, saturation s, and resolution r. P back-j represents the j-th pixel point in the background area, with a total of B, and P car-j represents the j-th pixel point in the license plate area, with a total of C. Conv represents the convolution module of the picture, which is used to extract the feature information of each metric within the area, and ω represents the hyperparameter of the convolution kernel;

[0033] The gain calculation module for the blur degree b performs Gaussian blur smoothing on a certain point on the side length of the rectangular area replacing the license plate and the surrounding pixel points. The calculation formula is as follows:

[0034]

[0035] Among them, (x, y) is the point coordinate, r represents the radius of the convolution kernel, s represents the convolution function, f(u, v) is the Gaussian filtering function of the convolution kernel size, and the filtering function is affected by the loss function. Through multiple iterations, the picture of the license plate area to be replaced and the surrounding pixel connection area tend to be smooth and not overly abrupt.

[0036] Furthermore, in step (6), taking the tampering probability value returned by the minimized discriminator as the optimization goal, designing a loss function to optimize the gain calculation module and the convolution module of each index, using Adam as the optimizer, inputting 20 pictures for training in each epoch, training for 10 epochs, and then replacing a new set of 20 pictures until the set goal of the number of tampered pictures is completed.

[0037] The present invention also provides a license plate tampering data generation device based on perspective transformation and multi-fusion post-processing network, including a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the above-mentioned license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network.

[0038] The present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the above-mentioned license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network.

[0039] The beneficial effects of the present invention are as follows: The license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network fills the blank of no large-scale tampering data set in the field of license plate tampering detection, making it possible to train deep learning-based methods. It enhances the security performance of the license plate tampering detection model based on deep learning and can effectively prevent security and economic losses. Brief Description of the Drawings

[0040] Figure 1 is the flowchart of the implementation of the method of the present invention.

[0041] Figure 2 is an example of the implementation of the method of the present invention.

[0042] Figure 3 is the structural diagram of the device of the present invention. Detailed Embodiments

[0043] The present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0044] A license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network provided by an embodiment of the present invention is as follows Figure 1 , 2 shown, including the following steps:

[0045] (1) Obtain a large-scale open-source Chinese license plate dataset for traversal and retrieval, initialize a pre-trained license plate target detection model based on Yolo-v5 and load the pre-trained weights; input two license plate images in each group into the license plate target detection model. The input of the license plate target detection model is a three-channel two-dimensional RGB image, and the output is the coordinate vector of the four vertices of the license plate corresponding to each vehicle image in the group.

[0046] Among them, large-scale open-source Chinese license plate datasets can use open-source datasets such as CCPD and CRPD. If the license plate of the vehicle image used for license plate tampering data generation is incomplete, or the output of the vehicle image input into the license plate target detection model does not meet the data requirements of 2*4, discard the vehicle image. The vehicle image is required to be a three-channel RGB image. If not, channel conversion is required.

[0047] (2) Establish a perspective transformation matrix according to the coordinates of the vertices of each group of license plates detected in step (1). Each group of two images is used as the source vehicle image and the target vehicle image respectively. Calculate and solve the perspective transformation matrix of the two according to the coordinates of the four vertices in their respective images, and map the original vehicle images of the two to the same two-dimensional coordinate system; specifically, the source vehicle image and the target vehicle image are processed as follows respectively:

[0048] (2.1) The four vertex coordinates of the license plate are set as 1, 2, 3, and 4 in a clockwise order starting from the upper left vertex. The coordinates of the upper left vertex of the license plate before perspective transformation are expressed as [x s1 , y s1 , z s1 , and the coordinates of the upper left vertex of the license plate after perspective transformation are expressed as [x t1 , y t1 , z t1 . Its mapping relationship is shown in formula (1), and the perspective transformation matrix A is shown in formula (2):

[0049]

[0050]

[0051] Where represents image linear transformation, represents image perspective transformation, T 3 =[a 31 a 32represents image translation, [x t1 , y t1 , z t1 is expressed as shown in formula (3). Since the license plate change is performed on a two-dimensional picture, z t = 1;

[0052]

[0053] (2.2) The coordinate system where the matrix after perspective transformation is located is a two-dimensional coordinate system. Among them, the upper left vertex coordinate is on the y-axis, x t1 = 0, and the lower left vertex coordinate is the origin of the coordinate system. It can be obtained that [x t4 , y t4 = [0, 0]. The lower right vertex coordinate is on the x-axis, y t3 = 0, and then the mapping relationships of the remaining three vertices are obtained.

[0054] (3) According to the perspective transformation matrix established in step (2), the transformed license plate rectangular area in the source vehicle picture and the transformed license plate rectangular area in the target vehicle picture are cropped and replaced using the opencv library function to obtain an intermediate picture. Then, the two vehicle pictures are inversely transformed by solving the inverse matrix of the perspective transformation matrix to restore to the original picture angle, obtaining the preliminary tampered license plate data. Specifically, the perspective transformation matrices of the source vehicle picture and the target vehicle picture are solved respectively using the license plate vertex coordinates of the source vehicle picture and the target vehicle picture. Thus, the source vehicle picture and the target vehicle picture are converted to an intermediate picture at the same angle, and the license plate rectangular area is cropped to achieve replacement. Finally, the inverse matrix of the perspective transformation matrix is solved to achieve the angle restoration of the source vehicle picture and the target vehicle picture. The calculation formula is as follows:

[0055]

[0056] (4) The preliminary tampered license plate data obtained in step (3) is input into the designed multi-fusion post-processing network. For specific scenarios of different vehicles, it is necessary to perform balanced adjustment on the replaced license plate area and background area. For example, if a license plate is photographed clearly, but the pasted background is blurred, then this tampered data is easily detected by the human eye and does not have training value. That is, post-processing technology needs to be added according to the different situations of the license plate and the background.

[0057] First, the gain values of the replaced license plate pixels and background pixels are calculated on four indicators: color balance degree, saturation, resolution, and blur degree. The calculation formula is as follows:

[0058] Gain = w c g(c) + w s g(s) + w r g(r) + w b g(b) (5)

[0059] Where \(w\) represents the weight of each index, \(c\) represents the color balance degree, \(s\) represents the saturation, \(r\) represents the resolution, \(b\) represents the blur degree, \(g(\cdot)\) represents the gain calculation module of each index, and Gain represents the total gain value;

[0060] Then, the replaced vehicle image and the vertex coordinates of the license plate area are input into the gain calculation modules of different indexes of the proposed multi-fusion post-processing network to calculate the pixel-level gain values of the license plate area and the background area under different indexes respectively. The processing methods of color balance degree, saturation and resolution are the same. Using the gain value as conditional information, the randomly initialized convolution module is used to process the pixels of the license plate area to achieve equalization. Through multiple iterations, the replaced license plate and the background tend to be consistent in each index. The calculation formula is as follows:

[0061]

[0062] Where \(g(i)\) represents the gain values of color balance degree \(c\), saturation \(s\) and resolution \(r\) respectively, \(P\) back-j represents the \(j\)-th pixel point in the background area, with a total of \(B\) pixels, and \(P\) car-j represents the \(j\)-th pixel point in the license plate area, with a total of \(C\) pixels. Conv represents the convolution module of the image, which consists of two-dimensional convolution (convolution kernel size \(k = 3\), stride is 1), ReLU activation function and pooling layer, and is used to extract the feature information of each index in the area. \(\omega\) represents the hyperparameter of the convolution kernel;

[0063] The gain calculation module of the blur degree \(b\) performs Gaussian blur smoothing processing on a certain point on the side length of the replaced license plate rectangular area and the surrounding pixel points. The calculation formula is as follows:

[0064]

[0065] Where \((x,y)\) is the point coordinate, \(r\) represents the radius of the convolution kernel, \(s\) represents the convolution function, and \(f(u,v)\) is the Gaussian filtering function of the convolution kernel size. The filtering function is affected by the loss function. Through multiple iterations, the image of the replaced license plate area and the surrounding pixel connection area tend to be smooth and not too abrupt.

[0066] (5) Modify the pixel information of the pasted license plate using the total gain value obtained in step (4). The calculation formula is as follows:

[0067]

[0068] Where Img represents the image of the new / old license plate area, It represents pixel-level filtering. The total gain value is used as conditional information to guide the randomly initialized convolution kernel to process and optimize the old license plate area to generate a new license plate area picture. ω represents the hyperparameter of the convolution kernel.

[0069] (6) Input the entire vehicle picture obtained in step (5) into the tampered picture discriminator, and output the tampering probability of the picture. Minimize this probability value as the optimization goal to guide the adjustment of the gain calculation module of each index and the parameters in the convolution kernel. Through multiple rounds of iteration, train the multi-fusion post-processing network to make the generated picture tend to be realistic until it successfully deceives the discriminator. The loss function loss of the multi-fusion post-processing network is as follows:

[0070]

[0071] Among them, Img i represents the i-th tampered entire vehicle picture, Discriminator represents the tampered picture discriminator, N is the number of training pictures, represents the multi-fusion post-processing network;

[0072] Specifically, the tampered picture discriminator uses a classic tampered picture or deepfake detector, such as: "[1] Zhou, Peng, et al. "Learning rich features for image manipulation detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "[2] Chen, Xinru, et al. "Image manipulation detection by multi-view multi-scale supervision." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021.", etc. Minimize the tampering probability value returned by the discriminator as the optimization goal, design the loss function to optimize the gain calculation module and convolution module of each index, and the optimizer uses Adam. Input 20 pictures for training in each epoch, train for 10 epochs, and then replace with a new set of 20 pictures until the set goal of the number of tampered pictures is completed.

[0073] (7) Using the training method in step (6), send a large number of tampered vehicle pictures into the multi-fusion post-processing network in groups of N, and batch-automatically generate a preset number of pictures to achieve the generation of a large-scale license plate tampering dataset.

[0074] To further verify the authenticity of the generated tampered data, we input the 100,000 pieces of data generated by this method into four classic or latest tampering detection methods for testing, as shown in Table 1.

[0075] Table 1 Detection Results of Tampering Detection Models

[0076]

[0077] See Figure 3 , a license plate tampered data generation device based on perspective transformation and multi-fusion post-processing network provided by an embodiment of the present invention includes a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it is used to implement the license plate tampered data generation method based on perspective transformation and multi-fusion post-processing network in the above embodiment.

[0078] The embodiment of the license plate tampered data generation device based on perspective transformation and multi-fusion post-processing network of the present invention can be applied to any device with data processing capabilities. The any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by the processor of any device with data processing capabilities reading the corresponding computer program instructions in the non-volatile memory into the memory for operation. From the hardware level, as Figure 3 shown, it is a hardware structure diagram of any device with data processing capabilities where the license plate tampered data generation device based on perspective transformation and multi-fusion post-processing network of the present invention is located. In addition to Figure 3 the shown processor, memory, network interface, and non-volatile memory, any device with data processing capabilities where the device in the embodiment is located usually also includes other hardware according to the actual functions of the any device with data processing capabilities, which will not be elaborated here.

[0079] The specific implementation process of the functions and roles of each unit in the above device can be specifically seen in the implementation process of the corresponding steps in the above method, which will not be elaborated here.

[0080] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0081] The embodiments of the present invention also provide a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the method for generating license plate tampering data based on perspective transformation and multi-fusion post-processing network in the above embodiments is implemented.

[0082] The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium can also include both the internal storage unit and the external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store the data that has been output or will be output.

[0083] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

[0084] The specific embodiments described above have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating license plate tampering data based on perspective transformation and multi-fusion post-processing network, specifically including the following steps: (1) Obtain a license plate dataset for traversal and retrieval, initialize a pre-trained license plate target detection model and load the pre-trained weights; For each group of two license plate images obtained by traversal, input them into the license plate target detection model. The input of the license plate target detection model is a three-channel two-dimensional RGB image, and the output is the coordinate vector of the four vertices of the license plate corresponding to each vehicle image in the group; (2) Establish a perspective transformation matrix according to the coordinates of the license plate vertices detected in step (1). Each group of two images is used as the source vehicle image and the target vehicle image respectively. Calculate and solve the perspective transformation matrix between the two according to the coordinates of the four vertices in their respective images, and map the original vehicle images of the two to the same two-dimensional coordinate system; (3) According to the perspective transformation matrix established in step (2), crop and replace the transformed license plate rectangular area in the source vehicle image with the transformed license plate rectangular area in the target vehicle image to obtain an intermediate image. Then, perform an inverse transformation on the two vehicle images by solving the inverse matrix of the perspective transformation matrix to restore them to the original image angle, and obtain the preliminary tampered license plate data; (4) Input the preliminary tampered license plate data obtained in step (3) into the designed multi-fusion post-processing network, and calculate the gain values of the replaced license plate pixels and background pixels in four indicators: color balance degree, saturation, resolution, and blur degree. The calculation formula is as follows: Gain=w c g(c)+w s g(s)+w r g(r)+w b g(b) (1) where w represents the weight of each indicator, c represents the color balance degree, s represents the saturation, r represents the resolution, b represents the blur degree, g(·) represents the gain calculation module of each indicator, and Gain represents the total gain value; (5) Modify the pasted license plate pixel information using the total gain value obtained in step (4). The calculation formula is as follows: where Img represents the image of the new / old license plate area, represents pixel-level filtering, using the total gain value as conditional information to guide the randomly initialized convolutional kernel to process and optimize the old license plate area to generate a new license plate area image, and ω represents the hyperparameter of the convolutional kernel; (6) Input the entire vehicle image obtained in step (5) into the tampered image discriminator, and output the tampering probability of the image. Minimize this probability value as the optimization goal to guide the adjustment of the gain calculation module of each indicator and the parameters in the convolutional kernel. Through multiple rounds of iteration, train the multi-fusion post-processing network to make the generated image tend to be realistic until it successfully deceives the discriminator. The loss function loss of the multi-fusion post-processing network is as follows: Among them, Img i represents the entire tampered vehicle image of the i-th one, Discriminator represents the tampered image discriminator, N is the number of training images, represents the multi-fusion post-processing network; (7) Using the training method in step (6), send a large number of tampered vehicle images into the multi-fusion post-processing network in groups of N, and batch-automatically generate a preset number of images to realize the generation of a large-scale license plate tampering dataset.

2. A method for generating license plate tampering data based on perspective transformation and multi-fusion post-processing network according to claim 1, characterized in that, in step (1), if the license plate of the vehicle image used for generating license plate tampering data is incomplete, or the output of the vehicle image input into the license plate target detection model does not meet the data requirements, discard the vehicle image.

3. A method for generating license plate tampering data based on perspective transformation and multi-fusion post-processing network according to claim 1, characterized in that, In step (1), the vehicle picture is required to be a three-channel RGB image. If not, channel conversion is needed.

4. A license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network according to claim 1, characterized in that, in step (2), the source vehicle picture and the target vehicle picture are processed as follows: (2.1) The four vertex coordinates of the license plate are set as 1, 2, 3, and 4 in clockwise order starting from the upper left vertex. The coordinates of the upper left vertex of the license plate before perspective transformation are expressed as [x s1 , y s1 , z s1 , and the coordinates of the upper left vertex of the license plate after perspective transformation are expressed as [x t1 , y t1 , z t1 . Their mapping relationship is shown in formula (4), and the perspective transformation matrix A is shown in formula (5): Among them represents an image linear transformation, represents an image perspective transformation, T 3 = [a 31 a 32 represents an image translation, [x t1 , y t1 , z t1 is shown in formula (6), where the license plate change is performed on a two-dimensional picture, so z t = 1; (2.2) The coordinate system where the matrix after perspective transformation is located is a two-dimensional coordinate system, where the upper left vertex coordinate is on the y-axis and x t1 = 0, and the lower left vertex coordinate is the zero point of the coordinate system. It can be obtained that [x t4 , y t4 = [0, 0]. The lower right vertex coordinate is on the x-axis and y t3 = 0, and then the mapping relationships of the remaining three vertices are obtained.

5. A license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network according to claim 4, characterized in that, in step (3), the perspective transformation matrices are respectively solved using the license plate vertex coordinates of the source vehicle picture and the target vehicle picture, thereby converting the source vehicle picture and the target vehicle picture into intermediate pictures at the same angle, cropping the license plate rectangular area for replacement, and finally solving the inverse matrix of the perspective transformation matrix to restore the angles of the source vehicle picture and the target vehicle picture. The calculation formula is as follows:

6. A license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network according to claim 1, characterized in that, in step (4), the replaced vehicle picture and the license plate area vertex coordinates are input into the gain calculation modules of different metrics of the multi-fusion post-processing network, and the pixel-level gain values of the license plate area and the background area under different metrics are calculated respectively. Among them, the processing methods of color balance, saturation, and resolution are the same. Using the gain value as conditional information, the randomly initialized convolution module processes the license plate area pixels to achieve equalization. Through multiple iterations, the replaced license plate and the background tend to be consistent in each metric. The calculation formula is as follows: where g(i) represents the gain values of color balance c, saturation s, and resolution r respectively, and P back-j represents the j-th pixel point in the background area, with a total of B, and P car-j represents the j-th pixel point in the license plate area, with a total of C. Conv represents the convolutional module of the image, which is used to extract the feature information of each index in the area, and ω represents the hyperparameter of the convolutional kernel; The gain calculation module of the blur degree b performs Gaussian blur smoothing processing on a certain point on the side length of the replaced license plate rectangular area and the surrounding pixel points. The calculation formula is as follows: where (x, y) is the point coordinate, r represents the radius of the convolution kernel, s represents the convolution function, f(u, v) is the Gaussian filtering function of the convolution kernel size, and the filtering function is affected by the loss function. Through multiple iterations, the picture of the replaced license plate area is made to be smooth and not too abrupt when connected to the surrounding pixels.

7. A license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network according to claim 1, characterized in that, in step (6), taking the tampering probability value returned by the minimized discriminator as the optimization target, designing a loss function to optimize the gain calculation modules and convolution modules of each metric. The optimizer uses Adam. 20 pictures are input for training in each epoch, and training is carried out for 10 epochs, and then a new set of 20 pictures is replaced until the set target number of tampered pictures is completed.

8. A license plate tampering data generation device based on perspective transformation and multi-fusion post-processing network, including a memory and one or more processors, and executable code is stored in the memory, characterized in that, when the processor executes the executable code, it is used to implement the license plate tampering data generation method according to any one of claims 1-7.

9. A computer-readable storage medium, on which a program is stored, characterized in that, When executed by a processor, it is used to implement the license plate tampering data generation method based on perspective transformation and multi-fusion post-processing network as described in any one of claims 1-7.

Images