Image generation method and vehicle

Abstract

The application discloses an image generation method and a vehicle, and relates to the field of vehicle identification. The method comprises the following steps: acquiring a first original image set, wherein the first original image set comprises at least a plurality of first original images and at least one first mask image; generating a first mask label based on the image shape of a first target icon and preset transformation parameters; filling the first mask image based on the first mask label to generate a first icon image; and processing the first original image, the first mask image and the first icon image by using an image generation model to generate a target image set. The application solves the technical problem that the type of traffic signboards contained in the data set used for training a traffic sign recognition model is limited, and the recognition accuracy of the traffic sign recognition model is low.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of traffic sign recognition, in particular to an image generation method and a vehicle. BACKGROUND

[0002] At present, in the field of traffic sign recognition, the data set used for training the traffic sign recognition model is mostly real scene images collected by image collection devices. However, due to the characteristics of small target, complex background and various types of traffic signboards, they are easily affected by different image collection devices, light, weather and other factors, resulting in limited clarity of the images collected by the image collection devices. Moreover, there are many types of traffic signs, and the types of images collected by the image collection devices are limited, and the images of rare categories with fewer instances cannot be collected, resulting in low recognition accuracy of the traffic sign recognition model.

[0003] At present, no effective solution has been proposed for the above problems. SUMMARY

[0004] The embodiments of the present application provide an image generation method and a vehicle to at least solve the technical problem that the types of traffic signboards contained in the data set used for training the traffic sign recognition model are limited, resulting in low recognition accuracy of the traffic sign recognition model.

[0005] According to an aspect of an embodiment of the present application, an image generation method is provided, comprising: obtaining a first original image set, wherein the first original image set at least includes: a plurality of first original images, and at least one first mask image, the first original image is used to represent the image of the real scene photographed, and the first mask image is used to represent the image of the first target icon corresponding to the traffic signboard of the first category; generating a first mask label based on the image shape of the first target icon and the preset transformation parameter, wherein the preset transformation parameter at least includes: a size parameter, a position parameter and a deformation parameter; filling the first mask image based on the first mask label to generate a first icon image; processing the first original image, the first mask image and the first icon image by using an image generation model to generate a target image set, wherein the target image set at least includes: a plurality of target images, and the target image is used to represent the image of the real scene containing the traffic signboard of the first category.

[0006] Optionally, generating the first mask label based on the image shape of the first target icon and the preset transformation parameter comprises: adjusting the image shape based on the deformation parameter to obtain an adjusted icon; generating a background image based on the size parameter; performing fusion processing on the adjusted icon and the background image based on the position parameter to obtain a first label image; and generating the first mask label based on each pixel in the first label image.

[0007] Optionally, the first mask label is generated based on each pixel in the first label image, including: in response to a first pixel in the first label image belonging to the adjustment icon, determining that a label corresponding to the first pixel in the first mask label is a first preset value; and in response to a second pixel in the first label image not belonging to the adjustment icon, determining that a label corresponding to the second pixel in the first mask label is a second preset value.

[0008] Optionally, the first icon image is generated by filling the first mask image based on the first mask label, including: performing deformation processing on a first target icon in the first mask image based on the deformation parameter to obtain a first deformation image; determining a color of each pixel in the first deformation image based on the first mask label; and filling the first deformation image based on the position parameter and the color of each pixel to obtain the first icon image.

[0009] Optionally, the color of each pixel in the first deformation image is determined based on the first mask label, including: in response to a first pixel in the first deformation image belonging to the first target icon, determining that the color of the first pixel is a first preset color; and in response to a second pixel in the first deformation image not belonging to the first target icon, determining that the color of the second pixel is a second preset color, wherein a contrast between the second preset color and the first preset color is greater than a first preset threshold.

[0010] Optionally, the target image set is generated by processing the first original image, the first mask image and the first icon image using the image generation model, including: adjusting the first original image based on the size parameter to obtain a target original image; performing splicing processing on the target original image and the first icon image based on a background region of the first icon image, wherein the background region is used to represent a region in which pixels not belonging to the first target icon in the first icon image are located; and performing sampling restoration processing on the first splicing image based on the first deformation image to obtain the target image set, wherein the first deformation image is used to represent an image obtained by adjusting the first target icon in the first mask image based on the deformation parameter.

[0011] Optionally, the method further comprises: obtaining a second original image set, wherein the second original image set at least comprises: at least one second original image, a second mask label corresponding to the original image, and at least one second mask image, the original image is used to represent an image of a real scene containing a traffic sign of a second category, the second mask label is used to represent whether a pixel in the original image belongs to the traffic sign, and the second mask image is used to represent an image containing a second target icon corresponding to the traffic sign of the second category; performing padding on a background part in the second mask image to generate a second icon image, wherein the background part is used to represent other pixels except target pixels corresponding to the second target icon in the second mask image; and training the initial generation model based on the second original image, the second mask label and the second icon image to obtain the image generation model.

[0012] Optionally, the training of the initial generation model based on the second original image, the second mask label and the second icon image to obtain the image generation model comprises: performing affine transformation on the second target icon based on the second icon image and the second mask label to obtain a first virtual image; performing sampling and splicing processing on the first virtual image and the second original image to obtain a second spliced image; and performing adversarial training on the initial generation model based on the second spliced image and the second original image to obtain the image generation model.

[0013] Optionally, the adversarial training of the initial generation model based on the second spliced image and the second original image to obtain the image generation model comprises: obtaining an adversarial loss value of the second spliced image and the second original image, and a category loss value corresponding to the second original image, wherein the adversarial loss value is used to represent a similarity between the second spliced image and the second original image, and the category loss value is used to represent an error between a classification label obtained by classifying the second original image by using the initial generation model and a preset label corresponding to the second original image; in response to the adversarial loss value and the category loss value not satisfying a preset condition, repeatedly performing sampling and splicing processing on the second spliced image, wherein the preset condition is that the adversarial loss value is less than a second preset threshold, the category loss value is less than a third preset threshold, and the adversarial loss value and the category loss value do not change within a preset training number of times; and in response to the adversarial loss value and the category loss value satisfying the preset condition, determining that the image generation model is successfully trained.

[0014] According to another aspect of the embodiments of the present application, a vehicle is also provided, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above methods.

[0015] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which includes a stored program, wherein the program, when executed, controls the device where the computer readable storage medium is located to perform the image generation method.

[0016] According to another aspect of the embodiments of the present application, a processor is also provided, which is configured to execute a program, wherein the program, when executed, performs the image generation method.

[0017] In the embodiments of the present application, after the first original image set is obtained, the first mask label can be generated based on the image shape of the first target icon and the preset transformation parameter, the first mask image is further filled based on the first mask label to generate the first icon image, and finally the image generation model is used to process the first original image, the first mask image and the first icon image to generate the target image set. It is easy to note that the first original image in the first original image set is an image of a real scene shot, and the first mask image contains the first target icon corresponding to the first category of traffic signs, so that the finally generated target image set contains not only the first target icon corresponding to the first category of traffic signs, but also the real scene, which is equivalent to that the target image set is an image of a real scene shot, and the purpose of data enhancement for the data set used to train the traffic sign recognition model is achieved, thereby realizing the technical effect of generating rare category data with scene diversity and improving the recognition accuracy of the traffic sign recognition model, and further solving the technical problem that the type of traffic signs contained in the data set used to train the traffic sign recognition model is limited, resulting in low recognition accuracy of the traffic sign recognition model. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0019] Figure 1 is a flowchart of an optional image generation method according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of an optional TT100K data set instance number according to an embodiment of the present application;

[0021] Figure 3 is a comparison schematic diagram of original images and label images of the pl20 category in an optional TT100K data set according to the present embodiment;

[0022] Figure 4 is an optional training data schematic diagram according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of an optional conditional generation network architecture according to an embodiment of the present application;

[0024] Figure 6 is a specific step flow chart of an optional image generation method according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of an optional image generation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product, or device.

[0028] First, the technical terms appearing in the present application are explained as follows:

[0029] TT100K dataset: The dataset has a total of 221 categories, including 3000 traffic signs, with an image size of 2048*2048, and available labels including category labels, mask labels, and bounding box labels for target detection. Among all 221 categories of image data, there are 73 categories with instance numbers in the [0, 10] interval, accounting for about 1 / 3 of the total categories. Figure 2 is a schematic diagram of an optional TT100K dataset instance number according to an embodiment of the present application, which counts the instance number of each category in the TT100K dataset with more than 100 instances, such as Figure 2As shown, in the categories with more than 100 instances, most of the categories have fewer instances, which cannot meet the requirement of the deep learning algorithm that the training set data distribution is extensive, and affects the overall recognition effect of the deep model on all categories. In addition to the original image of the traffic sign in the data set, an independent traffic sign mark icon corresponding to each category is also provided, and the image size is 280x280, and the part other than the icon is a transparent background.

[0030] JavaScript Object Notation (JSON) file: The JSON file is a file used to store simple data structures and objects, and can be used for data exchange in web applications. It is a method of storing information in an organized and easily accessible manner.

[0031] Spatial Transformer Network (STN): STN can keep the recognition effect of the picture unchanged after various transformations, and is composed of a convolutional network and has spatial invariance.

[0032] Residual neural network: The residual neural network belongs to the convolutional neural network, and its characteristics are easy to optimize and can improve the accuracy by increasing the depth. The residual block inside uses a jump connection to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0033] Embodiment 1

[0034] According to the embodiment of the present application, a method for generating an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0035] Figure 1 is a flowchart of an optional image generation method according to an embodiment of the present application, as shown, the method comprises the following steps: Figure 1

[0036] Step S102, acquiring a first original image set, wherein the first original image set at least includes: a plurality of first original images, and at least one first mask image, the first original image is used to represent the image of the real scene photographed, and the first mask image is used to represent the image of the first target icon corresponding to the traffic signboard of the first category.

[0037] ​The first category can refer to warning signs, prohibition signs, indication signs, and road signs, and the first target icon refers to an icon on the traffic signboard of the first category. The first original image can be an image obtained by photographing a real scene of vehicle driving through an image acquisition device such as a camera, a camera, or a video camera. The image can or can not contain a traffic signboard. The first mask image can be an image containing only the first target icon in the traffic signboard, and the background before the first target icon is transparent.

[0038] In an optional embodiment, the first original image set can be obtained by a vehicle-mounted camera or other image acquisition device. In another optional embodiment, the first original image set can be obtained from the Internet.

[0039] In step S104, the first mask label is generated based on the image shape of the first target icon and the preset transformation parameter. The preset transformation parameter at least includes a size parameter, a position parameter, and a deformation parameter.

[0040] The size parameter, the position parameter, and the deformation parameter are all adjustable parameters, and the user can customize the corresponding first mask label according to the need. The size parameter can include parameters such as width, length, height, and angle, which can be used to adjust the size of the image. The position parameter can be a coordinate point (x, y), which can be used to represent a specific position on the image and can be used to determine the display position of the first target icon. The deformation parameter can include parameters such as width, length, height, and angle, which can be used to adjust the image shape.

[0041] The above generation of the first mask label based on the image shape of the first target icon and the preset transformation parameter includes: adjusting the image shape based on the deformation parameter to obtain an adjusted icon; generating a background image based on the size parameter; fusing the adjusted icon and the background image based on the position parameter to obtain a first label image; and generating a first mask label based on each pixel in the first label image, wherein the first mask label refers to the value of each pixel in the first label image. Figure 3 An optional comparison diagram of original images and label images of the pl20 category in a TT100K data set according to the embodiment is shown. The data set also includes categories such as wo, w46, and pa10. As shown in Figure 3 The left image is the original image, and the right image is the label image. The label image is composed of 0 and 1, and is often visualized as a black and white image to depict the specific boundary of the traffic signboard. The white part in the figure is the position of 1, representing the image foreground; the black part is the position of 0, representing the image background.

[0042] Due to different angles of shooting, the shape of the image collected by the image collection device will also change accordingly. For example, the shape of the original image is a circle, and according to different angles of shooting, the actual shape of the shooting will become an ellipse; the shape of the original image is an equilateral triangle, and according to different angles of shooting, the actual shape of the shooting will become an irregular triangle. Therefore, in order to enhance the diversity of the scene, the shape of the first target icon can be adjusted to make the shape of the image more consistent with the shape of the image shot in the real scene. For example, the shape of the first target icon is a regular circle or triangle, and in order to be more consistent with the shape of the image shot in the real scene, the image can be stretched into an oval or an irregular triangle. In an optional embodiment, after the first target icon is obtained, the image shape of the target icon can be adjusted based on the deformation parameter, so that the adjusted icon is more consistent with the shape of the icon corresponding to the traffic sign shot in the real scene.

[0043] Since the size of the viewfinder of the camera when shooting an image is fixed, in order to make the size of the generated background image closer to the size of the image shot in the real scene, the background image can be set as a rectangular box with a fixed size, for example, if the size of the image shot in the real scene is 2048*2048, the background image is set as a 2048*2048 rectangular box. In an optional embodiment, a background image with a corresponding size can be generated according to the size parameter of the camera viewfinder.

[0044] In an optional embodiment, the adjusted icons can be placed on the background image according to the preset position parameter, and merged together to achieve the purpose of fusion processing of the adjusted icons and the background image, so as to obtain the first label image.

[0045] The first label image refers to a black and white binary image with the same size as the original image after the JSON file in the original data set is parsed, and the white color is the foreground image, that is, the area where the traffic sign is located, and the black color is the background image, that is, other areas except the target icon. Since the first label image is annotated from the real image, it is irregular in shape and not uniform in size due to the influence of the shooting distance and the shooting angle.

[0046] In an optional embodiment, each pixel point in the first label image can be annotated to obtain a first mask label. The specific annotation process is as follows: in response to a first pixel in the first label image belonging to the adjusted icon, it is determined that the label corresponding to the first pixel in the first mask label is a first preset value; in response to a second pixel in the first label image not belonging to the adjusted icon, it is determined that the label corresponding to the second pixel in the first mask label is a second preset value.

[0047] The first preset value can be 1, and the second preset value can be 0. In order to distinguish the foreground image and the background image in the first label image obtained after the fusion processing, each pixel point in the foreground image and the background image can be assigned a value, for example, all the pixel points in the foreground image are assigned 1, and all the pixel points in the background image are assigned 0.

[0048] In step S106, the first mask image is filled based on the first mask label to generate a first icon image.

[0049] The first mask image is filled based on the first mask label to generate the first icon image, including: performing deformation processing on the first target icon in the first mask image based on a deformation parameter to obtain a first deformation image; determining the color of each pixel in the first deformation image based on the first mask label; and filling the first deformation image based on the position parameter and the color of each pixel to obtain the first icon image, wherein the first icon image refers to an image generated by the first mask label and the first mask image, and the size of the image is consistent with that of the first original image.

[0050] In an optional embodiment, affine transformation can be performed on the first target icon, that is, the first target icon is transformed, stretched and scaled to form a first deformation image consistent with the position, shape and size of the foreground image, so that the first target icon can be consistent with the position, shape and size of the foreground image when the first deformation image is filled, and the corresponding foreground image is easily extracted. Further, the affine transformation module STN can be used to perform affine transformation on the first target icon.

[0051] The above determining the color of each pixel in the first deformation image based on the first mask label includes: in response to a first pixel in the first deformation image belonging to the first target icon, determining the color of the first pixel as a first preset color; and in response to a second pixel in the first deformation image not belonging to the first target icon, determining the color of the second pixel as a second preset color, wherein the contrast between the second preset color and the first preset color is greater than a first preset threshold.

[0052] The first pixel can refer to a pixel point in the foreground image, and the second pixel can refer to a pixel point in the background image. The second preset color and the first preset color can be any color with a contrast greater than the first preset threshold, which is user-defined. The first preset threshold can be a value with a relatively large contrast, such as 3000:1 or 4000:1. The larger the contrast, the more accurate the image recognition. For example, the first preset color is white, and the second preset color is black, that is, the foreground image region is determined as white, and the background image region is determined as black.

[0053] In an optional embodiment, the specific position of the first mask image in the background image can be determined according to the shape and center point of the first label image, and then the first deformation image is filled according to the color of each pixel, that is, the foreground image is filled with white and the background image is filled with black. Figure 4 is an optional training data diagram according to an embodiment of the present application, as shown in FIG. 5, from left to right, in order are a mask image, an icon image, a label image, and an original image. Figure 4 Further, the first deformation image can be filled by using an Open Source Computer Vision Library (OpenCV).

[0054] In step S108, the first original image, the first mask image, and the first icon image are processed by using the image generation model to generate a target image set, wherein the target image set at least includes a plurality of target images, and the target images are used to represent images of real scenes containing traffic signs of the first category.

[0055] The above processing of the first original image, the first mask image, and the first icon image by using the image generation model to generate the target image set includes: adjusting the first original image based on a size parameter to obtain a target original image; performing splicing processing on the target original image and the first icon image based on a background region of the first icon image to obtain a first splicing image, wherein the background region is used to represent a region in which pixels not belonging to the first target icon in the first icon image are located; and performing sampling restoration processing on the first splicing image based on the first deformation image to obtain the target image set, wherein the first deformation image is used to represent an image obtained by adjusting the first target icon in the first mask image based on a deformation parameter.

[0056] In an optional embodiment, the size of the first original image can be adjusted to be the same as the size of the first icon image according to the size parameter, so as to reduce the number of subsequent adjustments and improve the image generation efficiency. The target original image is taken as a background to replace the black area in the first icon image, and the final image obtained is the first spliced image. The specific splicing formula is: i = icon fake * mask + x * (1-mask), wherein i is the input image, icon fake is the first spliced image, mask is the first icon image, and x is the first original image. The first spliced image is first down-sampled and then up-sampled to restore, for example, the input image is first down-sampled to a feature map with a size of 128*128, and then up-sampled to restore to a size of 2048*2048. In this way, the problem of connection disorder caused by the position of the target original image and the first icon image not matching in the previous splicing process can be processed. Wherein, down-sampling refers to generating a thumbnail of the corresponding image, which can be used to process large images and reduce the amount of calculation, and up-sampling refers to enlarging the image, which can display the image at a higher resolution.

[0057] In another optional embodiment, the first spliced image can be processed by a residual neural network for sampling and restoring.

[0058] The method of the embodiment further includes the following training method of the image generation model: obtaining a second original image set, wherein the second original image set at least includes at least one second original image, a second mask label corresponding to the original image, and at least one second mask image, the original image is used to represent an image of a real scene containing a traffic sign of a second category, the second mask label is used to represent whether a pixel in the original image belongs to the traffic sign, and the second mask image is used to represent an image of a second target icon corresponding to the traffic sign of the second category; the background part in the second mask image is filled to generate a second icon image, wherein the background part is used to represent other pixels except the target pixels corresponding to the second target icon in the second mask image; and the initial generation model is trained based on the second original image, the second mask label and the second icon image to obtain the image generation model.

[0059] Figure 5 is an optional conditional generation network architecture diagram according to an embodiment of the application, as Figure 5As shown, STN is an affine transformation module, G is a generator module, D is a discriminator module, a real image and a mask label are combined to obtain an icon image, the icon image is subjected to affine transformation by STN to obtain a virtual image, the virtual image, the mask label and the real image are subjected to splicing processing to obtain an initial fake image, the initial fake image is subjected to the generator G to obtain a synthetic image, the synthetic image and the real image are subjected to the discriminator D, and a generative adversarial loss and a category loss are calculated. In the training process of the image generation model, through iterative optimization of the loss, the neural network composed of G and STN gradually has the ability to generate realistic images, and can be used in a data augmentation task after the training is completed to generate target images of any size, position, shape and background. The data augmentation refers to a technique of expanding new training samples by using an existing data set, which can increase data diversity and improve the performance of an algorithm at a low cost from the perspective of data.

[0060] The second category can refer to traffic sign types such as warning signs, prohibition signs, indication signs and direction signs, and the second target icon refers to an icon on a traffic sign of the second category. The second icon image refers to an image with a size consistent with the second original image after the background part in the second mask image is filled.

[0061] In an optional embodiment, the second original image set can be obtained by using an image acquisition device such as a camera, a lens or a video camera.

[0062] In an optional embodiment, the background part in the second mask image can be filled by using OpenCV.

[0063] The above training of the initial generation model based on the second original image, the second mask label and the second icon image to obtain the image generation model includes: performing affine transformation on the second target icon based on the second icon image and the second mask label to obtain a first virtual image; performing sampling splicing processing on the first virtual image and the second original image to obtain a second splicing image; and performing adversarial training on the initial generation model based on the second splicing image and the second original image to obtain the image generation model.

[0064] The first virtual image refers to an image consistent with the position, shape and size of the foreground image after affine transformation is performed on the second target icon based on the second icon image and the second mask label.

[0065] In an optional embodiment, affine transformation can be performed on the second target icon by using an affine transformation module STN. The second original image is used as a background to replace the black area in the first virtual image, and the final obtained image is the second splicing image.

[0066] The initial generation model is adversarially trained based on the second spliced image and the second original image to obtain the image generation model, including: obtaining an adversarial loss value of the second spliced image and the second original image, and a category loss value corresponding to the second original image, wherein the adversarial loss value is used to represent the similarity of the second spliced image and the second original image, and the category loss value is used to represent the error of a classification label obtained by classifying the second original image by the initial generation model and a preset label corresponding to the second original image, wherein the classification label refers to the category of the traffic sign, such as a warning sign, a prohibition sign, an indication sign, and a guide sign, and the preset label refers to the category of the traffic sign to be obtained; in response to the adversarial loss value and the category loss value not satisfying a preset condition, repeatedly performing sampling and splicing processing on the second spliced image, wherein the preset condition is that the adversarial loss value is less than a second preset threshold, the category loss value is less than a third preset threshold, and the adversarial loss value and the category loss value do not change within a preset training number of times; and in response to the adversarial loss value and the category loss value satisfying the preset condition, determining that the image generation model is successfully trained.

[0067] The adversarial loss value and the category loss value can be obtained by iteratively calculating the second spliced image and the second original image multiple times.

[0068] In an optional embodiment, the adversarial loss value of the second spliced image and the second original image and the category loss value corresponding to the second original image can be calculated by a residual neural network using a generative adversarial network (GAN) loss function and a cross-entropy loss function.

[0069] The second preset threshold can refer to the minimum similarity when the second spliced image and the second original image are of the same category, the third preset threshold can refer to the minimum error of the classification label obtained by classifying the second original image and the preset label corresponding to the second original image, and the preset training number of times can be set by the user, for example, the preset training number of times is 20 times.

[0070] In an optional embodiment, when the adversarial loss value is less than the second preset threshold, the category loss value is less than the third preset threshold, and the adversarial loss value and the category loss value do not change within the preset training number of times, it can be considered that the image generation model is successfully trained.

[0071] With the updating of the neural network parameters in the training process, a neural network capable of generating high-quality synthetic images according to the second target icon, the second mask label and the second original image can be trained. At this time, the network parameters are fixed, and the input of the network is changed to the category of the target image to be synthesized, the position of the mask label and the original image obtained in the above steps, and the synthetic image of any category, any position and any background we want can be obtained.

[0072] Figure 6 is a specific step flow chart of an optional image generation method according to an embodiment of the application, as shown in the figure, the specific steps are: obtaining training data, training a conditional generation network model, obtaining input data for generation, and outputting generated data. Figure 6

[0073] Through the above steps, after obtaining the first original image set, the first mask label can be generated based on the image shape of the first target icon and the preset transformation parameter, the first mask image is further filled based on the first mask label, the first icon image is generated, and finally the image generation model is used to process the first original image, the first mask image and the first icon image to generate the target image set. It is easy to note that the first original image in the first original image set is an image of a real scene taken, and the first mask image contains the first target icon corresponding to the traffic sign of the first category, so that the finally generated target image set contains not only the first target icon corresponding to the traffic sign of the first category, but also the real scene, which is equivalent to that the target image set is an image of a real scene taken, achieving the purpose of data augmentation of the data set for training the traffic sign recognition model, thereby realizing the technical effect of generating rare category data with scene diversity and improving the recognition accuracy of the traffic sign recognition model, and further solving the technical problem that the type of traffic sign contained in the data set for training the traffic sign recognition model is limited, resulting in low recognition accuracy of the traffic sign recognition model.

[0074] Embodiment 2

[0075] According to another aspect of the embodiment of the application, an image generation device is also provided, which can execute the image generation method in the above-mentioned embodiment 1. The specific implementation scheme and application scenario in this embodiment are the same as those in the above-mentioned embodiment 1, and will not be repeated here.

[0076] Figure 7 is a schematic diagram of an optional image generation device according to an embodiment of the application, as shown in the figure, Figure 6 ​As shown, the apparatus comprises: an image acquisition module 702, configured to acquire a first original image set, wherein the first original image set comprises at least: a plurality of first original images, and at least one first mask image, the first original images being used to represent images of a photographed real scene, and the first mask image being used to represent an image of a first target icon containing a traffic sign of a first category; a label generation module 704, configured to generate a first mask label based on an image shape of the first target icon and a preset transformation parameter, wherein the preset transformation parameter comprises at least: a size parameter, a position parameter, and a deformation parameter; an image filling module 706, configured to fill the first mask image based on the first mask label to generate a first icon image; and an image generation module 708, configured to process the first original image, the first mask image, and the first icon image by using an image generation model to generate a target image set, wherein the target image set comprises at least: a plurality of target images, and the target images being used to represent images of the real scene containing the traffic sign of the first category.

[0077] The label generation module 704 comprises: a first adjustment unit, configured to adjust the image shape based on the deformation parameter to obtain an adjusted icon; a first generation unit, configured to generate a background image based on the size parameter; a fusion unit, configured to perform fusion processing on the adjusted icon and the background image based on the position parameter to obtain a first label image; and a second generation unit, configured to generate the first mask label based on each pixel in the first label image.

[0078] The second generation unit comprises: a first determination sub-unit, configured to determine, in response to a first pixel in the first label image belonging to the adjusted icon, that a label corresponding to the first pixel in the first mask label is a first preset value; and a second determination sub-unit, configured to determine, in response to a second pixel in the first label image not belonging to the adjusted icon, that a label corresponding to the second pixel in the first mask label is a second preset value.

[0079] The image filling module 706 comprises: a deformation unit, configured to perform deformation processing on the first target icon in the first mask image based on the deformation parameter to obtain a first deformed image; a first determination unit, configured to determine a color of each pixel in the first deformed image based on the first mask label; and a filling unit, configured to fill the first deformed image based on the position parameter and the color of each pixel to obtain the first icon image.

[0080] The first determination unit comprises: a third determination sub-unit, configured to determine, in response to a first pixel in the first deformed image belonging to the first target icon, that the color of the first pixel is a first preset color; and a fourth determination sub-unit, configured to determine, in response to a second pixel in the first deformed image not belonging to the first target icon, that the color of the second pixel is a second preset color, wherein a contrast between the second preset color and the first preset color is greater than a first preset threshold.

[0081] The image generation module 708 comprises: a second adjusting unit configured to adjust the first original image based on the size parameter to obtain a target original image; a first splicing unit configured to splice the target original image and the first icon image based on a background region of the first icon image to obtain a first spliced image, wherein the background region is used to represent a region in which pixels of the first target icon in the first icon image are located; and a restoring unit configured to sample and restore the first spliced image based on the first deformation image to obtain a target image set, wherein the first deformation image is used to represent an image obtained by adjusting the first target icon in the first mask image based on the deformation parameter.

[0082] The device further comprises: the image acquisition module 702 is further configured to acquire a second original image set, wherein the second original image set comprises at least one second original image, a second mask label corresponding to the original image, and at least one second mask image, the original image is used to represent an image of a real scene containing a traffic sign of a second category, the second mask label is used to represent whether a pixel in the original image belongs to the traffic sign, and the second mask image is used to represent an image of a second target icon corresponding to the traffic sign of the second category; the image filling module 706 is further configured to fill a background part in the second mask image to generate a second icon image, wherein the background part is used to represent other pixels except target pixels corresponding to the second target icon in the second mask image; and the model training module is configured to train the initial generation model based on the second original image, the second mask label, and the second icon image to obtain the image generation model.

[0083] The model training module comprises: a transformation unit configured to perform affine transformation on the second target icon based on the second icon image and the second mask label to obtain a first virtual image; a second splicing unit configured to sample and splice the first virtual image and the second original image to obtain a second spliced image; and a training unit configured to perform adversarial training on the initial generation model based on the second spliced image and the second original image to obtain the image generation model.

[0084] The training unit comprises: an acquisition subunit, configured to acquire an adversarial loss value of the second spliced image and the second original image, and a category loss value corresponding to the second original image, wherein the adversarial loss value is used to represent the similarity between the second spliced image and the second original image, and the category loss value is used to represent the error of a preset label corresponding to the second original image after the second original image is classified by using the initial generation model; a splicing subunit, configured to, in response to the adversarial loss value and the category loss value not satisfying a preset condition, repeatedly perform sampling and splicing processing on the second spliced image, wherein the preset condition is that the adversarial loss value is less than a second preset threshold, the category loss value is less than a third preset threshold, and the adversarial loss value and the category loss value do not change within a preset training number of times; and a fifth determination subunit, configured to, in response to the adversarial loss value and the category loss value satisfying the preset condition, determine that the image generation model training is successful.

[0085] Embodiment 3

[0086] According to another aspect of the embodiments of the present application, a vehicle is also provided, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the above.

[0087] Embodiment 4

[0088] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, comprising a stored program, wherein the program, when executed, controls the device where the computer readable storage medium is located to perform the image generation method.

[0089] Embodiment 5

[0090] According to another aspect of the embodiments of the present application, a processor is also provided, configured to execute a program, wherein the program, when executed, performs the image generation method.

[0091] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0092] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0093] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other means. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0094] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0095] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0096] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0097] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. An image generation method, characterized in that, include: Obtain a first set of original images, wherein the first set of original images includes at least: multiple first original images and at least one first mask image, wherein the first original images are used to represent images of real scenes captured, and the first mask image is used to represent images containing first target icons corresponding to traffic signs of the first category; Based on the image shape of the first target icon and preset transformation parameters, a first mask label is generated, wherein the preset transformation parameters include at least: size parameters, position parameters, and deformation parameters; The first icon image is generated by filling the first mask image with the first mask label; The first original image, the first mask image, and the first icon image are processed using an image generation model to generate a target image set, wherein the target image set includes at least: multiple target images, which are used to represent images of real scenes containing traffic signs of the first category; The process of generating a first mask label based on the image shape of the first target icon and preset transformation parameters includes: adjusting the image shape based on the transformation parameters to obtain an adjusted icon; generating a background image based on the size parameters; fusing the adjusted icon and the background image based on the position parameters to obtain a first label image; and generating the first mask label based on each pixel in the first label image.

2. The method according to claim 1, characterized in that, Generating the first mask label based on each pixel in the first label image includes: In response to the fact that the first pixel in the first label image belongs to the adjustment icon, the label corresponding to the first pixel in the first mask label is determined to be a first preset value; In response to the fact that the second pixel in the first label image does not belong to the adjustment icon, the label corresponding to the second pixel in the first mask label is determined to be the second preset value.

3. The method according to claim 1, characterized in that, The first icon image is generated by filling the first mask image based on the first mask label, including: The first target icon in the first mask image is deformed based on the deformation parameters to obtain the first deformed image; Based on the first mask label, determine the color of each pixel in the first deformed image; Based on the position parameters and the color of each pixel, the first deformed image is filled to obtain the first icon image.

4. The method according to claim 3, characterized in that, Based on the first mask label, the color of each pixel in the first deformed image is determined, including: In response to the fact that the first pixel in the first deformed image belongs to the first target icon, the color of the first pixel is determined to be a first preset color; In response to the fact that the second pixel in the first deformed image does not belong to the first target icon, the color of the second pixel is determined to be a second preset color, wherein the contrast between the second preset color and the first preset color is greater than a first preset threshold.

5. The method according to claim 1, characterized in that, The first original image, the first mask image, and the first icon image are processed using an image generation model to generate a target image set, including: The first original image is adjusted based on the size parameters to obtain the target original image; Based on the background area of ​​the first icon image, the target original image and the first icon image are stitched together to obtain a first stitched image, wherein the background area is used to represent the area where pixels in the first icon image that do not belong to the first target icon are located; The first stitched image is sampled and restored based on the first deformed image to obtain the target image set, wherein the first deformed image is used to represent the image obtained after adjusting the first target icon in the first mask image based on the deformation parameters.

6. The method according to claim 1, characterized in that, The method further includes: Obtain a second set of original images, wherein the second set of original images includes at least: at least one second original image, a second mask label corresponding to the original image, and at least one second mask image, wherein the original image is used to represent an image of a real scene containing traffic signs of the second category, the second mask label is used to represent whether a pixel in the original image belongs to the traffic sign, and the second mask image is used to represent an image containing a second target icon corresponding to a traffic sign of the second category; The background portion of the second mask image is filled to generate a second icon image, wherein the background portion is used to represent other pixels in the second mask image other than the target pixel corresponding to the second target icon; The initial generation model is trained based on the second original image, the second mask label, and the second icon image to obtain the image generation model.

7. The method according to claim 6, characterized in that, Based on the second original image, the second mask label, and the second icon image, the initial generation model is trained to obtain the image generation model, including: Based on the second icon image and the second mask label, an affine transformation is performed on the second target icon to obtain the first virtual image; The first virtual image and the second original image are sampled and stitched together to obtain the second stitched image; Based on the second stitched image and the second original image, the initial generation model is subjected to adversarial training to obtain the image generation model.

8. The method according to claim 7, characterized in that, Based on the second stitched image and the second original image, the initial generation model is subjected to adversarial training to obtain the image generation model, including: The adversarial loss values ​​of the second stitched image and the second original image, and the category loss value corresponding to the second original image are obtained. The adversarial loss value is used to characterize the similarity between the second stitched image and the second original image, and the category loss value is used to characterize the error between the classification label of the second original image after classification using the initial generation model and the preset label corresponding to the second original image. If the adversarial loss value and the class loss value do not meet the preset conditions, the sampling and stitching process is repeated on the second stitched image. The preset conditions are that the adversarial loss value is less than a second preset threshold, the class loss value is less than a third preset threshold, and the adversarial loss value and the class loss value have not changed within a preset number of training iterations. When the adversarial loss value and the class loss value satisfy the preset condition, the image generation model is determined to have been successfully trained.

9. A vehicle, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

Images