Image migration method and system for reducing domain differences
Through a generative adversarial network based on the convolution kernel, multiple auxiliary images of virtual images are used to generate migration images, which solves the problem of domain differences between virtual images and real images, and improves the accuracy and generalization ability of deep learning training.
Patent Information
- Application Number
- CN202210086618.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-25
- Publication Date
- 2025-01-17
- Estimated Expiration
- 2042-01-25
AI Technical Summary
The domain difference between virtual images and real images leads to limited generalization capabilities of deep learning algorithms in different scenarios and reduced accuracy, limiting the availability of virtual images.
Using a convolution kernel-based generative adversarial network, through alternating updates of generators and discriminators, migrating images are generated using multiple auxiliary maps of virtual images (color maps, depth maps, normal maps and semantic segmentation maps) to reduce virtual images The domain difference between an image and a real image.
The domain difference between the transferred image and the real image is significantly reduced, the authenticity of the transferred image and the efficiency of deep learning training are improved, and the generalization ability of the algorithm is enhanced.
Smart Images
Figure CN114429436B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an image migration method and system for reducing domain differences. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] With the continuous development and iteration of hardware such as graphics processing units (GPUs), modern graphics technology represented by real-time ray tracing technology has gradually become well-known to the public in recent years. NVIDIA's RTX series graphics cards and Unreal Engine 5 technical demonstrations have further increased the popularity of computer graphics. Virtual images generated by high-quality rendering are almost the same as real images. For example, realistic movie special effects are enough to make fakes look real. Advanced computer graphics technology has gradually entered the field of vision of artificial intelligence researchers. The purpose of realistic virtual images is to assist deep learning algorithms in training. The performance of algorithms trained by combining a large amount of virtual data with a small amount of real data as training sets can be close to or even exceed that of algorithms trained using only real data. Generating virtual data has the following advantages over collecting real data: high generation efficiency and low cost. With the support of high-performance graphics processors and high-quality real-time rendering algorithms, tens of thousands of virtual data with labeled information can be generated in one hour. Collecting the same amount of real data requires at least several weeks from finding a site, setting up collection equipment, collecting data to data labeling.
[0004] However, although virtual images have many advantages, there is a key problem: there is a domain gap between virtual images and real images. The training process of deep learning is highly dependent on data. Different tasks in different scenarios often require completely different training data, which also makes the generalization ability of deep learning very limited. As a result, when there is a certain domain gap between the training set and the test set, the accuracy will drop significantly. The problem of domain gap restricts the availability of virtual images. Summary of the invention
[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an image migration method and system for reducing domain differences, which reduces the domain difference between the generated migration image and the real image and improves the authenticity of the generated migration image.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A first aspect of the present invention provides an image migration method for reducing domain differences, comprising:
[0008] Get a virtual image;
[0009] Based on the virtual image, multiple auxiliary images of the virtual image are generated by using different shaders;
[0010] Based on multiple auxiliary images of the virtual image, a generator based on a convolution kernel-based generative adversarial network is used to obtain a migration image;
[0011] Among them, the generator of the generative adversarial network based on the convolution kernel uses a convolution layer to convolve multiple auxiliary maps respectively, adds the output of the convolution layer and inputs it into a convolution layer to obtain a fused feature map, and obtains a migration image based on the fused feature map.
[0012] Furthermore, the training method of the convolution kernel-based generative adversarial network is:
[0013] The generator of the convolution kernel-based generative adversarial network inputs multiple auxiliary images of virtual images in the training set and outputs a migration image;
[0014] Input the transfer image and the real image in the training set into the discriminator of the convolution kernel-based generative adversarial network;
[0015] The parameters of the generator and the discriminator are updated alternately.
[0016] Furthermore, the parameters of the generator and the discriminator are updated alternately using a gradient descent method.
[0017] Furthermore, the training set includes a real image and several virtual data, and each virtual data includes a virtual image and multiple auxiliary images thereof.
[0018] Furthermore, the auxiliary images include a color image, a depth image, a normal image and a semantic segmentation image.
[0019] Furthermore, the virtual image consists of a virtual scene and a three-dimensional object.
[0020] Furthermore, the shader is obtained through a shader editor of Blender.
[0021] A second aspect of the present invention provides an image migration system for reducing domain differences, comprising:
[0022] An image acquisition module is configured to: acquire a virtual image;
[0023] An auxiliary image generation module is configured to: generate multiple auxiliary images of the virtual image using different shaders based on the virtual image;
[0024] An image migration module is configured to obtain a migration image based on multiple auxiliary images of the virtual image and using a generator of a generative adversarial network based on a convolution kernel;
[0025] Among them, the generator of the generative adversarial network based on the convolution kernel uses a convolution layer to convolve multiple auxiliary maps respectively, adds the output of the convolution layer and inputs it into a convolution layer to obtain a fused feature map, and obtains a migration image based on the fused feature map.
[0026] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned image migration method for reducing domain differences.
[0027] A fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the image migration method for reducing domain differences as described above are implemented.
[0028] Compared with the prior art, the present invention has the following beneficial effects:
[0029] The present invention provides an image migration method for reducing domain differences. In response to different needs and tasks, a large number of virtual images can be quickly output in the form of virtual images, and a generative adversarial network based on convolution kernels is used to perform domain migration on the virtual images to reduce the domain differences between them and real images. Ultimately, the deep learning algorithm can be trained only with a large number of virtual images and make predictions on real images, which greatly reduces the domain differences between the migrated images and the real images, improves the feasibility of using the migrated images as deep learning training data, and further improves the implementation efficiency of deep learning in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0031] Figure 1 is a partial structural diagram of a generator according to the first embodiment of the present invention;
[0032] Figure 2 is a schematic diagram of a virtual image according to the first embodiment of the present invention;
[0033] Figure 3 is a schematic diagram of a color map, a depth map, a normal map, and a semantic segmentation map of a virtual image according to the first embodiment of the present invention;
[0034] Figure 4It is a schematic diagram of the migration result of the first embodiment of the present invention. DETAILED DESCRIPTION
[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0036] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0038] Embodiment 1
[0039] This embodiment provides an image migration method for reducing domain differences, using a rendering engine to generate a virtual image with a label, and using a generative adversarial network (CycleGAN) to further reduce the domain difference, specifically including the following steps:
[0040] Step 1: Get the virtual image built by the user.
[0041] Specifically, according to the needs and tasks, a reasonable virtual scene is built in the 3D graphics engine, and 3D objects are placed. The 3D scene and 3D objects form a virtual 3D scene. After placing a virtual camera, the virtual 3D scene is projected onto the display plane to generate a virtual 3D scene. Figure 2 The virtual image shown. The 3D graphics engine includes the rendering engine (renderer), 3D scene editing, 3D modeling, etc. For example, Blender, C4D and other software are 3D graphics engines.
[0042] Step 2: Based on the three-dimensional virtual scene, the renderer uses four different shaders designed by the user to synchronously generate multiple auxiliary maps (color map, depth map, normal map and semantic segmentation map) of the virtual image.
[0043] Specifically, four different shaders are designed, and the renderer generates the following based on the four different shaders: Figure 3 Color image, depth map, normal map and semantic segmentation map shown.
[0044] In specific applications, color images, depth images, normal images, and semantic segmentation images can be used for image semantic segmentation, target detection, and classification detection in addition to image migration tasks. At this time, corresponding labels need to be marked (for example, when doing image semantic segmentation tasks, the label of each pixel is the object category to which each pixel belongs, when doing target detection, the label is whether each pixel is a detection box, and when doing classification detection, the label is the category to which the image belongs). Four different shaders can be designed according to the required labels to generate the following: Figure 3 The color image, depth map, normal map, and semantic segmentation map and their labels are shown.
[0045] The shader is designed using Blender's shader editor; the renderer is the Cycles renderer.
[0046] Step 3: Based on the color map, depth map, normal map and semantic segmentation map of the virtual image, a generator based on a convolution kernel generative adversarial network is used to obtain a migration image. Specifically, Figure 4 As shown in the figure, the generator of the convolution kernel-based generative adversarial network uses four convolution layers (Conv) to convolve the color map, depth map, normal map and semantic segmentation map respectively, and then adds the outputs of the four convolution layers and inputs them into a convolution layer to obtain a fused feature map, and obtains a migration image (an image migrated to the target domain) based on the fused feature map.
[0047] Specifically, the input module of the generator in the generative adversarial network CycleGAN is modified as follows Figure 4 The convolution block Rconv shown in Figure 1 obtains a generative adversarial network based on the convolution kernel.
[0048] The training method of the convolution kernel-based generative adversarial network is:
[0049] Obtain a training set, which contains a real image and several virtual data. Each virtual data contains a virtual image (a source domain) and its color map, depth map, normal map, and semantic segmentation map.
[0050] The generator of the convolution kernel-based generative adversarial network inputs the color image, depth map, normal map and semantic segmentation map of the virtual image (source domain) in the training set, and outputs the migration image (the image migrated to the target domain); the migration image and the real image in the training set (data in the target domain) are input into the discriminator of the convolution kernel-based generative adversarial network; the parameters of the generator and the discriminator are updated alternately using the gradient descent method.
[0051] After training, only the generator is used to perform domain migration on the rendered image, migrating the virtual image to the real image, so as to reduce the domain difference between the generated migration image and the real image and improve the accuracy of the generated migration image.
[0052] Due to the introduction of RConv, the identity mapping loss function in the loss function of CycleGAN also needs to be modified accordingly. Let the color image of the source domain be x_r, the depth map x_d, the normal map x_n, the semantic segmentation map x_m, the generator be G, and the output of the generator be G(X), where X = [x_r, x_d, x_n, x_m], and the real image (data in the target domain) be y; the convolution kernel-based generative adversarial network is the same as CycleGAN, and also includes an auxiliary generator F, whose input is the data of the target domain and whose output is the data of the source domain. The purpose is to combine the identity mapping loss to keep the input content of the generator G consistent with the output content (only perform domain migration without changing the content). The present invention modifies the identity mapping loss in CycleGAN as follows:
[0053] L identity (G) = E x～pdata(x) ||F(G(X))–x_r||1
[0054] Among them, pdata is F(G(X)). The RConv block can make the most of the advantages of virtual data and use normal maps, depth maps, etc. as supplementary information to make the convolution kernel-based generative adversarial network generate results that are more consistent with real images. The virtual image of the building and the real image of the target domain are input into the convolution kernel-based generative adversarial network to perform domain migration on the virtual data. The results are shown in Figure 2. Figure 4 As shown, it can be seen that the CycleGAN generator ( Figure 4 The lower right image) is compared to the original CycleGAN generator ( Figure 4 The output has fewer ghost images and the content is closer to the original rendering. Using the migrated images as a dataset to train deep learning algorithms can improve the accuracy of the algorithm. Compared with using unmigrated virtual data, the migrated data can train deep learning more stably and obtain better generalization.
[0055] Embodiment 2
[0056] This embodiment provides an image migration system for reducing domain differences, which specifically includes the following modules:
[0057] An image acquisition module is configured to: acquire a virtual image;
[0058] An auxiliary image generation module is configured to: generate multiple auxiliary images of the virtual image using different shaders based on the virtual image;
[0059] An image migration module is configured to obtain a migration image based on multiple auxiliary images of the virtual image and using a generator of a generative adversarial network based on a convolution kernel;
[0060] Among them, the generator of the generative adversarial network based on the convolution kernel uses a convolution layer to convolve multiple auxiliary maps respectively, adds the output of the convolution layer and inputs it into a convolution layer to obtain a fused feature map, and obtains a migration image based on the fused feature map.
[0061] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.
[0062] Embodiment 3
[0063] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the image migration method for reducing domain differences as described in the first embodiment are implemented.
[0064] Embodiment 4
[0065] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the image migration method for reducing domain differences as described in the first embodiment are implemented.
[0066] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0067] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0070] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An image migration method for reducing domain differences, characterized in that: include: Get a virtual image; Based on the virtual image, different shaders are used to generate multiple auxiliary images of the virtual image; the auxiliary images include color images, depth images, normal images and semantic segmentation images, which are used for image migration, image semantic segmentation, target detection and classification detection tasks; when doing image semantic segmentation tasks, the label of each pixel is the object category to which each pixel belongs, when doing target detection, the label is marked as whether each pixel is a detection box, and when doing classification detection, the label is marked as the category to which the image belongs; Based on multiple auxiliary images of the virtual image, a generator based on a convolution kernel-based generative adversarial network is used to obtain a migration image; the input module of the generator in the generative adversarial network CycleGAN is modified to a convolution block Rconv to obtain a generative adversarial network based on a convolution kernel; the training method of the generative adversarial network based on a convolution kernel is as follows: the generator based on the convolution kernel-based generative adversarial network inputs multiple auxiliary images of the virtual image in the training set, and outputs a migration image; The training set includes a real image and several virtual data, each virtual data includes a virtual image and multiple auxiliary images thereof; the migration image and the real image in the training set are input into a discriminator of a generative adversarial network based on a convolution kernel; the parameters of the generator and the discriminator are updated alternately; After training, the generator is used to migrate the virtual image to the real image, reducing the domain difference between the generated migration image and the real image, and improving the accuracy of the generated migration image; Among them, the generator of the generative adversarial network based on the convolution kernel uses a convolution layer to convolve multiple auxiliary images respectively, adds the output of the convolution layer and inputs it into a convolution layer to obtain a fused feature map, and obtains a migration image based on the fused feature map; The convolution kernel-based generative adversarial network also includes an auxiliary generator F, whose input is a real image and output is a virtual image. The identity mapping loss is combined to keep the input content and output content of the generator G consistent; the identity mapping loss in the loss function of the generative adversarial network CycleGAN is modified to: L identity (G) = E x～pdata(x) ||F(G(X))–x_r||1; Among them, x_r is the color map of the virtual image, x_d is the depth map, x_n is the normal map, x_m is the semantic segmentation map, the generator is G, the output of the generator is G(X), X=[x_r,x_d,x_n,x_m], the real image is y, and pdata is F(G(X)). The RConv block maximizes the advantages of virtual data and uses the normal map and depth map as supplementary information to make the generative adversarial network based on the convolution kernel generate results that are more consistent with the real image; the virtual image and the real image are input into the generative adversarial network based on the convolution kernel to perform domain migration on the virtual data.
2. The image migration method for reducing domain differences as claimed in claim 1, characterized in that: The parameters of the generator and the discriminator are updated alternately using the gradient descent method.
3. The image migration method for reducing domain differences as claimed in claim 1, characterized in that: The virtual image consists of a virtual scene and three-dimensional objects.
4. The image migration method for reducing domain differences as claimed in claim 1, characterized in that: The shader is obtained through the shader editor of Blender.
5. An image migration system for reducing domain differences, using the image migration method for reducing domain differences as claimed in claim 1, characterized in that: include: An image acquisition module is configured to: acquire a virtual image; An auxiliary image generation module is configured to: generate multiple auxiliary images of the virtual image using different shaders based on the virtual image; An image migration module is configured to obtain a migration image based on multiple auxiliary images of the virtual image and using a generator of a generative adversarial network based on a convolution kernel; Among them, the generator of the generative adversarial network based on the convolution kernel uses a convolution layer to convolve multiple auxiliary maps respectively, adds the output of the convolution layer and inputs it into a convolution layer to obtain a fused feature map, and obtains a migration image based on the fused feature map.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of an image migration method for reducing domain differences as described in any one of claims 1 to 4 are implemented.
7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the image migration method for reducing domain differences as described in any one of claims 1 to 4 are implemented.