An image sample generation method, device, medium and computing device

Abstract

The application discloses an image sample generation method, device, medium and computing equipment, wherein the method comprises: acquiring at least one first real image containing a target and at least one three-dimensional rendering image about the target; generating a fusion image about the target based on the first real image, the fusion image having a pure background; extracting a style of the fusion image as a target style; and converting the three-dimensional rendering image about the target into a new image sample consistent with the target style. The application acquires a fusion sample with a pure background about a target based on a target real image sample, and then extracts a target style from the fusion sample with the pure background, so that the three-dimensional rendering image about the target can be converted into an image sample with the same target style based on the extracted target style.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to an image sample generation method and device, medium and computing equipment. BACKGROUND

[0002] In machine learning, a large number of target image samples need to be provided for machine learning. In some fields, it is relatively easy to obtain target image samples for machine learning. However, in some fields where it is difficult to obtain target image samples or the number of target image samples is small, such as some target image samples (such as ships and aircrafts) taken by satellites, due to the limitations of satellite orbits, day-night changes, and weather conditions, the number of target image samples obtained directly by satellite shooting is limited, and therefore a large number of target image samples cannot be provided for machine learning. Therefore, a style transfer model can be used to generate the required target image samples. However, for some complex backgrounds, the effect of the style transfer model is not ideal, and the background information will interfere with the generation quality of the target main body and the entire image. SUMMARY

[0003] The main purpose of the present application is to provide an image sample generation method, device, medium and computing equipment, which aims to solve the problems mentioned in the background.

[0004] To achieve the above purpose, the present application provides an image sample generation method, comprising:

[0005] Obtaining at least one first real image containing a target and at least one three-dimensional rendering image about the target;

[0006] Generating a fusion image about the target based on the first real image, the fusion image having a pure background;

[0007] Extracting the style of the fusion image as a target style;

[0008] Converting the three-dimensional rendering image about the target into a new image sample consistent with the target style.

[0009] In an embodiment, generating a fusion image about the target based on the first real image, the fusion image having a pure background, comprises:

[0010] Extracting a target sample and a local background sample from the first real image;

[0011] Fusing the target sample with the local background sample as a background to generate the fusion image.

[0012] In an embodiment, the local background sample is extracted based on the uniformity of the first real image background.

[0013] In an embodiment, the target sample is fused with the local background sample as background to generate the fusion image, including:

[0014] The local background sample is expanded to be able to accommodate the target sample.

[0015] The target sample is fused with the expanded local background sample to generate the fusion image.

[0016] In an embodiment, the style of the fusion image is extracted by a style transfer model to be a target style; and a three-dimensional rendering image about the target is converted into a new image sample consistent with the target style, including:

[0017] The style transfer model is trained using the fusion image and the three-dimensional rendering image about the target, and in the training process, a new image sample with the style of the fusion image and the content of the three-dimensional rendering image is generated.

[0018] In an embodiment, after the step of converting the three-dimensional rendering image about the target into a new image sample consistent with the target style, the method further includes:

[0019] Using the trained style transfer model, other three-dimensional rendering images about the target are converted into new image samples consistent with the target style.

[0020] In an embodiment, the three-dimensional rendering image about the target can be obtained by the following way:

[0021] Based on the target, a three-dimensional model is established to obtain a corresponding three-dimensional rendering image.

[0022] In an embodiment, after the three-dimensional rendering image about the target is converted into a new image sample consistent with the target style, the method further includes:

[0023]

[0024] A second real image without the target is obtained, and the background of the new image sample is replaced with the second real image as background, the second real image is a real image near the target area, and has the same light intensity as the converted three-dimensional rendering image.

[0025] In an embodiment, replacing the background of the new image sample with the second real image as background includes:

[0026] The target part of the new image sample is obtained.

[0027] ​Fusing the target part with a background of the second real image.

[0028] In an embodiment, the target part of the new image sample is obtained, comprising:

[0029] The new image sample is masked to remove the background to obtain the target part.

[0030] In an embodiment, the at least one first real image and the at least one three-dimensional rendering Figure One correspond, and the image state of the target in the corresponding first real image and the three-dimensional rendering image are consistent, and the image state consistency at least includes the same shooting angle and / or the same shooting light condition.

[0031] The application also provides an image sample generation device, comprising:

[0032] An acquisition module is configured to acquire at least one first real image containing a target and at least one three-dimensional rendering image about the target;

[0033] A fusion module is configured to generate a fusion image about the target based on the first real image, and the fusion image has a pure background;

[0034] A migration module is configured to extract the style of the fusion image as a target style, and

[0035] convert the three-dimensional rendering image about the target into a new image sample consistent with the target style.

[0036] In an embodiment, the fusion module is configured to:

[0037] extract a target sample and a local background sample from the first real image;

[0038] fuse the target sample with the local background sample as a background to generate the fusion image.

[0039] In an embodiment, the fusion module is further configured to:

[0040] extract the local background sample based on the uniformity of the background of the first real image.

[0041] In an embodiment, the fusion module is further configured to:

[0042] expand the local background sample to be able to accommodate the target sample;

[0043] fuse the target sample with the expanded local background sample into the fusion image.

[0044] In an embodiment, the migration module is configured:

[0045] The style transfer model is provided and trained based on the fusion image and the three-dimensional rendering image of the target, so that the style transfer model generates new image samples with the style of the fusion image and the content of the three-dimensional rendering image in the training process.

[0046] In an embodiment, the transfer module is further configured to:

[0047] Based on the trained style transfer model, other three-dimensional rendering images of the target are converted into new image samples consistent with the style of the target.

[0048] In an embodiment, the three-dimensional rendering image of the target can be obtained by:

[0049] Based on the target, a three-dimensional model is established to obtain a corresponding three-dimensional rendering image.

[0050] In an embodiment, the acquisition module is further configured to acquire a second real image without the target, the second real image being a real image near the target area and having the same light intensity as the converted three-dimensional rendering image.

[0051] The device further comprises a switching module configured to replace the background of the new image sample with the second real image.

[0052] In an embodiment, the switching module is configured to:

[0053] Acquire a target part of the new image sample.

[0054] Fuse the target part with the second real image as the background.

[0055] In an embodiment, the switching module is further configured to:

[0056] Mask the new image sample to remove the background to obtain the target part.

[0057] In an embodiment, the acquisition module is further configured to:

[0058] Acquire at least one corresponding first real image and at least one corresponding three-dimensional rendering image, and the three-dimensional rendering image is consistent with the image state of the target in the corresponding first real image, and the image state consistency at least includes the same shooting angle and / or the same shooting light condition.

[0059] The present application also provides a medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method according to any one of the above embodiments.

[0060] The application also provides a computing device comprising a processor configured to implement the above method when executing a computer program stored in a memory.

[0061] The technical scheme of the application, the application obtains a background-pure fusion sample about the target based on a target real image sample, and then extracts the style of the target from the background-pure fusion sample, so that the three-dimensional rendering image about the target can be converted into an image sample with the same style as the target based on the extracted target style. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical scheme in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only show some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from the structures shown in the drawings without creative labor.

[0063] Figure 1 A step diagram of an embodiment of the image sample generation method of the application;

[0064] Figure 2 A flowchart of an embodiment of the image sample generation method of the application;

[0065] Figure 3 A module diagram of an embodiment of the image sample generation device of the application;

[0066] Figure 4 A first real image of a ship photographed by a remote sensing satellite;

[0067] Figure 5 A target sample (ship body) extracted from Figure 4

[0068] Figure 6 A local background sample (local seawater background) extracted from Figure 4

[0069] An expanded background image; Figure 7 Figure 6 A fusion image generated after

[0070] Figure 8 Figure 5 Figure 7 and

[0071] Figure 9 A three-dimensional rendering image of the ship;

[0072] Figure 10 A simulated ship remote sensing image generated; ​​​​

[0073] Figure 11 is a complete background sample extracted from a real remote sensing image.

[0074] Figure 12 is a final sample image.

[0075] Figure 13 is a structural schematic diagram of an embodiment of the storage medium of the present application.

[0076] Figure 14 is a structural schematic diagram of an embodiment of the computer device of the present application.

[0077] The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with embodiments. DETAILED DESCRIPTION

[0078] The principles and spirits of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present application, and do not limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0079] Those skilled in the art know that the embodiments of the present application can be implemented as an apparatus, a device, a method or a computer program product. Therefore, the present disclosure can be embodied in the form of a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0080] According to the embodiments of the present application, an image sample generation method, device, medium and computer device are proposed.

[0081] Exemplary method

[0082] The technical solution of the present application can be applied in some fields where it is difficult to obtain target image samples or the number of target image samples is small. For example, some target image samples (such as ships, aircrafts, etc.) taken by remote sensing satellites can be used. In the following, an image sample generation method according to an exemplary embodiment of the present application will be described taking some target image samples taken by remote sensing satellites as an example, but this does not limit the technical solution of the present application to the field of satellite remote sensing images.

[0083] In the present exemplary embodiment, as shown in Figure 1 , Figure 2 the method comprises the following steps:

[0084] Step S100: acquiring at least one first real image containing a target and at least one three-dimensional rendering image about the target.

[0085] Step S200: generating a fusion image about the target based on the first real image, the fusion image having a pure background.

[0086] Step S300: extracting a style of the fusion image as a target style.

[0087] Step S400: converting a three-dimensional rendering graph about the target into a new image sample consistent with the target style.

[0088] For step S100, at least one first real image containing a target is acquired. Taking a ship as an example, the first real image needs to contain the target ship. The first real image can be directly acquired by a remote sensing satellite, or the first real image can be acquired by a remote sensing satellite in advance and then provided according to a preset interface or uploading mode. In addition, the first real image can be one or more. For the three-dimensional rendering graph about the target, if the target in the first real image is a ship, the three-dimensional rendering graph needs to be a three-dimensional rendering graph about a ship, which does not need to be the same ship. For example, in another embodiment, the target is an airplane, and the three-dimensional rendering graph needs to be a three-dimensional rendering graph about an airplane, such as a fighter or a transport plane. In addition, the number of acquired first real images and three-dimensional rendering graphs is not limited, for example, 500 first real images are acquired, and the three-dimensional rendering graph can be greater than 500, less than 500, or 500. The number of first real images and rendering graphs is not limited in the technical solution.

[0089] In an embodiment, the three-dimensional rendering graph about the target can be acquired by three-dimensional modeling. For example, taking a ship photographed by a satellite as an example, three-dimensional modeling can be performed on the target ship. After modeling, the ship at various angles can be obtained, and various light conditions can be selected for rendering.

[0090] For step S200, a fusion image about the target is generated based on the first real image, and the fusion image has a pure background. The steps include:

[0091] Step S210: extracting a target sample and a local background sample from the first real image.

[0092] For example, in an embodiment, for a first real image containing a target ship, the first real image contains two parts: a target ship part and a background part other than the target ship, so a ship sample (i.e., a target sample) can be extracted from the first real image, and a small-range local background sample can be extracted from the background part other than the target ship, and then the local background sample and the ship sample can be fused to obtain a fused image.

[0093] In another embodiment, the local background sample is extracted based on uniformity of the background of the first real image. For the first real image, the background part other than the target is composed of various local backgrounds, and the uniformity of each local background has a difference, so the region with the highest uniformity can be extracted from the background part other than the target as the local background sample.

[0094] Step S220: fuse the target sample with the local background sample as the background to generate the fused image. The size of the local background sample extracted in step S210 can be different from the target sample, and in this step, the local background sample can be first expanded, such as using a texture generation technology, to expand the local background sample to a size sufficient to accommodate the target sample, or to expand the local background sample to the same size as the background of the first real image. Then the target sample and the expanded local background sample can be fused into the fused image. Since the local background sample and the target sample extracted in step S210 are separately extracted, and the extracted local background sample has high uniformity, the fused image obtained by fusion in this step has a prominent target subject and a pure background.

[0095] After obtaining the fused image with a relatively pure background and a relatively prominent target subject, step S300 is performed to extract a style of the fused image as a target style. In this step, a style transfer model can be provided, and the style transfer model can be trained based on the fused image and a three-dimensional rendering image. First, a style transfer model is selected. For this technical solution, the first real image and the first rendering related to the target do not require one-to-one correspondence in quantity, so a non-paired input style transfer model can be selected, such as a CycleGAN model, a DualGAN model, and a GANILLA model, etc.

[0096] The present application does not limit the specific use of what style transfer model, and the following will be described by taking CycleGAN model as an example. For example, in step S100, 1000 first real images about ships are obtained based on ship images taken by remote sensing satellites, and 500 three-dimensional renderings are obtained. After steps S200 and S300, 1000 fusion images with relatively pure background are generated. At this time, the 1000 fusion images about ships and the 500 three-dimensional renderings about ships can be input into the CycleGAN model, so as to train the CycleGAN model, so that the trained CycleGAN model can extract the style of the ship in the 1000 real images of the ship, and can generate new image samples similar to the real ship images taken by remote sensing satellites (i.e. new image samples) based on, for example, three-dimensional renderings of other ships, so that the generated new image samples have the content of three-dimensional renderings and the same style as real ships.

[0097] It should be noted that the present technical solution does not require the first real image and the three-dimensional rendering Figure One correspondence, but is also applicable to one-to-one correspondence between the first real image and the three-dimensional rendering.

[0098] For example, in another embodiment, the first real image and the three-dimensional rendering Figure One correspondence, and the image state of the target in the three-dimensional rendering is consistent with that in the corresponding first real image, which at least includes the same shooting angle and / or the same shooting light condition. For example, first, 500 first real images for a target ship are obtained, and then three-dimensional modeling is performed based on the ship. Then, based on the ship in the three-dimensional modeling, the ship is adjusted to the same angle as in the 500 first real images, and the ship is rendered according to the light intensity of the ship in the 500 first real images, so as to obtain 500 three-dimensional renderings corresponding to the first real images and consistent with the image state. Since the first real images and the three-dimensional renderings obtained in step 100 are one-to-one correspondence, the fusion images and the three-dimensional renderings obtained after processing the first images in step 200 are also one-to-one correspondence. At this time, a style transfer model such as pix2pix model can be selected for paired input. When training, the fusion image and the corresponding three-dimensional rendering need to be input in pairs.

[0099] For step S400, the three-dimensional rendering image about the target is converted into a new image sample consistent with the style of the target. Since the style transfer model has been trained in step S300, in this step, based on the three-dimensional rendering image about the target, the trained style transfer model is used to generate a new image sample containing the target. For example, in the previous embodiment, the real image and the three-dimensional rendering image about the ship are used. At this time, a three-dimensional rendering image of the ship at another angle or under another lighting condition can be input into the trained style transfer model, so that a ship image sample of the same style as the real remote sensing image can be obtained.

[0100] Compared with the prior art, the technical solution first obtains a fusion image in which the target is prominent and the background is clean, and then trains the style transfer model using the target fusion image and the three-dimensional rendering image. Thus, the trained style transfer model can generate an image containing the target that is similar to the real image based on the three-dimensional rendering image. In the process of training the style transfer model, the fusion image with a clean background and a prominent subject is used. Therefore, the subject quality of the new image sample containing the target generated after training is high, and the similarity between the subject style and the subject of the real sample is high. On the other hand, three-dimensional modeling can be performed based on the target. After modeling, a three-dimensional model rendering image of the target under any angle and any lighting condition can be obtained. Therefore, by replacing the three-dimensional model rendering image of the target at different angles or under different lighting conditions, a new image sample containing the target under different angles or different lighting conditions can be generated adversarially. Thus, the deficiency of the prior art, in which the real sample is used as a single input, cannot generate new effective information, and can only help the recognition or classification algorithm to learn the original information in the real sample, is overcome.

[0101] In another embodiment of the present embodiment, after step S400, the method further includes step S500: obtaining a second real image not containing the target, and replacing the background of the new image sample with the background of the second real image. The second real image is a real image near the target region and has the same lighting intensity as the converted three-dimensional rendering image.

[0102] In the above embodiment, through steps S100-S400, a new image sample with high subject quality and similar subject and real style can be obtained based on the other three-dimensional rendering map of the target by the style transfer model. In this embodiment, after obtaining the new image sample, the background of the new image sample can be further restored. First, a second real image about the target needs to be obtained. Taking a ship in the field of remote sensing satellites as an example, assuming that the input of the trained style transfer model is a three-dimensional rendering map of the ship at a certain angle under certain lighting conditions, then according to the lighting intensity of the generated three-dimensional rendering map, a second image of the ship can be taken. It needs to be noted that although the second real image is an image of the ship in the vicinity, it cannot contain the ship. For example, the first real image obtained in step S100 is an image of the ship on a certain water area, then the water area near the ship can be photographed under the lighting conditions of the three-dimensional rendering map. In addition, the second image can be taken by the same remote sensing satellite, or other remote sensing satellites.

[0103] In step S200, the target sample under the real lighting condition and the local background sample are extracted, and then the local background sample is expanded and fused with the target sample. Therefore, for the target subject, the training process is the real style data of the target subject, and the background is obtained by expanding the local background, so there may be differences with the real background style. That is, using the trained model to input the three-dimensional rendering map to generate a new image containing the target, the style of the target subject part is real, and only the background part will have differences with the real style. Therefore, based on the lighting intensity or light parameter of the rendering map, the target area is photographed to obtain the real background (second real image). Then, the second real image is used to replace the background in the new image sample containing the ship obtained in step S500, and the background of the image sample containing the ship obtained thereafter is the background of the image sample of the ship photographed by the real remote sensing satellite. The specific replacement steps are as follows:

[0104] Step S510: obtaining the target part of the new image sample containing the target. For example, in an embodiment, the new image sample containing the target is masked to remove the background to obtain the target part. In this step, based on the use of the style transfer model, the target three-dimensional model rendering map is migrated to generate each new image sample containing the target. Each new image sample is masked to remove the background given by the style transfer model, and the target part of each new image sample is retained.

[0105] Step S520: fuse the target part with the second real image as the background. In this step, the second real image obtained is fused with the target part of each new image sample in step S610 as the background, and a final new image sample including the target is obtained. The final target image sample obtained at this time has a target main body style close to reality and a background belonging to the background environment in which the target is located in the actual situation, thereby avoiding the problems of background distortion and excessive noise after the style transfer model.

[0106] Next, in combination with Figures 4-12 , a specific description will be given. In this embodiment, as Figure 4 , a first real image of a ship photographed by a remote sensing satellite, Figure 5 , a target sample (ship main body) extracted from Figure 4 , a local background sample (local seawater background) extracted from Figure 6 , the local seawater background is expanded to a sufficient size by a texture generation algorithm to obtain an expanded background Figure 4 , and is fused with the ship target Figure 7 to obtain a ship fusion image with a pure background Figure 5 . Figure 8 , Figure 9 is a three-dimensional rendering image of the ship, 1000 Figure 8 and 1000 Figure 9 are input into a CycleGAN style transfer model for training. Then, other three-dimensional rendering images of the ship in Figure 9 are input into the trained CycleGAN style transfer model to obtain a simulated remote sensing image of the ship Figure 10 , Figure 11 is a complete background sample extracted from a real remote sensing image, and is fused with Figure 10 after the mask is removed to obtain a final sample Figure 12 .

[0107] An exemplary device

[0108] After introducing the method of the exemplary embodiment of the present application, next, an image sample generation device of the exemplary embodiment of the present application will be described, as shown in Figure 3 , the device comprises:

[0109] An acquisition module for acquiring at least one first real image containing a target and at least one three-dimensional rendering image about the target;

[0110] A fusion module for generating a fusion image about the target based on the first real image, the fusion image having a pure background;

[0111] a migration module configured to extract a style of the fusion image as a target style, and

[0112] convert a three-dimensional rendering image of the target into a new image sample consistent with the target style.

[0113] In an embodiment, the fusion module is configured to:

[0114] extract a target sample and a local background sample from the first real image;

[0115] fuse the target sample with the local background sample as a background to generate the fusion image.

[0116] In an embodiment, the fusion module is further configured to:

[0117] extract the local background sample based on homogeneity of a background of the first real image.

[0118] In an embodiment, the fusion module is further configured to:

[0119] extend the local background sample to be able to accommodate the target sample;

[0120] fuse the target sample with the extended local background sample into the fusion image.

[0121] In an embodiment, the migration module is configured:

[0122] provide a style migration model, and train the style migration model based on the fusion image and a three-dimensional rendering image of the target, so that the style migration model generates a new image sample with both the style of the fusion image and the content of the three-dimensional rendering image in an adversarial manner during the training process.

[0123] In an embodiment, the migration module is further configured to:

[0124] convert other three-dimensional rendering images of the target into new image samples consistent with the target style based on the trained style migration model.

[0125] In an embodiment, the three-dimensional rendering image of the target can be obtained by:

[0126] perform three-dimensional modeling based on the target to obtain a corresponding three-dimensional rendering image.

[0127] In an embodiment, the obtaining module is further configured to obtain a second real image without the target, the second real image being a real image near the target region and having the same light intensity as the converted three-dimensional rendering image.

[0128] The device further comprises a switching module configured to replace the background of the new image sample with the second real image.

[0129] In an embodiment, the switching module is configured to:

[0130] acquire a target part of the new image sample;

[0131] fuse the target part with the background of the second real image.

[0132] In an embodiment, the switching module is further configured to:

[0133] mask the new image sample to remove the background and obtain the target part.

[0134] In an embodiment, the acquiring module is further configured to:

[0135] acquire at least one of the first real image and at least one of the three-dimensional rendering image in one-to-one correspondence, and the three-dimensional rendering image is consistent with the image state of the target in the corresponding first real image, and the image state consistency at least includes the same shooting angle and / or the same shooting light condition.

[0136] The specific functions and methods of the above modules are described in the embodiments of the exemplary method, and will not be repeated here.

[0137] Exemplary medium

[0138] After introducing the method and device of the exemplary embodiments of the present application, next, with reference to Figure 13 The computer-readable storage medium of the exemplary embodiments of the present application is described.

[0139] Please refer to Figure 13The computer readable storage medium shown is an optical disc 70 having stored thereon a computer program (i.e. a program product) which, when run by a processor, implements the steps described in the above method embodiments, for example: obtaining at least one first real image containing a target and at least one three-dimensional rendering image of the target; generating a fusion image of the target based on the first real image, the fusion image having a clean background; extracting a style of the fusion image as a target style; and converting the three-dimensional rendering image of the target into a new image sample consistent with the target style. The specific implementation of each step is not repeated here. It should be noted that examples of the computer readable storage medium can also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical, magnetic storage media, which are not repeated here.

[0140] Exemplary computing device

[0141] Having introduced the method, apparatus and medium of the exemplary embodiments of the present application, next, reference is made to Figure 14 The computing device 80 of the exemplary embodiments of the present application is described.

[0142] Figure 14 A block diagram of an exemplary computing device 80 suitable for implementing embodiments of the present application is shown, which can be a computer system or a server. Figure 14 The computing device 80 shown is merely an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0143] As Figure 14 shown, the components of the computing device 80 can include, but are not limited to, one or more processors or processing units 801, a system memory 802, and a bus 803 that connects the different system components including the system memory 802 and the processing unit 801.

[0144] The computing device 80 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by the computing device 80 and includes both volatile and non-volatile media, removable and non-removable media.

[0145] The system memory 802 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022. The computing device 70 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a ROM 8023 can be used to read from and write to a non-removable, non-volatile magnetic media (e.g., a hard disk drive). Figure 14 not shown in the computing device 70, a magnetic hard disk drive, for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 14 not shown in the computing device 70, a magnetic hard disk drive, for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive").

[0146] A program / utility 8025 having a set (at least one) of program modules 8024, can be stored in, for example, system memory 802 and implemented by such a computer system. Each of the operating system, one or more application programs, other program modules, and program data can include implementations of the features of the present application.

[0147] The computing device 80 can also communicate with one or more external devices 804 such as a keyboard, a pointing device, a display, etc.; through Input / Output (I / O) interfaces. Additionally, the computing device 80 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the Internet through a network adapter 806. As Figure 14 illustrated, the network adapter 806 can communicate with the other components of the computing device 80 through the bus 803. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with the computing device 80. By way of example, the computing device 80 can be a Figure One stand-alone computing device, or it could be a client in a network computing environment. Also, a person of skill in the art will recognize that a computing device 80 could be implemented using client-server technology. A person of skill in the art will also recognize the extent to which the

[0148] The processing unit 801 performs various functional applications and data processing by running programs stored in the system memory 802, such as obtaining at least one first real image containing a target and at least one three-dimensional rendering image about the target, generating a fusion image about the target based on the first real image, the fusion image having a pure background, extracting a style of the fusion image as a target style, and converting the three-dimensional rendering image about the target into a new image sample consistent with the target style. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules.

[0149] In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or divided into multiple steps.

[0150] Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of aspects does not mean that the features in these aspects cannot be combined to benefit. This division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

[0151] The above description is only the preferred embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by referring to the content of the present application specification and drawings, or directly / indirectly applied to other related technical fields is included in the patent protection scope of the present application.

[0152] Through the above description, the embodiments of the present application at least provide the following technical solutions, but are not limited to this:

[0153] 1. An image sample generation method, comprising:

[0154] obtaining at least one first real image containing a target and at least one three-dimensional rendering image about the target;

[0155] generating a fusion image about the target based on the first real image, the fusion image having a pure background;

[0156] extracting a style of the fusion image as a target style;

[0157] convert the three-dimensional rendering map about the target into a new image sample consistent with the target style.

[0158] 2. The image sample generation method of claim 1, wherein a fusion image about the target is generated based on the first real image, the fusion image having a pure background, comprising:

[0159] extracting a target sample and a local background sample from the first real image;

[0160] fusing the target sample with the local background sample as a background to generate the fusion image.

[0161] 3. The image sample generation method of claim 1 or 2, wherein the local background sample is extracted based on the uniformity of the background of the first real image.

[0162] 4. The image sample generation method of any one of claims 1-3, wherein the target sample is fused with the local background sample as a background to generate the fusion image, comprising:

[0163] extending the local background sample to be able to accommodate the target sample;

[0164] fusing the target sample with the extended local background sample into the fusion image.

[0165] 5. The image sample generation method of any one of claims 1-4, wherein the style of the fusion image is extracted by a style transfer model to be a target style; and converting the three-dimensional rendering map about the target into a new image sample consistent with the target style, comprising:

[0166] training the style transfer model using the fusion image and the three-dimensional rendering map about the target, and in the training process, generating a new image sample that has both the style of the fusion image and the content of the three-dimensional rendering map by adversarial generation.

[0167] 6. The image sample generation method of any one of claims 1-5, wherein after the step of converting the three-dimensional rendering map about the target into a new image sample consistent with the target style, the method further comprises:

[0168] using the trained style transfer model to convert other three-dimensional rendering maps about the target into new image samples consistent with the target style.

[0169] 7. The image sample generation method of any one of claims 1-7, wherein the three-dimensional rendering map about the target can be obtained by:

[0170] based on the target, a three-dimensional model is established, and a corresponding three-dimensional rendering image is obtained.

[0171] 8. The image sample generation method of any one of claims 1-7, wherein the at least one three-dimensional rendering image of the target is converted into a new image sample that is consistent with a style of the target.

[0172] The method further comprises the following steps after the three-dimensional rendering image is converted into the new image sample that is consistent with the style of the target:

[0173] A second real image without the target is obtained, and a background of the new image sample is replaced with the second real image, the second real image is a real image near the target region, and has the same lighting intensity as the converted three-dimensional rendering image.

[0174] 9. The image sample generation method of any one of claims 1-8, wherein replacing the background of the new image sample with the second real image comprises:

[0175] A target part of the new image sample is obtained.

[0176] The target part is fused with the second real image as the background.

[0177] 10. The image sample generation method of any one of claims 1-9, wherein the target part of the new image sample is obtained by:

[0178] The new image sample is masked to remove the background to obtain the target part.

[0179] 11. The image sample generation method of any one of claims 1-10, wherein the at least one first real image and the at least one three-dimensional rendering image correspond to each other, and the three-dimensional rendering image is consistent with an image state of the target in the corresponding first real image, the image state consistent at least including the same shooting angle and / or the same shooting lighting condition. ​

[0180] 12. An image sample generation device, comprising:

[0181] An acquisition module is configured to acquire at least one first real image containing a target and at least one three-dimensional rendering image of the target.

[0182] A fusion module is configured to generate a fusion image of the target based on the first real image, the fusion image having a pure background.

[0183] A migration module is configured to extract a style of the fusion image as a target style, and

[0184] The three-dimensional rendering image of the target is converted into a new image sample that is consistent with the style of the target.​

[0185] 13. The image sample generation apparatus of claim 12, wherein the fusion module is configured to:

[0186] extract a target sample and a local background sample from the first real image;

[0187] fuse the target sample with the local background sample as background to generate the fused image.

[0188] 14. The image sample generation apparatus of claim 12 or 13, wherein the fusion module is further configured to:

[0189] extract the local background sample based on uniformity of background of the first real image.

[0190] 15. The image sample generation apparatus of any one of claims 12-14, wherein the fusion module is further configured to:

[0191] extend the local background sample to be able to accommodate the target sample;

[0192] fuse the target sample with the extended local background sample to generate the fused image.

[0193] 16. The image sample generation apparatus of any one of claims 12-15, wherein the migration module is configured to:

[0194] provide a style migration model, and train the style migration model based on the fused image and a three-dimensional rendering map about the target, so that the style migration model generates, in the training process, a new image sample that has both the style of the fused image and the content of the three-dimensional rendering map.

[0195] 17. The image sample generation apparatus of any one of claims 12-16, wherein the migration module is further configured to:

[0196] convert other three-dimensional rendering maps about the target into new image samples that are consistent with the style of the target based on the trained style migration model.

[0197] 18. The image sample generation apparatus of any one of claims 12-17, wherein the three-dimensional rendering map about the target is obtained by:

[0198] performing three-dimensional modeling based on the target to obtain the corresponding three-dimensional rendering map.

[0199] 19. The image sample generation apparatus of any of claims 12-18, wherein the obtaining module is further configured to obtain a second real image that does not include the target, the second real image being a real image of a region near the target region and having the same lighting intensity as the converted three-dimensional rendering.

[0200] The apparatus further includes a switching module configured to replace a background of the new image sample with the second real image.

[0201] 20. The image sample generation apparatus of any of claims 12-19, wherein the switching module is configured to:

[0202] obtain a target portion of the new image sample;

[0203] fuse the target portion with the second real image as a background.

[0204] 21. The image sample generation apparatus of any of claims 12-20, wherein the switching module is further configured to:

[0205] mask the new image sample to remove the background to obtain the target portion.

[0206] 22. The image sample generation apparatus of any of claims 12-21, wherein the obtaining module is further configured to:

[0207] obtain at least one of the first real image and at least one of the three-dimensional rendering in one-to-one correspondence, and the three-dimensional rendering is consistent with an image state of the target in the corresponding first real image, the image state consistency including at least a same shooting angle and / or a same shooting lighting condition.

[0208] 23. A medium having stored thereon a computer program, the computer program being executed by a processor to implement the method of any of claims 1-11.

[0209] 24. A computing device comprising a processor configured to implement the method of any of claims 1-11 when executing a computer program stored in a memory.

Claims

1. An image sample generation method, comprising: obtaining at least one first real image containing a target and at least one three-dimensional rendering image about the target; generating a fusion image about the target based on the first real image, the fusion image having a pure background; extracting a style of the fusion image as a target style through a style transfer model; converting the three-dimensional rendering image about the target into a new image sample consistent with the target style; wherein converting the three-dimensional rendering image about the target into a new image sample consistent with the target style comprises: training the style transfer model using the fusion image and the three-dimensional rendering image about the target, and generating a new image sample having both the style of the fusion image and the content of the three-dimensional rendering image in the training process.

2. The method of claim 1, wherein, generating a fusion image about the target based on the first real image, the fusion image having a pure background, comprises: extracting a target sample and a local background sample from the first real image; fusing the target sample with the local background sample to generate the fusion image.

3. The method of claim 2, wherein, extracting the local background sample based on the uniformity of the background of the first real image.

4. The method of claim 2, wherein, fusing the target sample with the local background sample to generate the fusion image comprises: extending the local background sample to accommodate the target sample; and fusing the target sample with the extended local background sample into the fusion image.

5. The method of claim 1, wherein, After the step of converting the three-dimensional rendering image about the target into a new image sample consistent with the target style, the method further comprises: using the trained style transfer model to convert other three-dimensional rendering images about the target into new image samples consistent with the target style.

6. The method of claim 1, wherein, The three-dimensional rendering image about the target can be obtained by: performing three-dimensional modeling based on the target to obtain a corresponding three-dimensional rendering image.

7. The method of claim 5, wherein the image sample is generated by: After converting the other three-dimensional rendering images about the target into new image samples consistent with the target style, the method further comprises: obtaining a second real image not containing the target, and replacing the background of the new image sample with the second real image as the background, the second real image being a real image near the target region and having the same illumination intensity as the converted three-dimensional rendering image.

8. The image sample generation method of claim 7, wherein, Replacing the background of the new image sample with the second real image as the background comprises: obtaining a target part of the new image sample; fusing the target part with the second real image as the background.

9. The method of claim 8, wherein, Obtaining the target part of the new image sample comprises: masking the new image sample to remove the background to obtain the target part.

10. The method of claim 1, wherein, The at least one first real image and the at least one three-dimensional rendering image correspond one-to-one, and the three-dimensional rendering image is consistent with the image state of the target in the corresponding first real image, the image state consistency at least including the same shooting angle and / or the same shooting illumination condition.

11. An image sample generation apparatus, characterized by comprising: comprising: an obtaining module, configured to obtain at least one first real image containing a target and at least one three-dimensional rendering image about the target; a fusion module configured to generate a fusion image of the target based on the first real image, the fusion image having a clean background; a migration module configured to extract a style of the fusion image as a target style, and convert a three-dimensional rendering image of the target into a new image sample consistent with the target style; the migration module is configured to: provide a style migration model, and train the style migration model based on the fusion image and the three-dimensional rendering image of the target, so that the style migration model generates a new image sample having both the style of the fusion image and the content of the three-dimensional rendering image in an adversarial manner during the training process.

12. The video sample generation apparatus of claim 11, wherein, the fusion module is configured to: extract a target sample and a local background sample from the first real image; fuse the target sample with the local background sample as a background to generate the fusion image.

13. The video sample generation apparatus of claim 12, wherein, the fusion module is further configured to: extract the local background sample based on the uniformity of the background of the first real image.

14. The video sample generation apparatus of claim 12, wherein, the fusion module is further configured to: extend the local background sample to be able to accommodate the target sample; fuse the target sample with the extended local background sample into the fusion image.

15. The video sample generation apparatus of claim 11, wherein the migration module is further configured to: convert other three-dimensional rendering images of the target into new image samples consistent with the target style based on the trained style migration model.

16. The video sample generation apparatus of claim 11, wherein The three-dimensional rendering image of the target can be obtained by: performing three-dimensional modeling based on the target to obtain a corresponding three-dimensional rendering image.

17. The video sample generation apparatus of claim 11, wherein, The obtaining module is further configured to obtain a second real image not containing the target, the second real image being a real image near the target region and having the same light intensity as the converted three-dimensional rendering image; the device further comprises a switching module configured to replace a background of the new image sample with the second real image as a background.

18. The video sample generation apparatus of claim 17, wherein, the switching module is configured to: obtain a target part of the new image sample; fuse the target part with the second real image as a background.

19. The video sample generation apparatus of claim 18, wherein, the switching module is further configured to: mask the new image sample to remove the background to obtain the target part.

20. The video sample generation apparatus of claim 11, wherein, the obtaining module is further configured to: obtain at least one of the first real image and at least one of the three-dimensional rendering image in a one-to-one correspondence, and the three-dimensional rendering image is consistent with the image state of the target in the corresponding first real image, the image state consistency at least including the same shooting angle and / or the same shooting light condition.

21. A medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1-10.

22. A computing device, comprising: The computing device includes a processor configured to implement the method of any one of claims 1-10 when the processor executes a computer program stored in a memory.

Images