Method and device for synthesizing image

Abstract

The invention provides a method and a device for synthesizing an image, relates to the field of artificial intelligence, in particular to the fields of deep learning, data synthesis, image processing and computer vision, and can be used for scenes of industry, manufacturing industry and the like. The specific implementation scheme is as follows: acquiring an original image and a target image; carrying out image matting on the original image to obtain a target object; pasting the target object into the target image to obtain a spliced image; and carrying out local redrawing on the splicing part of the target object and the target image in the spliced image to obtain a composite image. According to the embodiment, a reasonable image with natural foreground and background transition can be quickly and conveniently synthesized, and certain illumination / structure difference can be tolerated. Manual parameter adjustment is not needed, and the model does not need to be retrained in different use scenes.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of computer vision, and specifically to a method and device for synthesizing images. Background Art

[0002] Data is an important foundation for deep learning. A good data set can help deep learning models learn better feature representations and improve the generalization ability of the model. Specifically, the importance of data to deep learning is reflected in the following aspects: Sufficient quantity and high quality are the key to improving model performance. A large amount of high-quality labeled data can provide the model with rich information, allowing it to learn more accurate and complete feature representations. The larger the amount of data, the better the performance of the model. Data containing rich feature information, data from different perspectives and different environments can increase the diversity of features and avoid overfitting of certain features, which can improve the generalization ability of the model. The quality and quantity of data directly determine the upper limit of the model.

[0003] In industrial scenarios, the collection of image data required for deep learning model training does face certain difficulties, which are mainly reflected in the following aspects: the industrial site environment is complex and changeable, the equipment and environment of the industrial site are often adjusted and changed, and the lighting conditions and viewing angle ranges vary greatly. It is very difficult to collect data covering various situations. Some objects are rare and the sample size is insufficient. Some image samples such as faults have a low probability of occurrence, and it is difficult to collect a sufficient number of samples. This will lead to a serious shortage of data for such samples. Direct collection of some images containing sensitive information will involve commercial secrets or privacy leaks, and they need to be isolated or processed before they can be used for model training. Collecting high-quality industrial image data is a task that consumes a lot of manpower and material resources. Summary of the invention

[0004] The present disclosure provides a method, apparatus, device, storage medium and computer program product for synthesizing images.

[0005] According to a first aspect of the present disclosure, a method for synthesizing an image is provided, comprising: acquiring an original image and a target image; cutting out a target object from the original image; pasting the target object into the target image to obtain a stitched image; and locally redrawing a joint between the target object and the target image in the stitched image to obtain a synthesized image.

[0006] According to a second aspect of the present disclosure, a device for synthesizing an image is provided, comprising: an acquisition unit, configured to acquire an original image and a target image; a cutout unit, configured to cut out the original image to obtain a target object; a splicing unit, configured to paste the target object into the target image to obtain a spliced ​​image; and a redrawing unit, configured to locally redraw the splicing portion between the target object and the target image in the spliced ​​image to obtain a synthesized image.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in the first aspect.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to the first aspect.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the methods according to the first aspect when executed by a processor.

[0010] The method and device for synthesizing images provided by the embodiments of the present disclosure use deep learning technology to quickly synthesize the defective targets that have been collected onto the background image without defects, thereby achieving the purpose of expanding the amount of training sample data. The present disclosure is applicable to all targets with clear contours, and allows the synthesized targets to adapt to the new background image context information while ensuring that the semantic content of the target body does not change.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0014] Figure 2 is a flow chart of one embodiment of a method for synthesizing an image according to the present disclosure;

[0015] Figure 3is a schematic diagram of an application scenario of the method for synthesizing images according to the present disclosure;

[0016] Figure 4 is a flowchart of another embodiment of a method for synthesizing an image according to the present disclosure;

[0017] Figure 5 is a structural schematic diagram of an embodiment of a device for synthesizing images according to the present disclosure;

[0018] Figure 6 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the method for synthesizing an image or the apparatus for synthesizing an image of the present disclosure can be applied.

[0021] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0022] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as image synthesis applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0023] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and supporting image browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0024] The server 105 may be a server that provides various services, such as a background synthesis server that provides synthesis functions for images uploaded by the terminal devices 101, 102, and 103. The background synthesis server may analyze and process the received synthesis request and other data, and feed back the processing results (such as the synthesized image) to the terminal device. The synthesized image may also be stored in the server as a sample for model training.

[0025] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0026] It should be noted that the method for synthesizing images provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for synthesizing images is generally disposed in the server 105 .

[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0028] Continue to refer Figure 2, shows a process 200 of an embodiment of a method for synthesizing an image according to the present disclosure. The method for synthesizing an image comprises the following steps:

[0029] Step 201, obtaining an original image and a target image.

[0030] In this embodiment, the execution subject of the method for synthesizing an image (eg Figure 1 The server shown in the figure can receive the original image and the target image from the terminal used by the user for image synthesis through a wired connection or a wireless connection. The original image is an image that needs to contain a target object, and the target object can be an entity, such as a person, a dog, a car, etc., or an abstract object, such as a defect such as a wormhole or a crack. The target image is a background image that does not contain the target object.

[0031] Step 202, cut out the original image to obtain the target object.

[0032] In this embodiment, cutout is one of the most common operations in image processing, which is to separate a part of a picture or image from the original picture or image into a separate layer. The main function is to prepare for later synthesis. The methods include direct selection such as lasso tool, selection box tool, eraser tool, quick mask, pen drawing path and then converting to selection area, extracting filter, plug-in filter extraction, channel, calculation, application of image method, etc. Existing cutout tools can be used for cutout.

[0033] Step 203, pasting the target object into the target image to obtain a spliced ​​image.

[0034] In this embodiment, the target object obtained by cutting out the image is pasted onto the background image (ie, the target image) to obtain a spliced ​​image. The pasting position can be specified or random.

[0035] Step 204 , locally redraw the joint between the target object and the target image in the stitched image to obtain a composite image.

[0036] In this embodiment, the inpainting technique is used to partially repaint the joints of the target object and the target image to achieve the goal of a natural transition. The specific method is as follows: for the input joint image x, a binary mask m is used to mask the joints x⊙m to form a four-channel input image x′=stack(x⊙m,m), and then a forward infer network fθ(·) is used to repair in a full convolutional manner to obtain a three-channel color image.

[0037] Existing methods do not have a sufficiently wide receptive field. Therefore, for large-resolution images or large-scale inpainting, it is easy to be affected by local details or masks, resulting in poor results. Therefore, the model has a larger receptive field at the initial stage of the network, and a network structure based on fast Fourier convolution (FFC) is proposed for local repainting. The pre-trained segmentation network is used for feature extraction to realize the construction of the loss function, serving large receptive fields and large masks.

[0038] The existing Poisson fusion technology uses the Poisson equation to seamlessly fuse the color features of the foreground image into the background image. The main disadvantages of Poisson fusion are: it requires an accurate foreground mask. Poisson fusion relies on an accurate foreground object mask for seamless fusion, and inaccurate mask extraction will result in obvious fusion gaps. Inconsistent lighting effects. Different lighting conditions for the foreground and background will result in uncoordinated lighting effects, reducing the realism of the fusion effect. Only color features are fused. Poisson fusion only adjusts the color and cannot adjust detail features such as texture and shadow according to the scene. Parameters need to be manually adjusted. Some parameters in the algorithm need to be manually selected according to the scene, which is not intelligent enough.

[0039] Existing technologies can also use generative adversarial networks (GANs) for image data synthesis, but there are several disadvantages: Mode collapse, the generator may converge to only output a few fixed samples and fail to generate images of sufficient diversity. Unstable training During the training process of GAN, it is difficult to achieve a balance between the capabilities of the generator and the discriminator, resulting in an unstable training process and easy convergence to a local optimum. Difficult to evaluate, due to the diverse network structures of GAN, the generated results are highly subjective and difficult to accurately evaluate. Limited image quality The quality and resolution of images generated by GAN may be difficult to meet the requirements of practical applications, and the details generated are not rich enough. A large amount of training data is required. The training of GAN relies on a large amount of real image data, which is also a challenge for some application scenarios. The generation process is uncontrollable. The process of generating new samples by GAN is a black box, and users cannot clearly control the properties of the generated images.

[0040] The method provided by the above embodiments of the present disclosure is that the synthesized target original image comes from a real image, and the synthesis position can be manually edited to ensure the rationality of the synthesized data. It is compatible with mask gaps within a certain size range and has low requirements for mask extraction accuracy. The foreground and background transition is natural and tolerates certain lighting differences. No manual parameter adjustment is required, and the model does not need to be retrained for different usage scenarios.

[0041] In some optional implementations of this embodiment, pasting the target object into the target image to obtain a stitched image includes: rotating and / or scaling the target object and then pasting it into the target image to obtain a stitched image. The purpose is to perform data augmentation operations while pasting the result obtained by cutting out onto the background image. The operations that can be performed include pasting position selection, size scaling, rotation, etc., so that data synthesis is more flexible and reasonable. The number of samples is expanded, which is conducive to training a model with higher accuracy.

[0042] In some optional implementations of this embodiment, pasting the target object into the target image to obtain a stitched image includes: detecting a pre-specified target position from the target image; pasting the target object into the target position to obtain a stitched image. For example, if you want to synthesize samples for road defect detection, you need to paste the cut-out cracks into the road in the background image, but not paste them into the sky at random. Optionally, the target position can be detected in the target image through a target detection model, and then the target object can be pasted into the target position to obtain a stitched image.

[0043] In some optional implementations of this embodiment, the locally redrawing the joint between the target object and the target image in the spliced ​​image to obtain a composite image includes: using a binary mask to cover the area within a predetermined range around the target object to form a four-channel image; inputting the four-channel image into a pre-trained prediction network to output a three-channel color image, wherein the prediction network repairs the four-channel image by full convolution to obtain a three-channel color image. For example, if the target object edge is covered within a width of 1 cm outward, the 1 cm width is locally redrawn.

[0044] Existing methods do not have a sufficiently wide receptive field, so for large-resolution images or large-scale inpainting, they are easily affected by local details or masks, resulting in poor results. Therefore, considering giving the model a larger receptive field at the initial stage of the network, a network structure based on fast Fourier convolution (FFC) is proposed.

[0045] The pre-trained segmentation network is used for feature extraction to realize loss function construction, serving large receptive fields and large masks.

[0046] Specific method: For the input stitching image x, use a binary mask m to mask the stitching x⊙m to form a four-channel input image x′=stack(x⊙m,m), and then use a forward infer network fθ(·) to repair it in a fully convolutional way to obtain a three-channel color image.

[0047] The spliced ​​image and mask form a pair to input the prediction network and after downsampling, enter the FFC residual block. In the FFC block, the input tensor is divided into two branches for operation. The local branch uses conventional convolution; the global branch uses Real FFT for global context attention. In the global branch, Real FFT2d and Inverse RealFFT2d operations are performed to achieve image reconstruction. In the output of FFC, the two branches merge the results and output the repaired image.

[0048] The loss function includes the high receptive field perceptual loss HRFPL, which is a calculation of the feature similarity between the input image and the generated image through a pre-trained basic network. Specifically, the HRF constructed using dilated convolution or Fourier convolution is calculated pixel by pixel, and then the mean of the same layer is taken first and then the mean of the layers is taken.

[0049] Optionally, the discriminator is referenced during training, and the prediction network is used as a generator to train the adversarial generative network, then the loss function also includes the adversarial loss.

[0050] In some optional implementations of this embodiment, the method of cutting out the original image to obtain the target object includes: cutting out the original image using an interactive segmentation algorithm to obtain the target object. The purpose is to perform pixel-level cutting out of the target to be pasted, using interactive segmentation and other similar methods. Interactive segmentation is to separate the foreground and background required to be segmented by the user by providing certain interactive information, including clicks, bboxes, closed curves, non-closed curves and other interactive methods. Interactive segmentation is a relatively basic task in computer vision and has been developed for a long time. In the past decade, due to the development of neural networks, interactive segmentation has achieved better results than in the past and has ushered in further development. The main purpose of interactive segmentation is to expand the data set, and a more satisfactory segmentation result can be obtained through simple annotation, thereby reducing the cost of segmentation annotation and improving the effect of the segmentation network. Another major application is cutting out, so Adobe has many research results in this field and the quality of the papers is also relatively good.

[0051] Its advantages are mainly in the following aspects: (1) Simple interaction can obtain better segmentation results. (2) It is more targeted than semantic segmentation and panoptic segmentation, and usually has better segmentation effect on single objects. (3) Since segmentation is generally only focused on the part of the image that the user needs to segment, the computational complexity is generally not high and the running time is short. (4) The training process mostly uses a category-agnostic method, so it has a certain generalization ability for categories that have not appeared.

[0052] In some optional implementations of this embodiment, the obtaining of the original image and the target image includes: detecting the image set through a defect detection model to obtain defective images and non-defective images; using the defective images as target images and the non-defective images as original images. The defective images can be automatically screened out as original images through the defect detection model, saving the cost of manually selecting original images.

[0053] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the method for synthesizing images according to this embodiment. Figure 3 In the application scenario, the original image containing the target object and the target image without the target object are obtained as the background image. The target object to be pasted is cut out at the pixel level using an interactive segmentation method. While pasting the cutout result onto the background image, data augmentation operations are performed. The operations that can be performed include pasting position selection, size scaling, rotation, etc., making data synthesis more flexible and reasonable. Then, the image joints are partially redrawn to achieve the goal of natural transition.

[0054] For example, in a wooden toy manufacturing workshop, it is necessary to detect defects (cracks, wormholes, etc.) on wooden toys through image detection algorithms. Due to the influence of the shape of wooden toys, dust, light in the manufacturing workshop, and shooting angle, it is very difficult to collect wooden toy images containing various defects. Through the method of the present application, defective image samples can be synthesized. For example, a square wooden toy image A obtained by looking down on the assembly line of the manufacturing workshop has a wormhole in the upper left corner, and another circular wooden toy image B has a crack in the lower right corner. The wormhole can be obtained by cutting out the wooden toy image A, and the crack can be obtained by cutting out the wooden toy image B. An image with at least one wormhole in the lower right corner, the middle, etc. can be synthesized with toys of various shapes as the background, or an image with at least one crack in the lower right corner, the middle, etc. can be synthesized with toys of various shapes as the background. Images with both wormholes and cracks in the lower right corner, the middle, etc. can also be synthesized with toys of various shapes as the background. In this way, the number of samples can be expanded and sufficient samples can be obtained, thereby improving the accuracy and convergence speed of model training. Further reference Figure 4 , which shows a process 400 of another embodiment of a method for synthesizing an image. The process 400 of the method for synthesizing an image comprises the following steps:

[0055] Step 401, obtaining a sample set.

[0056] In this embodiment, the execution subject of the method for synthesizing an image (eg Figure 1The server shown in the figure can receive a sample set from a terminal used by a user for image synthesis through a wired connection or a wireless connection. The samples in the sample set include an original sample image and a mask image after masking the original sample image; or only the original sample image can be received, and then the original sample image can be masked by a randomly generated mask to obtain a mask image. The original sample image and the mask image form a pair of samples. A variety of large mask generation methods are adopted to generate the mask image, but at the same time, attention is paid to avoid generating a mask larger than 50% of the original image.

[0057] Step 402: Use the mask image and the original sample image in the sample set as the input and expected output of the prediction network respectively, and perform supervised training.

[0058] In this embodiment, the prediction network can be a generator in a generative adversarial network (the generative adversarial network includes a generator and a discriminator), the input of the prediction network is a mask image, and the output is a restored image.

[0059] The mask image and the original sample image form a pair. The mask image is input into the prediction network and after downsampling, it enters the FFC residual block. In the FFC block, the input tensor is divided into two branches for operation. The local branch uses conventional convolution; the global branch uses Real FFT for global context attention. In the global branch, Real FFT2d and Inverse Real FFT2d operations are performed to achieve image reconstruction. In the output of FFC, the results of the two branches are merged to obtain the repaired image. The repaired image is compared and analyzed with the original sample image, and the loss value is calculated based on the difference. The network parameters of the prediction network and the discriminator are continuously adjusted to converge the loss value. When the loss value is less than the predetermined value, a trained prediction network is obtained, which can be used in steps 201-204.

[0060] The loss function includes the high receptive field perceptual loss HRFPL, which is a calculation of the feature similarity between the input image and the generated image through a pre-trained basic network. Specifically, the HRF constructed using dilated convolution or Fourier convolution is calculated pixel by pixel, and then the mean of the same layer is taken first and then the mean of the layers is taken.

[0061] from Figure 4 It can be seen that Figure 2 Compared with the corresponding embodiment, the process 400 of the method for synthesizing images in this embodiment embodies the steps of training the prediction network. Therefore, the scheme described in this embodiment can train a prediction network with high accuracy, thereby improving the effect of local redrawing.

[0062] Further references Figure 5As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for synthesizing images, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0063] like Figure 5 As shown, the apparatus 500 for synthesizing images in this embodiment includes: an acquisition unit 501, a cutout unit 502, a splicing unit 503, and a redrawing unit 504. The acquisition unit 501 is configured to acquire an original image and a target image; the cutout unit 502 is configured to cut out the original image to obtain a target object; the splicing unit 503 is configured to paste the target object into the target image to obtain a spliced ​​image; and the redrawing unit 504 is configured to partially redraw the splicing portion between the target object and the target image in the spliced ​​image to obtain a synthesized image.

[0064] In this embodiment, the specific processing of the acquisition unit 501, the cutout unit 502, the splicing unit 503 and the redrawing unit 504 of the device for synthesizing an image 500 can be referred to in Figure 2 Corresponding to step 201, step 202, step 203, and step 204 in the embodiment.

[0065] In some optional implementations of this embodiment, the stitching unit 503 is further configured to: rotate and / or scale the target object and then paste it into the target image to obtain a stitched image.

[0066] In some optional implementations of this embodiment, the stitching unit 503 is further configured to: detect a pre-specified target position from the target image; and paste the target object to the target position to obtain a stitched image.

[0067] In some optional implementations of this embodiment, the redrawing unit 504 is further configured to: use a binary mask to cover an area within a predetermined range around the target object to form a four-channel image; input the four-channel image into a pre-trained prediction network and output a three-channel color image, wherein the prediction network repairs the four-channel image by full convolution to obtain a three-channel color image.

[0068] In some optional implementations of the present embodiment, the device 500 further includes a training unit (not shown in the drawings), which is configured to: obtain a sample set, wherein the samples in the sample set include an original sample image and a mask image obtained by masking the original sample image; use the mask image and the original sample image in the sample set as the input and expected output of the prediction network, respectively, for supervised training.

[0069] In some optional implementations of this embodiment, the cutout unit 502 is further configured to: cut out the original image using an interactive segmentation algorithm to obtain the target object.

[0070] In some optional implementations of this embodiment, the acquisition unit 501 is further configured to: detect the image set through a defect detection model to obtain defective images and defect-free images; use the defective images as target images and the defect-free images as original images.

[0071] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0072] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0073] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in process 200 or 400.

[0074] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in process 200 or 400.

[0075] A computer program product includes a computer program, wherein the computer program implements the method described in flow 200 or 400 when executed by a processor.

[0076] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0077] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0078] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0079] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the method of synthesizing an image. For example, in some embodiments, the method of synthesizing an image may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method of synthesizing an image described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method of synthesizing an image in any other appropriate manner (e.g., by means of firmware).

[0080] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0082] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0083] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0084] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0085] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0086] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0087] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for synthesizing an image, comprising: Get the original image and the target image; Cut out the original image to obtain a target object; Pasting the target object into the target image to obtain a spliced ​​image; The joint portion between the target object and the target image in the stitched image is partially redrawn to obtain a composite image.

2. The method according to claim 1, wherein: Pasting the target object into the target image to obtain a spliced ​​image includes: The target object is rotated and / or scaled and then pasted into the target image to obtain a spliced ​​image.

3. The method according to claim 1, wherein: Pasting the target object into the target image to obtain a spliced ​​image includes: Detecting a pre-specified target position from the target image; Paste the target object to the target position to obtain a spliced ​​image.

4. The method according to claim 1, wherein: The step of partially redrawing the joint between the target object and the target image in the stitched image to obtain a composite image includes: Using a binary mask to cover an area within a predetermined range around the target object to form a four-channel image; The four-channel image is input into a pre-trained prediction network, and a three-channel color image is output, wherein the prediction network repairs the four-channel image by full convolution to obtain a three-channel color image.

5. The method according to claim 4, wherein: The prediction network is trained by the following steps: Acquire a sample set, wherein the samples in the sample set include an original sample image and a mask image obtained by masking the original sample image; The mask image and the original sample image in the sample set are used as the input and expected output of the prediction network respectively for supervised training.

6. The method according to claim 1, wherein: The step of clipping the original image to obtain the target object comprises: An interactive segmentation algorithm is used to cut out the original image to obtain the target object.

7. The method according to claim 1, wherein: The obtaining of the original image and the target image comprises: The image set is detected by a defect detection model to obtain defective images and non-defective images; The defective image is taken as the target image and the defect-free image is taken as the original image.

8. A device for synthesizing an image, comprising: An acquisition unit, configured to acquire an original image and a target image; A cutout unit is configured to cut out a target object from the original image; a stitching unit, configured to paste the target object into the target image to obtain a stitched image; The redrawing unit is configured to locally redraw the splicing portion between the target object and the target image in the spliced ​​image to obtain a composite image.

9. The device according to claim 8, wherein: The splicing unit is further configured to: The target object is rotated and / or scaled and then pasted into the target image to obtain a spliced ​​image.

10. The device according to claim 8, wherein: The splicing unit is further configured to: Detecting a pre-specified target position from the target image; Paste the target object to the target position to obtain a spliced ​​image.

11. The device according to claim 8, wherein: The redraw unit is further configured to: Using a binary mask to cover an area within a predetermined range around the target object to form a four-channel image; The four-channel image is input into a pre-trained prediction network, and a three-channel color image is output, wherein the prediction network repairs the four-channel image by full convolution to obtain a three-channel color image.

12. The device according to claim 11, wherein The apparatus further comprises a training unit configured to: Acquire a sample set, wherein the samples in the sample set include an original sample image and a mask image obtained by masking the original sample image; The mask image and the original sample image in the sample set are used as the input and expected output of the prediction network respectively for supervised training.

13. The device according to claim 8, wherein: The cutout unit is further configured to: An interactive segmentation algorithm is used to cut out the original image to obtain the target object.

14. The device according to claim 8, wherein: The acquisition unit is further configured to: The image set is detected by a defect detection model to obtain defective images and non-defective images; The defective image is taken as the target image and the defect-free image is taken as the original image.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images