Image generation method and device and computer equipment

Abstract

The embodiment of the invention provides an image generation method and device and computer equipment, and the method comprises the steps: obtaining a first image based on the zooming processing of an original image; performing down-sampling on the first image for multiple times by using a multi-scale feature fusion matting model to obtain a second image and a third image; extracting a first feature of the first image under the plurality of layers, a second feature of the second image under the plurality of layers and a third feature of the third image under the plurality of layers; performing feature fusion on the first feature, the second feature and the third feature corresponding to each of the plurality of hierarchies to obtain a multi-scale feature of each hierarchy; performing feature fusion on the multi-scale features of each level to generate a mask pattern; and generating a target image based on the original image, the mask image and a preset background image. The image matting quality and the combination effect of the target image can be improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and relates to image processing technology, and in particular to an image generation method, device and computer equipment. Background Art

[0002] With the continuous development of image content recognition technology, it is now possible to perform a variety of image processing tasks on images to obtain various information from images. For example, a variety of image processing tasks may include: text recognition of text in images, identification of objects in images, clipping of objects in images, or other types of image processing tasks, etc.

[0003] After completing various image processing tasks on an image, the user can edit or extract the information obtained from the image. For example, the user can perform a cutout process on the foreground of an image to replace the background image. However, there are complex scene elements such as multiple subjects in the image. Current cutout methods often have multiple cutouts or omissions, and cannot accurately obtain the scene elements required by the user, resulting in poor quality of the generated cutout, which in turn leads to poor quality of the newly synthesized image. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, apparatus and computer device, which can solve the technical problem of poor cutout effect.

[0005] An embodiment of the present application provides an image generation method, including: obtaining a first image based on a scaling process of an original image; performing multiple downsampling on the first image using a multi-scale feature fusion cutout model to obtain a second image and a third image; extracting first features of the first image at multiple levels, second features of the second image at the multiple levels, and third features of the third image at the multiple levels; performing feature fusion on the first features, second features, and third features corresponding to each level in the multiple levels to obtain multi-scale features of each level; performing feature fusion on the multi-scale features of each level to generate a mask image; and generating a target image based on the original image, the mask image, and a preset background image.

[0006] In some optional embodiments of the present application, the extracting of the first features of the first image at multiple levels, the second features of the second image at the multiple levels, and the third features of the third image at the multiple levels includes: extracting the first sub-features of the first level, the second sub-features of the second level, the third sub-features of the third level, and the fourth sub-features of the fourth level of the first image; extracting the first sub-features of the second level, the second sub-features of the third level, and the fourth sub-features of the fourth level of the second image; extracting the first sub-features of the first level, the second sub-features of the second level, the third sub-features of the third level, and the fourth sub-features of the fourth level of the third image.

[0007] In some optional embodiments of the present application, the performing feature fusion on the first feature, the second feature, and the third feature corresponding to each level in the multiple levels to obtain the multi-scale feature of each level includes: performing feature fusion on the first sub-feature of the first image, the first sub-feature of the second image, and the first sub-feature of the third image to obtain the first scale feature corresponding to the first level; performing feature fusion on the second sub-feature of the first image, the second sub-feature of the second image, and the second sub-feature of the third image to obtain the second scale feature corresponding to the second level; performing feature fusion on the third sub-feature of the first image, the third sub-feature of the second image, and the third sub-feature of the third image to obtain the third scale feature corresponding to the third level; performing feature fusion on the fourth sub-feature of the first image, the fourth sub-feature of the second image, and the fourth sub-feature of the third image to obtain the fourth scale feature corresponding to the fourth level.

[0008] In some optional embodiments of the present application, the fusing the multi-scale features of each level to generate a mask map includes: fusing the fourth scale feature with the third scale feature to generate a first intermediate result; fusing the first intermediate result with the second scale feature to obtain a second intermediate result; and fusing the second intermediate result with the first scale feature to obtain the mask map.

[0009] In some optional embodiments of the present application, the method also includes training the cutout model, including: iteratively training the cutout model according to a first training sample to obtain multiple intermediate models generated during the iterative training process; based on multiple preset product images, using the multiple intermediate models to generate a second training sample and sample labels corresponding to the second training sample; training the cutout model according to the second training sample to obtain a predicted value; calculating the loss value of the cutout model based on the predicted value and the sample label; if the loss value is less than a preset loss threshold, completing the training of the cutout model.

[0010] In some optional embodiments of the present application, based on a plurality of preset product images, generating a second training sample and a sample label corresponding to the second training sample using the plurality of intermediate models includes: processing the plurality of preset product images using the plurality of intermediate models to obtain a cutout result; and obtaining the second training sample and the sample label based on the cutout result.

[0011] In some optional embodiments of the present application, the multiple preset product images are processed using the multiple intermediate models to obtain a cutout result, including: based on a preset scaling ratio, obtaining a product scale image of each preset product image at different scales; and using each intermediate model to obtain a cutout result corresponding to each product scale image.

[0012] In some optional embodiments of the present application, the training sample and the sample label are obtained based on the cutout result, including: calculating the similarity of the cutout results output by any two intermediate models; if there is any similarity greater than a preset similarity threshold, determining the two cutout results corresponding to any similarity, and taking the preset product image corresponding to any one of the two cutout results as a training sub-sample; based on multiple training sub-samples, obtaining the second training sample and the sample label.

[0013] The present application also provides an image generating device, comprising: an image preprocessing module, used to obtain a first image based on scaling processing of an original image; a downsampling module, used to perform multiple downsampling on the first image using a multi-scale feature fusion cutout model to obtain a second image and a third image; an extraction module, used to extract first features of the first image at multiple levels, second features of the second image at the multiple levels, and third features of the third image at the multiple levels; a fusion module, used to perform feature fusion on the first features, second features, and third features corresponding to each level in the multiple levels to obtain multi-scale features of each level; the fusion module is also used to perform feature fusion on the multi-scale features of each level to generate a mask image; an image generating module, used to generate a target image based on the original image, the mask image, and a preset background image.

[0014] An embodiment of the present application further provides a computer device, comprising: a memory, and a processor, wherein the processor executes computer-readable instructions stored in the memory to implement the image generating method.

[0015] The image generation method provided in the embodiment of the present application performs scaling processing on the original image to obtain a first image. The first image is downsampled multiple times through a cutout model to obtain a second image and a third image. The first feature of the first image at multiple levels, the second feature of the second image at the multiple levels, and the third feature of the third image at the multiple levels are extracted; the first feature, the second feature, and the third feature corresponding to each level in the multiple levels are feature fused to obtain multi-scale features of each level; by fusing the features of each level, feature information can be enriched and background residue can be reduced, thereby improving the cutout accuracy and computational efficiency. Feature fusion is performed on the multi-scale features of each level to generate a mask map. By fusing the features of multiple levels, the probability of information loss is reduced and the changes in different scales are adapted to improve the cutout quality. Finally, based on the original image, the mask map, and the preset background map, a target map is generated, which can achieve the purpose of high-precision segmentation of the foreground map, and further improve the combination effect of the target map. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is an application scenario diagram of the image generation method provided in one embodiment of the present application.

[0018] Figure 2 It is a flowchart of an image generation method provided by an embodiment of the present application.

[0019] Figure 3 This is a schematic diagram of generating multi-scale features provided by another embodiment of the present application.

[0020] Figure 4 It is a schematic diagram of generating the first intermediate result provided in an embodiment of the present application.

[0021] Figure 5 It is a schematic diagram of generating a mask image provided in an embodiment of the present application.

[0022] Figure 6 It is a cutout schematic diagram provided in an embodiment of the present application.

[0023] Figure 7 It is another cutout schematic diagram provided in an embodiment of the present application.

[0024] Figure 8 This is another cutout schematic diagram provided in an embodiment of the present application.

[0025] Fig. 9 This is a training flowchart of the cutout model provided in one embodiment of the present application.

[0026] Fig.10 It is a schematic diagram of an image generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0029] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0030] With the continuous development of image content recognition technology, it is now possible to perform a variety of image processing tasks on images to obtain various information from images. For example, a variety of image processing tasks may include: text recognition of text in images, identification of objects in images, clipping of objects in images, or other types of image processing tasks, etc.

[0031] After completing various image processing tasks on the image, the user can edit or extract the information obtained from the image. For example, the user can perform cutout processing on the foreground in the image to achieve the purpose of replacing the background image. However, there are complex scene elements such as multiple subjects in the image. The current cutout methods often have multiple cutouts or omissions, and cannot accurately obtain the scene elements required by the user, resulting in poor quality of the generated cutouts, and then poor quality of the newly synthesized image. Especially in e-commerce platforms, exquisite product renderings can improve the grade and visual effects of the products, which is conducive to promoting product transactions. However, the original product image will have more complex scene elements such as subjects, resulting in poor quality of the synthesized product image, such as regenerating a new product image after cutting out the image of the target subject such as clothing and models from the original product image.

[0032] Figure 1 It is an application scenario diagram of the image generation method provided by an embodiment of the present application. The image generation method provided by the embodiment of the present application is applied to a computer device 10, and a user can access an online store 210 running in an e-commerce platform 20 through the computer device 10, thereby obtaining a product map in the online store 210. Among them, the e-commerce platform 20 can be implemented as a cloud computing service, software as a service (SaaS), infrastructure as a service (IaaS), platform as a service (PaaS), desktop as a service (DaaS), management software as a service (MSaaS), mobile backend as a service (MBaaS), information technology management as a service (ITMaaS), etc. In some embodiments, the elements of the e-commerce platform 20 can be implemented to operate on various platforms and operating systems, such as iOS, Android, on the Web, etc. (for example, the administrator 220 is implemented in multiple instances of a given online store for iOS, Android, and for the Web, each instance having similar functions).

[0033] The e-commerce platform 20 may provide a centralized system for providing merchants with online resources and facilities for managing their businesses. The facilities described herein may be deployed in part or in whole by a machine executing computer software, modules, program codes, and / or instructions on one or more processors that may be part of the e-commerce platform 20 or external to the e-commerce platform 20. Merchants may utilize the e-commerce platform 20 to manage commerce with consumers, such as by interacting with consumers via communications 230 of the e-commerce platform 20, or any combination thereof.

[0034] The online store 210 may represent a multi-tenant facility including multiple virtual storefronts. In an embodiment, a merchant may manage one or more storefronts in the online store 210, such as through a computer device 10 (e.g., a laptop computer, a mobile computing device, etc.). The merchant may upload images (e.g., images of products), videos, content, data, etc. to the e-commerce platform 20, such as for storage by the system (e.g., as data 240).

[0035] In addition, the e-commerce platform 20 can be configured with a business management engine 250 for content management, task automation, and data management to enable support and services for multiple online stores 210 (e.g., standalone stores). The business management engine 250 includes basic or "core" functions of the e-commerce platform 20 (e.g., common to most online store activities, such as across channels, administrator interfaces, merchant locations, industries, product types, etc.), which can be reused across online stores 210 (e.g., functions that can be reused / modified across core functions). In one example, the business management engine 250 can be used to extract the original image of each product in the online store 210 and store the original image in association with the data 240.

[0036] A cutout model may be deployed on the computer device 10. The computer device 10 may retrieve data in the data 240 to obtain original images corresponding to one or more stores in the online store 210, and update the original images to target images through the cutout model.

[0037] Please continue reading Figure 1 The computer device 10 may include, but is not limited to, a communication module 110, a memory 120, a processor 130, an input / output (I / O) interface 140, and a bus. The processor 130 is coupled to the communication module 110, the memory 120, and the I / O interface 140 through the bus.

[0038] Those skilled in the art will appreciate that the schematic diagram is merely an example of the computer device 10 and does not constitute a limitation on the computer device 10 , which may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the computer device 10 may also include a network access device, etc.

[0039] The communication module 110 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB), Controller Area Network (CAN), etc. The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0040] The memory 120 can be used to store computer-readable instructions and / or modules. The processor 130 implements various functions of the computer device 10 by running or executing the computer-readable instructions and / or modules stored in the memory 120 and calling the data stored in the memory 120. The memory 120 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the computer device 10, etc. The memory 120 can include non-volatile and volatile memories, such as: a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other storage devices.

[0041] The memory 120 may be an external memory and / or an internal memory of the computer device 10. Furthermore, the memory 120 may be a memory in a physical form, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0042] The processor 130 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 130 is the computing core and control center of the computer device 10, and uses various interfaces and lines to connect various parts of the entire computer device 10, and execute the operating system of the computer device 10 and various installed application programs, program codes, etc.

[0043] The I / O interface 140 is used to provide a channel for user input or output. For example, the I / O interface 140 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0044] The bus is at least used to provide a channel for mutual communication among the communication module 110 , the memory 120 , the processor 130 , and the I / O interface 140 in the computer device 10 .

[0045] It is understandable that the application scenarios illustrated in the embodiments of the present application do not constitute specific limitations. In some embodiments of the present application, the computer device 10 can also obtain the original image of the commodity from other electronic devices, and the present application does not limit this.

[0046] Figure 2 is a flow chart of an image generation method provided by an embodiment of the present application, such as Figure 2 As shown, the image generation method provided in the embodiment of the present application is applied to a computer device (such as Figure 1 According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. Figure 2 As shown, the following steps are included:

[0047] Step S201, obtaining a first image based on scaling processing of the original image.

[0048] In some embodiments of the present application, the original image may be a product image, and the product image may be a product image used by the product to be put on the shelf. Such product images may be stored in a storage medium of a computer device, or may be obtained from other storage devices. The present application does not limit the source of the product image used by the product to be put on the shelf.

[0049] The product image can also be the product image used for the products that have been put on the shelves. This type of product image can be obtained after accessing the store through an electronic device, where the store can be a store running in an independent station. The merchant can convert the independent station into a mobile application (Application, App), and the user can install the App on the mobile phone and access the store by clicking the App icon. When the APP runs in the foreground of the computer device, the computer device can display the application interface of the APP on the display screen, that is, the store page of the independent station store. Users can interact with the APP through the controls in the application interface of the APP. In addition, the merchant can also maintain the website form of the independent station, and users can visit the store by entering the URL or clicking the bookmark through the mobile browser.

[0050] In some embodiments of the present application, the original image is processed according to a preset scaling size, and the scaling size can be set according to actual needs, and the present application does not limit this. For example, the original image can be scaled to 1280*1280 to obtain a first image with a picture size of 1280*1280.

[0051] Step S202: Downsample the first image multiple times using a multi-scale feature fusion cutout model to obtain a second image and a third image.

[0052] In some embodiments of the present application, the cutout model may be a model composed of one or more neural networks selected from the group consisting of a convolutional neural network (CNN), a fully convolutional network (FCN), and a generative adversarial network (GAN). The training process of the cutout model may refer to the following: Fig. 9 The embodiment shown.

[0053] In some embodiments of the present application, the first image is downsampled multiple times using a multi-scale feature fusion cutout model, including: downsampling the first image by 2 times using the cutout model to obtain a second image downsampled by 2 times. Downsampling the first image by 4 times using the cutout model to obtain a third image downsampled by 4 times.

[0054] Step S203: extracting first features of the first image at multiple levels, second features of the second image at the multiple levels, and third features of the third image at the multiple levels.

[0055] In some embodiments of the present application, the cutout model includes an image encoder, and a residual network (ResNet), VIT, Swin-Transformer, etc. can be used as an image encoder. Among them, ResNet is a deep neural network structure that solves the gradient vanishing problem in the deep network by introducing residual connections. In the cutout model, ResNet can be used as an image encoder to extract deep features of the first image, and these deep features can be used to distinguish the foreground and background. VIT introduces the Transformer architecture from the field of natural language processing to computer vision for processing image data. By dividing the image into a series of fixed-size patches, the self-attention mechanism of the Transformer is used to capture the global relationship between the image patches. This structure enables VIT to effectively handle long-distance dependencies in the cutout model, thereby more accurately separating the foreground and background. Swin-Transformer is a new Transformer architecture that introduces a self-attention mechanism based on a moving window and adopts a hierarchical feature expression method, so that the cutout model achieves a balance between computational complexity and performance. In the cutout model, Swin-Transformer can use its powerful global feature capture capability to improve the accuracy of cutout.

[0056] In some embodiments of the present application, extracting a first feature of a first image at multiple levels using an image encoder includes: extracting multi-scale features of the first image at multiple levels using an image encoder, and the different scale features include: a first sub-feature of the first image at the first level, a second sub-feature at the second level, a third sub-feature at the third level, and a fourth sub-feature at the fourth level. Among them, multi-scale features refer to feature information extracted at different levels or different scales. These features not only include changes in spatial dimensions, but may also include the richness of semantic information.

[0057] An image encoder is used to extract multi-scale features of the second image at multiple levels, and the different scale features respectively include: second features of the second image at multiple levels, including: using the image encoder to extract a first sub-feature of the second image at the first level, a second sub-feature at the second level, a third sub-feature at the third level, and a fourth sub-feature at the fourth level.

[0058] An image encoder is used to extract multi-scale features of the third image at multiple levels, and the different-scale features respectively include: a third feature of the third image at the multiple levels, including: using the image encoder to extract a first sub-feature of the third image at the first level, a second sub-feature at the second level, a third sub-feature at the third level, and a fourth sub-feature at the fourth level.

[0059] Step S204: performing feature fusion on the first feature, the second feature, and the third feature corresponding to each level in the multiple levels to obtain a multi-scale feature of each level.

[0060] In some embodiments of the present application, the cutout model includes a first feature fusion module for fusing features at multiple different scales.

[0061] In some embodiments of the present application, a first feature fusion module is used to perform feature fusion on the first sub-feature of the first image, the first sub-feature of the second image, and the first sub-feature of the third image to obtain a first scale feature corresponding to the first level.

[0062] A first feature fusion module is used to perform feature fusion on the second sub-feature of the first image, the second sub-feature of the second image, and the second sub-feature of the third image to obtain a second scale feature corresponding to the second level.

[0063] The first feature fusion module is used to perform feature fusion on the third sub-feature of the first image, the third sub-feature of the second image, and the third sub-feature of the third image to obtain a third scale feature corresponding to the third level.

[0064] A first feature fusion module is used to perform feature fusion on the fourth sub-feature of the first image, the fourth sub-feature of the second image, and the fourth sub-feature of the third image to obtain a fourth scale feature corresponding to the fourth level.

[0065] In some embodiments of the present application, Figure 3 The structure of the first feature fusion module shown in FIG. 1 is used to describe the process of obtaining the multi-scale features of each level. Figure 3 As shown, a convolution kernel with a size of 1×1 is used to perform convolution processing on the first sub-feature of the first image to obtain a first convolution result. Batch normalization is used to perform batch normalization on the first convolution result to obtain a first normalized result. The activation function ReLU is used to learn a more complex and nonlinear feature representation of the first normalized result to obtain a first feature to be spliced.

[0066] The first sub-feature of the second image is subjected to 2× upsampling interpolation (2×Upsampling) processing to obtain the first upsampled feature. The first upsampled feature is convolved with a convolution kernel of size 1×1 to obtain a second convolution result. The second convolution result is batch normalized using batch normalization to obtain a second normalized result. The activation function ReLU is used to learn a more complex and nonlinear feature representation of the second normalized result to obtain the second feature to be spliced.

[0067] The first sub-feature of the third image is upsampled and interpolated by 4 times to obtain a second upsampled feature. The second upsampled feature is convolved with a convolution kernel of 1×1 to obtain a third convolution result. The third convolution result is batch normalized by batch normalization to obtain a third normalized result. The activation function ReLU is used to learn a more complex and nonlinear feature representation of the third normalized result to obtain a third feature to be spliced.

[0068] The first feature to be spliced, the second feature to be spliced, and the third feature to be spliced ​​are spliced ​​in the channel dimension to obtain a first spliced ​​feature. The first spliced ​​feature is convolved with a convolution kernel of size 1×1 to obtain a fourth convolution result. The fourth convolution result is batch normalized using batch normalization to obtain a fourth normalized result. The activation function ReLU is used to learn a more complex and nonlinear feature representation of the fourth normalized result to obtain a first scale feature.

[0069] The above is to use the first feature fusion module to perform feature fusion on the first sub-feature of the first image, the first sub-feature of the second image, and the first sub-feature of the third image to obtain the first scale feature corresponding to the first level. In addition, the process of generating the second scale feature, the third scale feature, and the fourth scale feature is similar to the process of generating the first size feature, and will not be described again here. Figure 3 The structure shown generates second scale features, third scale features, and fourth scale features.

[0070] In the above embodiment, by fusing the features of each level, it is possible to enrich the feature information and reduce the background residue, thereby improving the cutout accuracy and calculation efficiency.

[0071] Step S205 , performing feature fusion on the multi-scale features of each level to generate a mask image.

[0072] In some embodiments of the present application, the cutout model includes a second feature fusion module for feature fusion. The second feature fusion module is used to perform feature fusion on the multi-scale features of each level, including: using the second feature fusion module to fuse the fourth scale feature with the third scale feature to generate a first intermediate result. Using the second feature fusion module to fuse the first intermediate result with the second scale feature to obtain a second intermediate result. Using the second feature fusion module to fuse the second intermediate result with the first scale feature to obtain the mask image. Figure 4 The structure of the second feature fusion module shown is described.

[0073] like Figure 4 As shown, a convolution kernel with a size of 1×1 is used to convolve the third scale feature to obtain a fifth convolution result. Batch normalization is used to batch normalize the fifth convolution result to obtain a fifth normalized result. The activation function ReLU is used to learn a more complex and nonlinear feature representation of the fifth normalized result to obtain the fourth feature to be spliced.

[0074] The fourth scale feature is upsampled and interpolated by 2 times to obtain the third upsampled feature. The fourth scale feature is convolved with a convolution kernel of 1×1 to obtain the sixth convolution result. The sixth convolution result is batch normalized by batch normalization to obtain the sixth normalized result. The activation function ReLU is used to learn a more complex and nonlinear feature representation of the sixth normalized result to obtain the fifth feature to be spliced.

[0075] The fourth feature to be spliced ​​and the fifth feature to be spliced ​​are spliced ​​in the channel dimension to obtain a second spliced ​​feature. The second spliced ​​feature is convolved with a convolution kernel of size 1×1 to obtain a seventh convolution result. The seventh convolution result is batch normalized using batch normalization to obtain a seventh normalized result. The activation function ReLU is used to learn a more complex and nonlinear feature representation of the seventh normalized result to obtain a first intermediate result.

[0076] The above is a process of using the second feature fusion module to fuse the fourth scale feature with the third scale feature to generate the first intermediate result. In addition, the process of generating the second intermediate result and the mask image is similar to the process of generating the first intermediate result, which will not be described again here. Figure 4 The structure shown generates a second intermediate result and a mask image.

[0077] In order to better understand the process of generating mask images, Figure 5 Provide explanation.

[0078] like Figure 5 As shown, the first sub-feature 10, the second sub-feature 20, the third sub-feature 30 and the fourth sub-feature 40 of the first image are extracted, the first sub-feature 11, the second sub-feature 21, the third sub-feature 31 and the fourth sub-feature 41 of the second image are extracted, and the first sub-feature 12, the second sub-feature 22, the third sub-feature 32 and the fourth sub-feature 42 of the third image are extracted. The first feature fusion module is used to perform feature fusion on the first feature, the second feature and the third feature corresponding to each level, including: using the first feature fusion module to fuse the first sub-feature 10, the first sub-feature 11 and the first sub-feature 12 of the first level to obtain the first scale feature corresponding to the first level; using the first feature fusion module to fuse the second sub-feature 20, the second sub-feature 21 and the second sub-feature 22 of the second level to obtain the second scale feature corresponding to the second level; using the first feature fusion module to fuse the third sub-feature 30, the third sub-feature 31 and the third sub-feature 32 of the third level to obtain the third scale feature corresponding to the third level; using the first feature fusion module to fuse the fourth sub-feature 40, the fourth sub-feature 41 and the fourth sub-feature 42 of the fourth level to obtain the fourth scale feature corresponding to the fourth level.

[0079] Next, the second feature fusion module is used to fuse the features at each level, including: using the second feature fusion module to fuse the fourth size feature with the third scale feature to obtain a first intermediate result. Using the second feature fusion module to fuse the first intermediate result with the second scale feature to obtain a second intermediate result. Using the second feature fusion module to fuse the second intermediate result with the first scale feature to obtain a mask image.

[0080] In the above embodiment, by fusing multi-level features, the probability of information loss is reduced and the image cutout quality is improved by adapting to changes in different scales.

[0081] Step S206, generating a target image based on the original image, the mask image and the preset background image.

[0082] In some embodiments of the present application, after obtaining the mask image, the size of the mask image is adjusted to the size of the original image. A preset background image is obtained from a computer device, and the size of the preset background image is the same as the size of the original image. Figure 1 The preset background image may be a solid color image, for example, an image whose pixel values ​​are all 0, an image whose pixel values ​​are all 255, etc. The present application does not limit the type of the preset background image.

[0083] Based on the original image, the mask image and the preset background image, a target image is generated, which is expressed by the following formula:

[0084] C = αF + (1-α) B;

[0085] In the formula, C represents the target image, α represents the mask image, F represents the original image, and B represents the preset background image.

[0086] In the above embodiment, the original image is scaled to obtain a first image. The first image is downsampled multiple times through the cutout model to obtain a second image and a third image. The first feature of the first image at multiple levels, the second feature of the second image at the multiple levels, and the third feature of the third image at the multiple levels are extracted; the first feature, the second feature, and the third feature corresponding to each level in the multiple levels are fused to obtain multi-scale features of each level; by fusing the features of each level, feature information can be enriched and background residue can be reduced, thereby improving the cutout accuracy and computational efficiency. The multi-scale features of each level are fused to generate a mask image, and by fusing the features of multiple levels, the probability of information loss is reduced and the scale changes can be adapted to improve the cutout quality. Finally, based on the original image, the mask image, and the preset background image, a target image is generated, which can achieve the purpose of high-precision segmentation of the foreground image and further enhance the combination effect of the target image. The following is combined with Figure 6 , Figure 7 and Figure 8 The advantages of the target image shown in the embodiment of the present application are described compared with the cutout results generated by the related art.

[0087] like Figure 6 As shown in the figure, since the product image is quite special and contains many scene elements, the use of the cutout model is not just for segmenting a single object. Figure 1 . The target generated by the relevant cutout model Figure 1 The obtained area is the human figure, and the stool is not recognized, which leads to the target Figure 1 The foreground elements included are incomplete. The target image generated by the cutout model provided in the embodiment of the present application Figure 2 Can buckle the characters and stools well, target Figure 2 Ability to contain complete foreground elements.

[0088] like Figure 7 As shown, the target generated by the relevant cutout model Figure 1 The foreground element earrings can be included, but the buckled earrings are missing, resulting in the target Figure 1 The effect shown is not good. The target image generated by the cutout model provided in the embodiment of the present application Figure 2 Able to buckle the characters and earrings well, target Figure 2 Ability to contain complete foreground elements.

[0089] like Figure 8As shown, the relevant cutout model is used to generate the target Figure 1 ,Target Figure 1 Including the characters, but there are multiple buttons on the hands, resulting in the target Figure 1 The display effect of the hand details is not good. The target image generated by the cutout model provided in the embodiment of the present application Figure 2 Able to handle character details very well.

[0090] Fig. 9 is a training flow chart of a cutout model provided in an embodiment of the present application, such as Fig. 9 As shown, training the cutout model includes the following steps.

[0091] Step S901, iteratively training the cutout model according to the first training sample to obtain multiple intermediate models generated during the iterative training process.

[0092] In some embodiments of the present application, the first training sample may be a public dataset used by the matting model, such as MAGICK, Adobe Image Matting Dataset, etc., wherein the MAGICK dataset covers objects with various details. The Adobe Image Matting Dataset dataset contains images of portraits and images without portraits. The above are just examples, and other public datasets may also be used as the first training sample.

[0093] In some embodiments of the present application, the number of iterations can be preset, and the cutout model is iteratively trained using the first training sample based on the number of iterations. Multiple intermediate models generated during the iterative training process are obtained, wherein the number of intermediate models can be set according to actual conditions, and the present application does not limit this. For example, the number of intermediate models can be 3.

[0094] In one example, the number of iterations is 100, and multiple intermediate models can be randomly selected in the 100 iterations of training. For example, the models generated at the 20th, 59th, and 88th times during the iteration training process can be selected as intermediate models.

[0095] In other embodiments of the present application, a preset loss value range can be used to select multiple intermediate models from the iterative training process.

[0096] In one example, the cutout model is trained using the first training sample, and the loss value calculated after the forward propagation is completed in each iterative training process is calculated. If the loss value is within the loss value range, the current cutout model is classified as a candidate model. Then, based on a preset number, multiple intermediate models are selected from the candidate models.

[0097] In other embodiments of the present application, commonly used cutout models can also be used as intermediate models. For example, in cutout tasks, commonly used models include deep learning models, such as U-type convolutional neural network (U-Net), deep convolutional neural network (DeepLab), masked regional convolutional neural network (Mask R-CNN), etc.

[0098] Step S902: Based on a plurality of preset product images, the plurality of intermediate models are used to generate second training samples and sample labels corresponding to the second training samples.

[0099] In some embodiments of the present application, the preset product image can be collected by a computer device by accessing a store running on an independent station, and the present application does not limit the source of the preset product image. Input each preset product image into each intermediate model to obtain the cutout result output by each intermediate model, including: adjusting each preset product image according to a preset scaling ratio to obtain a product scale image of each preset product image at different scales. For example, each preset product image is adjusted according to scales of 1280, 1024, 896, and 768 to obtain 4 product scale images respectively. Using each intermediate model, the cutout result corresponding to each product scale image is obtained. For example, a product scale image with a scale of 1280 is input into each intermediate model respectively, and the cutout result after each intermediate model processes the product scale image with a scale of 1280 is obtained.

[0100] In some embodiments of the present application, after obtaining the cutout result, a second training sample and a corresponding sample label are obtained based on the cutout result, including: uniformly adjusting the size of the cutout result so that each cutout result has the same size. Since each cutout result has the same size, a preset similarity algorithm can be used to calculate the similarity of the cutout results output by any two intermediate models. Among them, the similarity algorithm may include but is not limited to a pixel comparison method, a histogram comparison method, a deep learning method, a feature point matching method, etc. The similarity can be the SSIM value of the two cutout results. The closer the SSIM value is to 1, the more similar the two cutout results are.

[0101] If there is any similarity greater than a preset similarity threshold, two cutout results corresponding to any similarity are determined, and a preset product image corresponding to any one of the two cutout results is used as a training subsample. In one example, the first cutout result is generated by processing the first product image based on the first intermediate model, and the second cutout result is generated by processing the second product image based on the first intermediate model. If the similarity between the first cutout result and the second cutout result is greater than the preset similarity, a product image can be randomly selected between the first product image and the second product image as a training subsample.

[0102] In another example, the first cutout result is generated by processing the first product image based on the first intermediate model, and the second cutout result is generated by processing the second product image based on the second intermediate model. If the similarity between the first cutout result and the second cutout result is greater than a preset similarity, a product image between the first product image and the second product image can be randomly selected as a training subsample.

[0103] Based on the above similarity comparison method, the obtained multiple training sub-samples are used as second training samples and corresponding sample labels, and about 300,000 high-quality product images can be obtained from 1 million preset product images as second training samples.

[0104] In the above embodiment, by processing the preset product images through multiple intermediate models and determining the expressiveness of the preset product images in the cutout model by means of similarity, higher quality product images can be effectively screened out from the multiple preset product images as the second training samples.

[0105] Step S903: training the cutout model according to the second training sample to obtain a prediction value.

[0106] In some embodiments of the present application, after the second training sample is determined, the second training sample can be used to train the cutout model to obtain the predicted value of each forward propagation output.

[0107] Step S904: Calculate the loss value of the cutout model based on the predicted value and the sample label.

[0108] In some embodiments of the present application, based on the predicted value and the sample label, the loss value of the cutout model is calculated, including binary cross entropy loss, IoU loss, and structural similarity loss (SSIM), which is expressed by the following formula:

[0109] L=λ1L BCE +λ2L IOU +λ3L SSIM ,λ1=20,λ2=1,λ3=10;

[0110] In the formula, L represents the loss value of the cutout model, L BCE represents the binary cross entropy loss, L IOU represents the IoU loss, L SSIM represents the structural similarity loss, and λ1, λ2, and λ3 are all weight parameters.

[0111] Step S905: If the loss value is less than the preset loss threshold, the training of the cutout model is completed.

[0112] In some embodiments of the present application, during the iterative training process, it is possible to determine whether the cutout model has reached a convergence state by comparing the loss value and the preset loss threshold. If the loss value is less than the preset loss threshold, it can be determined that the cutout model has reached a convergence state and the training of the cutout model is completed. If the loss value is greater than or equal to the preset loss threshold, the back propagation process can be continued to adjust the parameters of the cutout model and re-execute the forward propagation process.

[0113] In the above embodiment, the expressiveness of multiple preset product images in the cutout task is evaluated through multiple intermediate models, so as to select multiple preset product images with higher quality as the second training samples. The cutout model is trained with the second training samples, which improves the robustness of the cutout model to a certain extent.

[0114] See also Fig.10 , is a schematic diagram of an image generating device provided in an embodiment of the present application. It is a functional embodiment of the image generation method of the present application. The image generation device includes an image preprocessing module 1001, a downsampling module 1002, an extraction module 1003, a fusion module 1004 and an image generation module 1005, wherein: the image preprocessing module 1001 is used to obtain a first image based on a scaling process of the original image; the downsampling module 1002 is used to perform multiple downsampling on the first image using a matting model of multi-scale feature fusion to obtain a second image and a third image; the extraction module 1003 is used to extract the first feature of the first image at multiple levels, the second feature of the second image at the multiple levels and the third feature of the third image at the multiple levels; the fusion module 1004 is used to perform feature fusion on the first feature, the second feature and the third feature corresponding to each level in the multiple levels to obtain the multi-scale feature of each level; the fusion module 1004 is also used to perform feature fusion on the multi-scale features of each level to generate a mask image; the image generation module 1005 is used to generate a target image based on the original image, the mask image and the preset background image.

[0115] In some optional embodiments of the present application, the extracting of the first features of the first image at multiple levels, the second features of the second image at the multiple levels, and the third features of the third image at the multiple levels includes: extracting the first sub-features of the first level, the second sub-features of the second level, the third sub-features of the third level, and the fourth sub-features of the fourth level of the first image; extracting the first sub-features of the second level, the second sub-features of the third level, and the fourth sub-features of the fourth level of the second image; extracting the first sub-features of the first level, the second sub-features of the second level, the third sub-features of the third level, and the fourth sub-features of the fourth level of the third image.

[0116] In some optional embodiments of the present application, the performing feature fusion on the first feature, the second feature, and the third feature corresponding to each level in the multiple levels to obtain the multi-scale feature of each level includes: performing feature fusion on the first sub-feature of the first image, the first sub-feature of the second image, and the first sub-feature of the third image to obtain the first scale feature corresponding to the first level; performing feature fusion on the second sub-feature of the first image, the second sub-feature of the second image, and the second sub-feature of the third image to obtain the second scale feature corresponding to the second level; performing feature fusion on the third sub-feature of the first image, the third sub-feature of the second image, and the third sub-feature of the third image to obtain the third scale feature corresponding to the third level; performing feature fusion on the fourth sub-feature of the first image, the fourth sub-feature of the second image, and the fourth sub-feature of the third image to obtain the fourth scale feature corresponding to the fourth level.

[0117] In some optional embodiments of the present application, the fusing the multi-scale features of each level to generate a mask map includes: fusing the fourth scale feature with the third scale feature to generate a first intermediate result; fusing the first intermediate result with the second scale feature to obtain a second intermediate result; and fusing the second intermediate result with the first scale feature to obtain the mask map.

[0118] In some optional embodiments of the present application, it also includes: training the cutout model, including: iteratively training the cutout model according to a first training sample to obtain multiple intermediate models generated during the iterative training process; based on multiple preset product images, using the multiple intermediate models to generate a second training sample and a sample label corresponding to the second training sample; training the cutout model according to the second training sample to obtain a predicted value; calculating the loss value of the cutout model based on the predicted value and the sample label; if the loss value is less than a preset loss threshold, completing the training of the cutout model.

[0119] In some optional embodiments of the present application, based on a plurality of preset product images, generating a second training sample and a sample label corresponding to the second training sample using the plurality of intermediate models includes: processing the plurality of preset product images using the plurality of intermediate models to obtain a cutout result; and obtaining the second training sample and the sample label based on the cutout result.

[0120] In some optional embodiments of the present application, the multiple preset product images are processed using the multiple intermediate models to obtain a cutout result, including: based on a preset scaling ratio, obtaining a product scale image of each preset product image at different scales; and using each intermediate model to obtain a cutout result corresponding to each product scale image.

[0121] In some optional embodiments of the present application, the training sample and the sample label are obtained based on the cutout result, including: calculating the similarity of the cutout results output by any two intermediate models; if there is any similarity greater than a preset similarity threshold, determining the two cutout results corresponding to any similarity, and taking the preset product image corresponding to any one of the two cutout results as a training sub-sample; based on multiple training sub-samples, obtaining the second training sample and the sample label.

[0122] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0123] The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0124] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: include: Based on the scaling process of the original image, a first image is obtained; Downsampling the first image multiple times using a multi-scale feature fusion cutout model to obtain a second image and a third image; Extracting first features of the first image at multiple levels, second features of the second image at the multiple levels, and third features of the third image at the multiple levels; Performing feature fusion on the first feature, the second feature, and the third feature corresponding to each level in the multiple levels to obtain a multi-scale feature of each level; Performing feature fusion on the multi-scale features of each level to generate a mask map; A target image is generated based on the original image, the mask image and the preset background image.

2. The image generation method according to claim 1, characterized in that: The extracting the first features of the first image at multiple levels, the second features of the second image at the multiple levels, and the third features of the third image at the multiple levels comprises: Extracting a first sub-feature of the first image at a first level, a second sub-feature at a second level, a third sub-feature at a third level, and a fourth sub-feature at a fourth level; Extracting a first sub-feature of the second image at the first level, a second sub-feature at the second level, a third sub-feature at the third level, and a fourth sub-feature at the fourth level; A first sub-feature of the third image at the first level, a second sub-feature at the second level, a third sub-feature at the third level, and a fourth sub-feature at the fourth level are extracted.

3. The image generation method according to claim 2, characterized in that: The step of fusing the first feature, the second feature, and the third feature corresponding to each level in the multiple levels to obtain the multi-scale feature of each level includes: Performing feature fusion on the first sub-feature of the first image, the first sub-feature of the second image, and the first sub-feature of the third image to obtain a first scale feature corresponding to the first level; Performing feature fusion on the second sub-feature of the first image, the second sub-feature of the second image, and the second sub-feature of the third image to obtain a second scale feature corresponding to the second level; Performing feature fusion on the third sub-feature of the first image, the third sub-feature of the second image, and the third sub-feature of the third image to obtain a third scale feature corresponding to the third level; The fourth sub-feature of the first image, the fourth sub-feature of the second image, and the fourth sub-feature of the third image are subjected to feature fusion to obtain a fourth scale feature corresponding to the fourth level.

4. The image generation method according to claim 3, characterized in that: The step of fusing the multi-scale features at each level to generate a mask map includes: fusing the fourth scale feature with the third scale feature to generate a first intermediate result; fusing the first intermediate result with the second scale feature to obtain a second intermediate result; The second intermediate result is fused with the first scale feature to obtain the mask image.

5. The image generation method according to claim 1, characterized in that: The method further includes training the cutout model, including: Iteratively training the cutout model according to the first training sample to obtain a plurality of intermediate models generated during the iterative training process; Based on a plurality of preset product images, generating second training samples and sample labels corresponding to the second training samples using the plurality of intermediate models; Training the cutout model according to the second training sample to obtain a prediction value; Calculating a loss value of the cutout model based on the predicted value and the sample label; If the loss value is less than the preset loss threshold, the training of the cutout model is completed.

6. The image generation method according to claim 5, characterized in that: The step of generating second training samples and sample labels corresponding to the second training samples based on the plurality of preset product images and using the plurality of intermediate models includes: Processing the plurality of preset product images using the plurality of intermediate models to obtain a cutout result; Based on the cutout result, the second training sample and the sample label are obtained.

7. The image generation method according to claim 6, characterized in that: The using the multiple intermediate models to process the multiple preset product images to obtain a cutout result includes: Based on the preset scaling ratio, obtain the product scale map of each preset product map at different scales; Using each intermediate model, we can obtain the corresponding cutout results for each product scale image.

8. The image generation method according to claim 6, characterized in that: The obtaining the training sample and the sample label based on the cutout result includes: Calculate the similarity of the cutout results output by any two intermediate models; If there is any similarity greater than a preset similarity threshold, two cutout results corresponding to the any similarity are determined, and a preset product image corresponding to any one of the two cutout results is used as a training subsample; Based on the multiple training sub-samples, the second training sample and the sample label are obtained.

9. An image generating device, characterized in that: include: An image preprocessing module, used for obtaining a first image based on scaling processing of the original image; A downsampling module, used to perform multiple downsampling on the first image using a multi-scale feature fusion cutout model to obtain a second image and a third image; An extraction module, used for extracting first features of the first image at multiple levels, second features of the second image at the multiple levels, and third features of the third image at the multiple levels; A fusion module, used for fusing the first feature, the second feature and the third feature corresponding to each level in the multiple levels to obtain a multi-scale feature of each level; The fusion module is further used to fuse the multi-scale features of each level to generate a mask image; The image generation module is used to generate a target image based on the original image, the mask image and a preset background image.

10. A computer device, characterized in that: include: Memory; and Processor; the processor executes computer-readable instructions stored in the memory to implement the image generation method according to any one of claims 1 to 8.

Images