Layer fusion method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a layer fusion method and device, equipment and a storage medium, and is applied to the technical field of image processing. The method comprises the following steps: fusing a foreground image, a background image and a foreground mask image to obtain a first fused image; performing depth map and depth feature extraction on the image to obtain a depth feature map; image coding is carried out on the foreground image and the background image, obtained coding features are input into a generator for feature reconstruction, and a foreground feature map and a background feature map are obtained; performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fusion feature map; inputting the depth feature map, the first fusion feature map and the foreground mask map subjected to adaptive size adjustment into a generator for feature reconstruction to obtain a second fusion feature map; and carrying out image decoding on the second fusion feature map to obtain a second fusion map. In this way, a fusion image with consistent foreground and background illumination effects and reasonable foreground shadow can be obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a layer fusion method, device, equipment and storage medium. Background Art

[0002] In image editing and design, layer blending is a common operation used to blend multiple layers into a single image.

[0003] The existing layer fusion solutions are:

[0004] Solution 1: (1) The user provides a background image and one or more foreground images. The foreground image contains the corresponding foreground object segmentation mask; (2) The objects in the foreground image are placed in the corresponding positions in the background image through the segmentation mask; (3) Step (2) is repeated until all objects in the foreground image are placed in the background image and the final fusion image is obtained.

[0005] Solution 2: (1) The user provides a background image and one or more foreground images. The foreground image contains the corresponding foreground object segmentation mask; (2) The foreground object is placed in the corresponding position of the background image through the segmentation mask; (3) The fusion image of step (2) is generated by AI; (4) Steps (2) and (3) are repeated until all foreground images are integrated into the background image and a satisfactory result is obtained.

[0006] Solution 3: (1) The user provides a background image and one or more foreground images. The foreground image contains the corresponding foreground object segmentation mask; (2) The background image is input into the AI ​​model to generate background intermediate features; (3) The foreground image is input into the AI ​​model to generate foreground intermediate features; (4) The foreground intermediate features are inserted into the specific position of the background intermediate features through the scaled foreground object segmentation mask; (5) Steps (3) and (4) are repeated until all foreground intermediate features are inserted into the background intermediate features to obtain fused intermediate features; (6) The fused intermediate features are used to generate the final fused image through the AI ​​model.

[0007] Solution 4: (1) The user provides a background image and one or more foreground images. The foreground image contains the corresponding foreground object segmentation mask; (2) The background image and the foreground image are simultaneously input into the AI ​​model to obtain a fusion image; (3) Step (2) is repeated until a satisfactory result is obtained.

[0008] Problems with existing solutions:

[0009] 1. The objects and content in the fusion image are inconsistent with the content in the foreground and background (Solution 2 above);

[0010] 2. The foreground lighting effect in the fusion image is inconsistent with the background lighting effect (the above solutions 1 and 3);

[0011] 3. There is no shadow or unreasonable shadow of the foreground object in the fusion image (Solutions 1, 3, and 4 above). Summary of the invention

[0012] The present invention provides a layer fusion method, device, equipment and storage medium.

[0013] According to a first aspect of the present disclosure, a layer fusion method is provided. The method comprises:

[0014] Receiving a foreground image and a background image input by a user; performing mask extraction on the foreground image to obtain a foreground mask image;

[0015] The foreground image, the background image and the foreground mask image are fused to obtain a first fused image; a depth map is extracted from the first fused image to obtain a depth map of the first fused image; a depth feature is extracted from the depth map of the first fused image to obtain a depth feature map;

[0016] Performing image coding processing on the foreground image and the background image respectively to obtain coding features of the foreground image and coding features of the background image; inputting the coding features of the foreground image and the coding features of the background image into a generator respectively for feature reconstruction to obtain a foreground feature map after feature reconstruction and a background feature map after feature reconstruction;

[0017] Performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map;

[0018] Adaptively adjusting the foreground mask image to obtain an adaptively resized foreground mask image;

[0019] Inputting the depth feature map, the first fused feature map and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map;

[0020] Perform image decoding on the second fusion feature map to obtain a second fusion map.

[0021] According to the above aspects and any possible implementation, an implementation is further provided.

[0022] The foreground image is one or more images; the background image is one image.

[0023] According to the above aspects and any possible implementation, an implementation is further provided.

[0024] The foreground mask image is a binary image, wherein the foreground area is white and the background area is black;

[0025] The step of fusing the foreground image, the background image and the foreground mask image to obtain a first fused image includes:

[0026] Extracting an arbitrary region from the foreground image using the foreground mask image, wherein the operation of extracting the arbitrary region is achieved by performing a bitwise AND operation on the foreground mask image and the foreground image;

[0027] The extracted foreground area is fused with the corresponding area of ​​the background image to obtain a first fused image.

[0028] According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein extracting a depth map from the first fused image to obtain a depth map of the first fused image includes:

[0029] Obtaining the scene relevance of each pixel in the first fusion image;

[0030] Continuously downsampling the first fused image to obtain a scene correlation of each pixel in each downsampled image of the first fused image;

[0031] Perform block matching motion vector calculation on each pixel in the current downsampled image and the corresponding pixel in the previous downsampled image to obtain the motion vector value of each pixel in the current downsampled image;

[0032] Accumulate the motion vector value of each pixel in the current downsampled image, extract the initial depth value of each pixel from the accumulated motion vector, and the initial depth value constitutes an initial depth map;

[0033] Performing continuous super-smoothing filtering and up-sampling processing on each pixel in the initial depth map by using the scene correlation of each pixel in the first fused image and the scene correlation of each pixel in each down-sampled image, so as to obtain a depth map of the first fused image;

[0034] The scene correlation is the correlation between any pixel in a frame of image and its surrounding adjacent pixels.

[0035] According to the above aspects and any possible implementation, an implementation is further provided.

[0036] The step of performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map includes:

[0037] The foreground feature map and the background feature map are subjected to Alpha blending processing according to the foreground mask map to obtain a first fused feature map.

[0038] According to the above aspects and any possible implementation, an implementation is further provided, wherein the foreground mask image is adaptively adjusted by an adaptive mask filtering module, wherein the adaptive mask filtering module is configured with an adaptive threshold segmentation technology and an adaptive filtering technology.

[0039] According to the above aspects and any possible implementation, an implementation is further provided.

[0040] The depth feature map includes illumination information and shadow information of the depth map of the first fusion map;

[0041] The adaptively resized foreground mask image carries position information of the foreground object;

[0042] Inputting the depth feature map, the first fused feature map and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map, including:

[0043] A generator is used to generate a shadow image of a foreground object that meets the position information based on the illumination information and shadow information; and the first fused feature map is feature reconstructed according to the shadow image to obtain a second fused feature map after feature reconstruction.

[0044] According to a second aspect of the present disclosure, a layer fusion device is provided. The device comprises:

[0045] The data receiving module is used to receive a foreground image and a background image input by a user; perform mask extraction on the foreground image to obtain a foreground mask image;

[0046] a feature processing module, configured to fuse the foreground image, the background image and the foreground mask image to obtain a first fused image; perform depth map extraction on the first fused image to obtain a depth map of the first fused image; perform depth feature extraction on the depth map of the first fused image to obtain a depth feature map;

[0047] The feature processing module is further used to perform image coding processing on the foreground image and the background image respectively to obtain coding features of the foreground image and coding features of the background image; input the coding features of the foreground image and the coding features of the background image into the generator respectively to perform feature reconstruction to obtain a foreground feature map after feature reconstruction and a background feature map after feature reconstruction;

[0048] A fusion module, used for performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map;

[0049] An adaptive adjustment module, used for adaptively adjusting the foreground mask image to obtain a foreground mask image after adaptively adjusting the size;

[0050] The fusion module is further used to input the depth feature map, the first fused feature map and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map;

[0051] An image decoding module is used to perform image decoding on the second fusion feature map to obtain a second fusion map.

[0052] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0053] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0054] The present disclosure provides a layer fusion method, device, equipment and storage medium, which guides the generator to generate reasonable shadows of objects in the foreground image by introducing additional spatial position information (the foreground mask image after adaptive resizing provides the position information of the foreground object for the generator to generate a new feature map). At the same time, an adaptive mask filtering module is introduced. In the process of reconstruction, the mutual influence between the foreground and the background is dynamically considered, and the light and shadow effects are generated using the deep feature map. This method of aligning the lighting effect of the foreground object with the background lighting effect in a non-training manner improves efficiency.

[0055] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0057] Figure 1 A flowchart of a layer fusion method according to an embodiment of the present disclosure is shown;

[0058] Figure 2 A block diagram of a layer fusion method according to an embodiment of the present disclosure is shown;

[0059] Figure 3 A block diagram of a layer fusion device according to an embodiment of the present disclosure is shown;

[0060] Figure 4 A schematic block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0062] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0063] Figure 1 A flowchart of a layer fusion method 100 according to an embodiment of the present disclosure is shown. The method 100 includes:

[0064] Step 110, receiving a foreground image and a background image input by a user; performing mask extraction on the foreground image to obtain a foreground mask image.

[0065] In some embodiments, the foreground image is one or more images; the background image is one image. The foreground mask image is a binary image, in which the foreground area is white and the background area is black. Figure 2 The image of the person shown in the figure is the foreground image, the landscape image containing trees, grass, etc. is the background image, and the image with a black background and a white portrait is the foreground mask image, which is used to fuse the person into the landscape image. While ensuring that the foreground content and the background content remain unchanged in the fused image, and while keeping the background lighting effect unchanged, a reasonable shadow of the foreground object is generated.

[0066] In some embodiments, to obtain a mask from a foreground image, it is usually necessary to segment the foreground object of interest through image processing techniques. This process can be achieved through the following methods: (1) Threshold segmentation: If the foreground and background have obvious differences in brightness or color, they can be segmented through a threshold operation. Convert the image to a grayscale image, and then apply a threshold to set the foreground to white (255) and the background to black (0). This method is simple and fast, but has high requirements on the image contrast. (2) GrabCut image segmentation algorithm: GrabCut is an algorithm that segments the foreground and background based on the color information of the image. It uses an initial rectangular box to roughly specify the location of the foreground, and then improves the segmentation result through an iterative process. (3) Edge detection: Determine the outline of the foreground by detecting edges in the image. (4) Deep learning: Using a well-trained deep learning model, such as a convolutional neural network (CNN), the foreground can be segmented from a complex background very accurately. (5) Morphological operations: Use morphological operations such as erosion and dilation to refine the edges of the mask so that it fits the outline of the foreground more closely. (6) Manual creation: You can also manually create a mask by selecting an area of ​​interest in the image, such as a person, as a foreground mask. Regardless of the method, the foreground mask image must eventually be processed into a binary image.

[0067] Step 120: fuse the foreground image, the background image and the foreground mask image to obtain a first fused image; perform depth map extraction on the first fused image to obtain a depth map of the first fused image; perform depth feature extraction on the depth map of the first fused image to obtain a depth feature map.

[0068] In some embodiments, the foreground image, the background image and the foreground mask image are fused to obtain a first fused image, including: extracting an arbitrary area from the foreground image using the foreground mask image, wherein the operation of extracting the arbitrary area is realized by performing a bitwise AND operation on the foreground mask image and the foreground image; and fusing the extracted foreground area with the corresponding area of ​​the background image to obtain the first fused image. The preprocessing step is included: ensuring that the foreground image, the background image and the mask image are of the same size.

[0069] In some embodiments, the depth map extraction of the first fused image to obtain the depth map of the first fused image includes: obtaining the scene correlation of each pixel in the first fused image; continuously downsampling the first fused image to obtain the scene correlation of each pixel in each downsampled image of the first fused image; performing block matching motion vector calculation on each pixel in the current downsampled image and the corresponding pixel in the previous downsampled image to obtain the motion vector value of each pixel in the current downsampled image; accumulating the motion vector value of each pixel in the current downsampled image, extracting the initial depth value of each pixel from the accumulated sum of the motion vectors, and the initial depth value constitutes the initial depth map; using the scene correlation of each pixel in the first fused image and the scene correlation of each pixel in each downsampled image to continuously perform super-smoothing filtering and upsampling on each pixel in the initial depth map to obtain the depth map of the first fused image; wherein the scene correlation is the degree of correlation between any pixel in a frame of image and its surrounding adjacent pixels.

[0070] In some embodiments, depth maps can also be extracted through deep learning models such as Depth-AnythingV2 and U2Net ubiquitous ultra-wideband network architecture.

[0071] In some embodiments, depth features are extracted from the depth map of the first fusion map to obtain a depth feature map of the first fusion map, wherein the controlnet control network can be used to extract image features of the depth map of the first fusion map, that is, to obtain the depth feature map.

[0072] Step 130, performing image coding processing on the foreground image and the background image respectively to obtain the coding features of the foreground image and the coding features of the background image; inputting the coding features of the foreground image and the coding features of the background image into a generator respectively for feature reconstruction to obtain a foreground feature map after feature reconstruction and a background feature map after feature reconstruction.

[0073] In some embodiments, the image is encoded using a VAE variational autoencoder in stable-diffusion to encode the image from a dimension of W×H×3 to a feature of a dimension of W'×H'×C. Where W represents the width of the image, H represents the height of the image, and C represents the number of image channels.

[0074] In some embodiments, the generator uses a Unet convolutional neural network in stable-diffusion. The generator performs T-step reconstruction on the features of the foreground image and / or the background image to obtain a foreground feature image after feature reconstruction and a background feature image after feature reconstruction.

[0075] Step 140, performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map.

[0076] In some embodiments, the foreground feature map after feature reconstruction, the background feature map after feature reconstruction and the foreground mask map are subjected to feature fusion to obtain a first fused feature map, including: performing Alpha blending processing on the foreground feature map and the background feature map according to the foreground mask map to obtain a first fused feature map. It should be noted that the feature obtained by image feature extraction is the feature map. The feature map is the result of image feature extraction, which contains key information of the image and is used to assist machine learning and image recognition tasks. The feature map can be a feature map composed of features such as color histogram, texture, and shape. The formula for Alpha blending processing is as follows:

[0077] I=αF+(1-α)B

[0078] Among them, I is the pixel value of the first fusion feature (first fusion feature map), F is the pixel value of the foreground feature (foreground feature map), B is the pixel value of the background feature (background feature map), and α is the pixel value of the foreground mask map.

[0079] Step 150 , adaptively adjust the foreground mask image to obtain an adaptively resized foreground mask image.

[0080] In some embodiments, the adaptive adjustment of the foreground mask image is performed through an adaptive mask filtering module, and the adaptive mask filtering module is configured with adaptive threshold segmentation technology and adaptive filtering technology. Adaptive mask filtering module: In the process of self-attention module fusion, an adaptively resized foreground mask is introduced, so that the foreground and background contents can be kept unchanged during the reconstruction of the fusion image, and the shadow of the foreground object can be generated according to the adaptively adjusted foreground mask. The adaptive mask filtering module combines adaptive threshold segmentation and adaptive filtering technology for more refined adaptive adjustment of the image. First, the image is segmented into foreground and background using adaptive threshold segmentation. Then, the parameters of the filter are dynamically adjusted according to the characteristics of the foreground and background. Finally, the filter is applied to perform different processing on the foreground and background to achieve adaptive adjustment.

[0081] Step 160: input the depth feature map, the first fused feature map, and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map.

[0082] In some embodiments, the depth feature map includes illumination information and shadow information of the depth map of the first fused map; the adaptively resized foreground mask map carries position information of the foreground object; the depth feature map, the first fused feature map and the adaptively resized foreground mask map are input into a generator for feature reconstruction to obtain a second fused feature map, including: using the generator to generate a shadow image of the foreground object that meets the position information based on the illumination information and shadow information; and reconstructing the first fused feature map based on the shadow image to obtain a second fused feature map after feature reconstruction.

[0083] In some embodiments, the generator is used to generate a shadow image of a foreground object (e.g., a person) in combination with the image features (depth features) of the depth map of the first fusion map, i.e., the illumination information and shadow information of the image, and the adaptively resized foreground mask map, which carries the position information of the foreground object (e.g., a person). In the process of the generator using the shadow image and the first fusion feature map to generate a new feature map, the generator can also combine the adaptive mask filtering module to dynamically adjust the parameters of the filter, i.e., calculate the mean, variance and other statistics of the shadow image and the first fusion feature map, adaptively adjust the weight, size and other parameters of the filter according to these statistics, and apply the filter to filter the image, so that the position and size of the shadow of the new feature map generated, i.e., the second fusion feature map, is more reasonable. In this process, by introducing additional position information to control the generation process, on the basis of ensuring that the foreground and background content remain unchanged in the fusion map, the reasonable shadow of the foreground object is finally generated with the help of the illumination effect of the foreground background. Moreover, the alignment of the illumination effect of the foreground object with the background illumination effect is achieved in a non-training manner to improve efficiency.

[0084] Step 170: performing image decoding on the second fusion feature after feature reconstruction to obtain a second fusion image.

[0085] In some embodiments, VAE in stable-diffusion can be used to decode the features, and decode the features from the dimensions of W×H×C to an image of the dimensions of W'×H'×3 to obtain a final fusion image.

[0086] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0087] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0088] Figure 3 FIG. 3 shows a block diagram of a layer fusion device 300 according to an embodiment of the present disclosure. Figure 3 As shown, the device 300 includes:

[0089] The data receiving module 310 is used to receive a foreground image and a background image input by a user; perform mask extraction on the foreground image to obtain a foreground mask image;

[0090] The feature processing module 320 is used to fuse the foreground image, the background image and the foreground mask image to obtain a first fused image; perform depth map extraction on the first fused image to obtain a depth map of the first fused image; perform depth feature extraction on the depth map of the first fused image to obtain a depth feature map;

[0091] The feature processing module 320 is further used to perform image coding processing on the foreground image and the background image respectively to obtain the coding features of the foreground image and the coding features of the background image; input the coding features of the foreground image and the coding features of the background image into the generator respectively to perform feature reconstruction to obtain a foreground feature map after feature reconstruction and a background feature map after feature reconstruction;

[0092] A fusion module 330 is used to perform feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map;

[0093] An adaptive adjustment module 340 is used to adaptively adjust the foreground mask image to obtain a foreground mask image after adaptive resizing;

[0094] The fusion module 330 is further used to input the depth feature map, the first fused feature map and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map;

[0095] The image decoding module 350 is used to perform image decoding on the second fusion feature map to obtain a second fusion map.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0097] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0098] Figure 4A schematic block diagram of an electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0099] The electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in the ROM 402 or a computer program loaded from the storage unit 408 into the RAM 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An I / O interface 405 is also connected to the bus 404.

[0100] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0101] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0102] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0104] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0106] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0107] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0108] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0109] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A layer fusion method, characterized in that: include: Receive the foreground image and background image input by the user; Performing mask extraction on the foreground image to obtain a foreground mask image; fusing the foreground image, the background image and the foreground mask image to obtain a first fused image; Performing depth map extraction on the first fused image to obtain a depth map of the first fused image; Performing depth feature extraction on the depth map of the first fusion map to obtain a depth feature map; Performing image coding processing on the foreground image and the background image respectively to obtain coding features of the foreground image and coding features of the background image; inputting the coding features of the foreground image and the coding features of the background image into a generator respectively for feature reconstruction to obtain a foreground feature map after feature reconstruction and a background feature map after feature reconstruction; Performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map; Adaptively adjusting the foreground mask image to obtain an adaptively resized foreground mask image; Inputting the depth feature map, the first fused feature map and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map; Perform image decoding on the second fusion feature map to obtain a second fusion map.

2. The method according to claim 1, characterized in that The foreground image is one or more images; the background image is one image.

3. The method according to claim 1, characterized in that The foreground mask image is a binary image, wherein the foreground area is white and the background area is black; The step of fusing the foreground image, the background image and the foreground mask image to obtain a first fused image includes: Extracting an arbitrary region from the foreground image using the foreground mask image, wherein the operation of extracting the arbitrary region is achieved by performing a bitwise AND operation on the foreground mask image and the foreground image; The extracted foreground area is fused with the corresponding area of ​​the background image to obtain a first fused image.

4. The method according to claim 1, characterized in that: The extracting the depth map of the first fused image to obtain the depth map of the first fused image includes: Obtaining the scene relevance of each pixel in the first fusion image; Continuously downsampling the first fused image to obtain a scene correlation of each pixel in each downsampled image of the first fused image; Perform block matching motion vector calculation on each pixel in the current downsampled image and the corresponding pixel in the previous downsampled image to obtain the motion vector value of each pixel in the current downsampled image; Accumulate the motion vector value of each pixel in the current downsampled image, extract the initial depth value of each pixel from the accumulated motion vector, and the initial depth value constitutes an initial depth map; Performing continuous super-smoothing filtering and up-sampling processing on each pixel in the initial depth map by using the scene correlation of each pixel in the first fused image and the scene correlation of each pixel in each down-sampled image, so as to obtain a depth map of the first fused image; The scene correlation is the correlation between any pixel in a frame of image and its surrounding adjacent pixels.

5. The method according to claim 1, characterized in that The step of performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map includes: The foreground feature map and the background feature map are subjected to Alpha blending processing according to the foreground mask map to obtain a first fused feature map.

6. The method according to claim 1, characterized in that The adaptive adjustment of the foreground mask image is performed through an adaptive mask filtering module, and the adaptive mask filtering module is configured with an adaptive threshold segmentation technology and an adaptive filtering technology.

7. The method according to claim 6, characterized in that The depth feature map includes illumination information and shadow information of the depth map of the first fusion map; The adaptively resized foreground mask image carries position information of the foreground object; Inputting the depth feature map, the first fused feature map and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map, including: Using a generator, based on the illumination information and the shadow information, to generate a shadow image of the foreground object that meets the position information; And feature reconstruction is performed on the first fused feature map according to the shadow image to obtain a second fused feature map after feature reconstruction.

8. A layer fusion device, characterized in that: include: A data receiving module, used for receiving a foreground image and a background image input by a user; Performing mask extraction on the foreground image to obtain a foreground mask image; A feature processing module, used for fusing the foreground image, the background image and the foreground mask image to obtain a first fused image; Performing depth map extraction on the first fused image to obtain a depth map of the first fused image; performing depth feature extraction on the depth map of the first fused image to obtain a depth feature map; The feature processing module is further used to perform image coding processing on the foreground image and the background image respectively to obtain coding features of the foreground image and coding features of the background image; input the coding features of the foreground image and the coding features of the background image into the generator respectively to perform feature reconstruction to obtain a foreground feature map after feature reconstruction and a background feature map after feature reconstruction; A fusion module, used for performing feature fusion on the foreground feature map, the background feature map and the foreground mask map to obtain a first fused feature map; An adaptive adjustment module, used for adaptively adjusting the foreground mask image to obtain a foreground mask image after adaptive resizing; The fusion module is further used to input the depth feature map, the first fused feature map and the adaptively resized foreground mask map into a generator for feature reconstruction to obtain a second fused feature map; An image decoding module is used to perform image decoding on the second fusion feature map to obtain a second fusion map.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images