Image processing method and device, computer device, storage medium and program product

Abstract

Embodiments of the present application provide an image processing method and device, computer equipment, storage medium and program product, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The method comprises: obtaining a target image and an initial foreground mask image corresponding to the target image; the target image contains a target object, and the initial foreground mask image is obtained by performing segmentation processing on the target object in the target image; performing image stitching processing on the target image and the initial foreground mask image to obtain a stitched image; performing feature learning on the stitched image to obtain a reference foreground mask image corresponding to the target image; performing optimization processing on the reference foreground mask image to obtain a target foreground mask image corresponding to the target image; a target foreground region contained in the target foreground mask image is used to indicate a region of the target object in the target image; and the mask image corresponding to the image can be accurately determined.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, storage medium, and program product. Background Technology

[0002] With the rapid development of computer technology, image matting technology has been widely used in image processing, video processing and other tasks. The demand for replacing the background of portrait areas or general object areas in an image and adding novel material effects is increasing day by day. Image matting technology refers to the technique of using a mask image corresponding to the image to separate the content of interest (such as people, vehicles, flowers, trees and so on) from the image.

[0003] Image matting, like image segmentation, requires accurate segmentation of target objects within an image, while also demanding meticulous, "hair-level" precision and high edge blending. Furthermore, matting arbitrary objects involves complex scenes, ranging from cluttered indoor scenes to outdoor scenes with varying lighting conditions. This necessitates semantic differentiation of different objects, robustness to different lighting and scene variations, and accurate distinction between the object and background in finely defined local areas. In practical matting scenarios, these requirements for precise matting rely on accurate mask images; in other words, accurate mask images facilitate precise matting. Therefore, accurately determining the corresponding mask image has become a hot research topic. Summary of the Invention

[0004] This application provides an image processing method, apparatus, computer equipment, storage medium, and program product that can accurately determine the mask image corresponding to an image.

[0005] On one hand, embodiments of this application provide an image processing method, which includes...

[0006] The process involves: acquiring the target image and its corresponding initial foreground mask image; obtaining the target image containing the target object, and the initial foreground mask image obtained by segmenting the target object in the target image; stitching the target image and the initial foreground mask image together to obtain a stitched image; performing feature learning on the stitched image to obtain a reference foreground mask image corresponding to the target image; optimizing the reference foreground mask image to obtain the target foreground mask image corresponding to the target image; and obtaining the target foreground mask image containing the target foreground region corresponding to the target object, which indicates the location of the target object in the target image.

[0007] Accordingly, embodiments of this application provide an image processing apparatus, which includes:

[0008] The acquisition unit is used to acquire the target image and the corresponding initial foreground mask image; the target image contains a target object, and the initial foreground mask image is obtained by segmenting the target object in the target image;

[0009] The processing unit is used to perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image; to perform feature learning on the stitched image to obtain a reference foreground mask image corresponding to the target image; to perform optimization processing on the reference foreground mask image to obtain a target foreground mask image corresponding to the target image; the target foreground mask image contains a target foreground region corresponding to the target object, and the target foreground region is used to indicate the location of the target object in the target image.

[0010] In one implementation, the number of reference foreground mask images used for optimization is N, and the number of optimization processes is N-1, where N is an integer greater than 1. The processing unit, used to optimize the reference foreground mask images to obtain the target foreground mask image corresponding to the target image, specifically performs the following steps:

[0011] The first and second reference foreground masks from the N reference foreground mask images are optimized for the first time to obtain the first intermediate foreground mask image. The third reference foreground mask image and the first intermediate foreground mask image from the N reference foreground mask images are optimized for the second time to obtain the second intermediate foreground mask image. Subsequent optimization processes are performed until the (N-1)th optimization process is performed to obtain the (N-1)th intermediate foreground mask image. The (N-1)th intermediate foreground mask image is determined as the target foreground mask image corresponding to the target image.

[0012] In one implementation, the processing unit, when performing a first optimization process on the first reference foreground mask image and the second reference foreground mask image among N reference foreground mask images to obtain the first intermediate foreground mask image, specifically performs the following steps:

[0013] Based on the image size of the second reference foreground mask image, the first reference foreground mask image is upsampled to obtain the upsampled first reference foreground mask image; the upsampled first reference foreground mask image is then transformed to obtain a transformed foreground mask image, which includes ambiguous regions; a transformed background mask image corresponding to the transformed foreground mask image is determined; the upsampled first reference foreground mask image and the transformed background mask image are then fused to obtain a first fused image; the second reference foreground mask image and the transformed foreground mask image are then fused to obtain a second fused image; and the first fused image and the second fused image are then overlaid to obtain a first intermediate foreground mask image.

[0014] In one implementation, the reference foreground mask image and the target foreground mask image are obtained through a foreground mask optimization network. The intermediate foreground mask image obtained in each of the N-1 optimization processes is used to train the foreground mask optimization network. The processing unit, when training the foreground mask optimization network based on the i-th intermediate foreground mask image obtained in the i-th optimization process of the N-1 optimization processes, specifically performs the following steps:

[0015] Obtain the labeled foreground mask image and labeled original image corresponding to the i-th intermediate foreground mask image, where i is a positive integer less than or equal to N-1; determine the network loss information for the i-th optimization process based on the i-th intermediate foreground mask image, labeled foreground mask image, and labeled original image; optimize the network parameters of the foreground mask optimization network in the direction of reducing network loss information.

[0016] In one implementation, the processing unit, when determining the network loss information for the i-th optimization process based on the i-th intermediate foreground mask image, the labeled foreground mask image, and the labeled original image, specifically performs the following steps:

[0017] Based on the difference between the i-th intermediate foreground mask image and the labeled foreground mask image, the first loss information is determined; image fusion processing is performed on the i-th intermediate foreground mask image and the labeled original image to obtain the third fused image, and image fusion processing is performed on the labeled foreground mask image and the labeled original image to obtain the fourth fused image, and the second loss information is determined based on the difference between the third fused image and the fourth fused image; gradient processing is performed on the i-th intermediate foreground mask image and the labeled foreground mask image, and the third loss information is determined based on the difference between the gradient-processed i-th intermediate foreground mask image and the gradient-processed labeled foreground mask image; based on the first loss information, the second loss information, and the third loss information, the network loss information for the i-th optimization process is determined.

[0018] In one implementation, the reference foreground mask image and the target foreground mask image are obtained through a foreground mask optimization network; the processing unit is further configured to perform the following steps:

[0019] Obtain a training sample set for the foreground mask optimization network. The training sample set includes at least a first sample image and a second sample image. The first sample image contains a first object, and the second sample image contains a second object. Separate the foreground image containing the first object from the first sample image, and overlay the foreground image containing the first object with the second sample image to obtain a third sample image. Separate the foreground image containing the second object from the second sample image, and overlay the foreground image containing the second object with the first sample image to obtain a fourth sample image. Update the training sample set based on the third and fourth sample images. The target image is any image in the updated training sample set.

[0020] In one implementation, the number of reference foreground mask images used for optimization processing is N, where N is an integer greater than 1; the processing unit, used to perform feature learning on the stitched image to obtain the reference foreground mask image corresponding to the target image, specifically performs the following steps:

[0021] M convolutions are performed on the stitched image to obtain M convolutional feature maps; M upsampling processes are performed on the M convolutional feature maps to obtain M reference foreground mask images; N reference foreground mask images are determined from the M reference foreground mask images, and the N reference foreground mask images include at least the reference foreground mask image with the largest image size among the M reference foreground mask images, where M is an integer greater than or equal to N.

[0022] In one implementation, when the processing unit performs M convolutional processes on the stitched image to obtain M convolutional feature maps, it specifically executes the following steps:

[0023] The stitched image is subjected to a first convolution process to obtain a convolution feature map of the first convolution process; the convolution feature map of the first convolution process is subjected to a second convolution process to obtain a convolution feature map of the second convolution process; subsequent convolution processes are continued until the convolution feature map of the Mth convolution process is obtained.

[0024] In one implementation, the M-th convolutional processing includes the j-th convolutional processing. Before performing the j-th convolutional processing, the processing unit is further configured to perform the following steps: standardize the initial foreground mask image to obtain a standard foreground mask image.

[0025] When the processing unit performs the j-th convolution process, it specifically performs the following steps: overlaying the standard foreground mask image and the convolution feature map of the (j-1)-th convolution process to obtain a standard convolution overlay image; and performing convolution processing on the standard convolution overlay image to obtain the convolution feature map of the j-th convolution process, where j is a positive integer greater than 1 and less than or equal to M.

[0026] In one implementation, the processing unit, used to normalize the initial foreground mask image to obtain a standard foreground mask image, specifically performs the following:

[0027] The initial foreground mask image is normalized to obtain a normalized foreground mask image; the initial foreground mask image is convolved to obtain an adjusted feature map; the normalized foreground mask image is adjusted based on the adjusted feature map to obtain a standard foreground mask image.

[0028] In one implementation, the adjusted feature map includes a scaling feature map and a bias adjustment feature map; the processing unit, used to adjust the normalized foreground mask image according to the adjusted feature map to obtain a standard foreground mask image, specifically performs the following steps:

[0029] The normalized foreground mask image and the scaling feature map are fused together to obtain the first scaling feature map; the first scaling feature map and the offset scaling feature map are overlaid to obtain the standard foreground mask image.

[0030] In one implementation, when the processing unit performs M upsampling operations based on M convolutional feature maps to obtain M reference foreground mask images, it specifically performs the following steps:

[0031] The convolution feature map of the Mth convolution process is upsampled for the first time to obtain the reference foreground mask image of the first upsampled process. The convolution feature map of the (M-1)th convolution process and the reference foreground mask image of the first upsampled process are overlaid to obtain the reference convolution overlay image. The reference convolution overlay image is then upsampled for the second time to obtain the reference foreground mask image of the second upsampled process. Subsequent upsampling processes are continued until the reference foreground mask image of the Mth upsampled process is obtained.

[0032] In one implementation, the initial foreground mask image includes an initial foreground region corresponding to the target object; the processing unit is used to perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image, specifically performing the following steps:

[0033] The image cropping region is determined based on the initial foreground region; the target image is cropped based on the image cropping region to obtain the cropped target image; the initial foreground mask image is cropped based on the image cropping region to obtain the cropped initial foreground mask image; the cropped target image and the cropped initial foreground mask image are stitched together to obtain the stitched image.

[0034] In one implementation, the processing unit, after optimizing the reference foreground mask image to obtain the target foreground mask image corresponding to the target image, is further configured to perform the following steps:

[0035] Determine the target background mask image corresponding to the target foreground mask image; perform image separation processing on the target image based on the target foreground mask image to obtain a foreground image containing the target object; obtain a reference image, and perform image separation processing on the reference image based on the target background mask image to obtain a background image; perform image overlay processing on the foreground image containing the target object and the background image to obtain a composite image.

[0036] Accordingly, embodiments of this application provide a computer device, which includes a processor and a computer-readable storage medium; wherein the processor is adapted to implement a computer program, the computer-readable storage medium stores the computer program, and the computer program is adapted to be loaded by the processor and executed by the above-described image processing method.

[0037] Accordingly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the image processing method described above.

[0038] Accordingly, embodiments of this application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method described above.

[0039] In this embodiment, a target image and an initial foreground mask image corresponding to the target image can be obtained. The initial foreground mask image is obtained by segmenting the target object in the target image. Next, the target image and the initial foreground mask image can be stitched together, and feature learning can be performed on the stitched image to obtain a reference foreground mask image corresponding to the target image. Then, the reference foreground mask image can be optimized. The target foreground region in the optimized target foreground mask image can be used to accurately indicate the location of the target object in the target image. That is, after performing feature learning on the stitched image obtained by stitching the target image and the initial foreground mask image to obtain the reference foreground mask image, the accuracy of the target foreground mask region obtained by optimizing the reference foreground mask image is improved compared to both the reference foreground mask image and the initial foreground mask region. Thus, this embodiment can accurately determine the foreground mask image corresponding to the target image. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of a mask image provided in an embodiment of this application;

[0042] Figure 2a This is a schematic diagram of a foreground matting method provided in an embodiment of this application;

[0043] Figure 2b This is a schematic diagram of a background matting method proposed in an embodiment of this application;

[0044] Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0045] Figure 4a This is a schematic diagram of the structure of a deep learning segmentation network provided in an embodiment of this application;

[0046] Figure 4b This is a schematic diagram of an image post-processing procedure provided in an embodiment of this application;

[0047] Figure 5 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0048] Figure 6aThis is a schematic diagram of the structure of a foreground mask optimization network provided in an embodiment of this application;

[0049] Figure 6b This is a schematic diagram of the structure of a standardized module provided in an embodiment of this application;

[0050] Figure 6c This is a schematic diagram of an optimization process provided in an embodiment of this application;

[0051] Figure 7 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0052] Figure 8 This is a schematic diagram illustrating the training process of a foreground mask optimization network provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] To better understand the technical solutions provided in the embodiments of this application, the key terms involved in the embodiments of this application are introduced here:

[0057] (1) This application relates to Artificial Intelligence (AI). AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. AI technology is a multidisciplinary field involving a wide range of areas, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0058] (2) The embodiments of this application relate to computer vision (CV) technology in artificial intelligence. Computer vision technology is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes to identify and measure targets, and further performs image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (3-dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies.

[0059] (3) This application relates to image matting technology in the field of image processing within computer vision technology. Image matting technology refers to the technique of using a mask image corresponding to an image to separate content of interest (e.g., people, vehicles, flowers, trees, etc.) from the image. Generally, the mask image can include regions of interest and non-regions of interest. Regions of interest can further include defined regions of interest and uncertain regions of interest. Regions of interest in the mask image can be used to separate content of interest from the image. The pixel value of a pixel located in a defined region of interest in the mask image can be 1, the pixel value of a pixel located in a non-region of interest can be 0, and the pixel value of a pixel located in an uncertain region of interest belongs to the interval (0, 1). This uncertain region of interest can be understood as the area where content such as hair strands or clothing mesh is located in the image. A pixel with a value of 1 will appear white in the mask image, a pixel with a value of 0 will appear black, and a pixel with a value in the range (0, 1) will appear gray. Furthermore, pixels with different values ​​within the range (0, 1) will appear as different shades of gray in the mask image; for example, a pixel with a value of 0.5 will appear as a different shade of gray than a pixel with a value of 0.2. Figure 1 Taking the mask image shown as an example, the region of interest in the mask image is used to separate the person from the image. The region of interest is shown as white in the mask image, the non-region of interest is shown as black in the mask image, and the uncertain region of interest where the person's hair is located is shown as gray in the mask image.

[0060] In this embodiment, the mask images can be divided into two categories: foreground mask images and background mask images. For any given image, the foreground mask image and the background mask image exist relative to each other. The foreground mask image refers to the mask image whose region of interest is the foreground region. The foreground region can indicate the location of the target object (e.g., any object such as a person, vehicle, tree, flower, etc.) in the image, and it can be used to separate the foreground image containing the target object from the image. The background mask image refers to the mask image whose region of interest is the background region. The background region can indicate the location of other image content in the image besides the target object, and it can be used to separate the background image containing other image content from the image. It should be noted that the background mask image can be determined based on the foreground mask image. The background mask image can be obtained by subtracting the foreground mask image from a template image with the same image size as the foreground mask image. That is, the pixel value of each point in the background mask image can be equal to the pixel value of the corresponding pixel in the template image minus the pixel value of the corresponding pixel in the foreground mask image. The pixel value of each point in the template image can be the target pixel value (e.g., 1). Figure 2a As shown, using the foreground mask image 201, a foreground image 204 containing the target object 203 can be extracted from image 202; as Figure 2b As shown, the background mask image 205 is based on Figure 2a The foreground mask image 201 shown is used to determine the background mask image 205, from which a background image 206 containing image content other than the target object 203 can be extracted from the image 202.

[0061] Based on the aforementioned descriptions of key terms, this application provides an image processing scheme. This scheme can obtain a precise foreground mask image corresponding to any given target image and a coarse foreground mask image obtained by coarsely segmenting the target object in the target image. The foreground region in the precise foreground mask image can accurately indicate the location of the target object in the target image; that is, the precise foreground mask image can accurately separate the target object from the target image. This application decouples semantic segmentation and matting effects. Semantics can be understood as the object type indicated by the pixel values ​​of each pixel in the image. For example, the pixel value of a certain pixel indicates the object type as sky, and the pixel value of another pixel indicates the object type as grass, etc. Decoupling semantic segmentation and matting effects means that this application does not require refined object categories and can achieve adaptation to any object. After providing coarse segmentation guidance for the target image, a refined foreground mask image can be obtained for the target object identified by the coarse segmentation guidance, thereby enabling precise matting based on the refined foreground mask image. In addition, the image processing solution provided in this application embodiment can be integrated into a plug-and-play functional module. This functional module can be an application, software, or plugin with image processing capabilities, and this application embodiment does not limit this. When image processing capabilities are required, the functional module can be called to accurately determine the foreground mask image corresponding to the image, thereby achieving precise image matting.

[0062] In specific implementations, the image processing scheme provided in this application embodiment can be executed by a computer device, which can be a terminal or a server; the terminal mentioned in this application embodiment can be a smartphone, tablet computer, laptop computer, desktop computer, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, head-mounted device, etc., but is not limited to these; the server mentioned in this application embodiment can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers.

[0063] It is worth mentioning that when the target object is a person, the precise foreground mask image provided by the image processing solution in this application embodiment can accurately cut out areas such as hair and clothing mesh, resulting in better integration of the portrait with the background when compositing a new image, thus enhancing the realism of the synthesized image. This has significant practical value in scenarios such as album production, film and television processing, video production, short video production, and special effects texturing. When the target object is any object other than a person, the foreground of interest in the image or video can be cut out, and special effects materials can be added to the separated foreground or background areas to achieve various different gameplay solutions. In particular, this image processing solution can also be applied to scenarios such as intelligent transportation and assisted driving. For example, during vehicle operation, traffic signs, traffic lights, and obstacles with potential safety hazards in the surrounding environment image can be accurately cut out, serving as a reminder in intelligent transportation and assisted driving scenarios.

[0064] It is understood that in the specific embodiments of this application, data such as images of people are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0065] The image processing scheme provided in the embodiments of this application will be described in more detail below with reference to the accompanying drawings.

[0066] This application provides an image processing method that can be executed by the aforementioned computer device. This application primarily describes the image processing method from a network application perspective, focusing on the segmentation process of a target object in a target image, the image stitching process, and the image post-processing process based on the target foreground mask image obtained through optimization. For example... Figure 3 As shown, the image processing method may include the following steps S301 to S304:

[0067] S301, acquire the target image and the initial foreground mask image corresponding to the target image.

[0068] The target image may contain target objects. The initial foreground mask image can be obtained by segmenting the target objects in the target image. This segmentation can be performed using a deep learning segmentation network. Here, we will first combine... Figure 4aThe network structure of a deep learning segmentation network is introduced as follows: A deep learning segmentation network is a semantic segmentation model designed with a depthwise separable convolutional structure, capable of distinguishing foreground and background of general objects. A deep learning segmentation network can include an encoder and a decoder. The encoder can be composed of stacked modules based on convolution (e.g., depthwise separable convolution). The decoder can use a deconvolutional structure (e.g., depthwise separable deconvolution) to upsample the feature maps layer by layer. The encoder and decoder of the deep learning segmentation network can transfer features between shallow and deep layers through skip connections, enabling the overall network to fuse features from different stages. The semantic information in the shallow features and the structural information in the deep features guide the deep learning segmentation network to output the initial foreground mask image corresponding to the target image.

[0069] To facilitate the description of the process by which a deep learning segmentation network segments objects in a target image, the convolutional structure of the encoder in the deep learning segmentation network can be represented as X layers, and the deconvolutional structure of the decoder in the deep learning segmentation network can be represented as X layers, where X is an integer greater than 1. The process by which the deep learning segmentation network segments objects in a target image can include: calling the X-layer convolutional structure to perform X convolutional processing based on the target image, obtaining the convolutional feature map of the X-layer convolutional structure; and then calling the X-layer deconvolutional structure to perform X upsampling processing based on the convolutional feature map of the X-layer convolutional structure, obtaining the initial foreground mask image corresponding to the target image.

[0070] The process of calling an X-layer convolutional structure to perform X convolutional processes on the target image to obtain the convolutional feature map of the X-layer convolutional structure can include: calling the first-layer convolutional structure to perform a first convolutional process on the target image to obtain the convolutional feature map of the first-layer convolutional structure; calling the second-layer convolutional structure to perform a second convolutional process on the convolutional feature map of the first-layer convolutional structure to obtain the convolutional feature map of the second-layer convolutional structure; continuing to perform subsequent convolutional processes until the convolutional feature map of the X-th convolutional structure is obtained.

[0071] The process of calling an X-layer deconvolution structure to perform X upsampling operations on the convolutional feature map of the X-layer convolution structure to obtain the initial foreground mask image corresponding to the target image may include: calling the first deconvolution structure to upsample the convolutional feature map of the X-layer convolution structure to obtain the upsampled feature map of the first deconvolution structure; performing image overlay processing on the upsampled feature map of the first deconvolution structure and the convolutional feature map of the (X-1)-layer convolution structure, and then calling the second deconvolution structure to upsample the feature map obtained by the overlay processing to obtain the upsampled feature map of the second deconvolution structure; continuing to perform subsequent upsampling operations until the upsampled feature map of the X-layer deconvolution structure is obtained, which is the initial foreground mask image. It should be noted that image overlay processing of the upsampled feature map of the first deconvolutional layer and the convolutional feature map of the (X-1)th convolutional layer means: adding the pixel value of each pixel in the upsampled feature map of the first deconvolutional layer to the pixel value of the corresponding pixel in the convolutional feature map of the (X-1)th convolutional layer, and obtaining the pixel value of the corresponding pixel in the overlay feature map.

[0072] Figure 4a The deep learning segmentation network shown is an encoder-decoder network structure based on depthwise separable convolution. This structure greatly reduces the computational cost (the overall computational cost of the network is around 1 G Flpos (Floating Point Operations)) and the number of parameters when using deep learning segmentation networks to segment any object in an image, while still ensuring that the deep learning segmentation network has a certain level of segmentation performance.

[0073] S302, perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image.

[0074] After obtaining the target image and its corresponding initial foreground mask image, image stitching can be performed on the target image and the initial foreground mask image to obtain a stitched image. Image stitching can include direct image stitching or indirect image stitching, which will be described below:

[0075] The first method is direct image stitching. Direct image stitching of the target image and the initial foreground mask image means that no additional processing is required before stitching the target image and the initial foreground mask image; they are simply stitched together directly. Specifically, the target image is a multi-channel image, meaning it can contain multiple channels, while the initial foreground mask image is a single-channel image. Channel stitching is performed between the multiple channels in the target image and the initial foreground mask image, resulting in a stitched image. The number of channels in the stitched image can be equal to the number of channels in the target image plus one.

[0076] For example, when the color mode of the target image is RGB (Red, Green, Blue), the target image includes three channels: the R channel (red channel), the G channel (green channel), and the B channel (blue channel). The R, G, and B channels of the target image can be combined with the initial foreground mask image through channel shifting to obtain a stitched image, which includes four channels.

[0077] The second method is indirect image stitching. Indirect image stitching of the target image and the initial foreground mask image refers to the following: before performing image stitching on the target image and the initial foreground mask image, image cropping can be performed on both images. Then, the cropped target image and the cropped initial foreground mask image can be stitched together. The image cropping process for the target image and the initial foreground mask image can include: the initial mask image may include an initial foreground region corresponding to the target object; the image cropping region can be determined based on the initial foreground region; and then the target image and the initial foreground mask image can be cropped based on the image cropping region. The process of determining the image cropping region based on the initial foreground region can include: determining the minimum bounding rectangle of the initial foreground region, and then expanding the minimum bounding rectangle to obtain the image cropping region, for example, increasing both the length and width of the minimum bounding rectangle by 1 cm, or increasing the length of the minimum bounding rectangle by 1 cm and the width by 0.5 cm; the minimum bounding rectangle refers to the smallest enclosing rectangle of the initial foreground region represented by two-dimensional coordinates, and can be determined based on the maximum horizontal coordinate, maximum vertical coordinate, minimum horizontal coordinate, and minimum vertical coordinate of each pixel in the initial foreground region.

[0078] Similarly, the cropped target image is a multi-channel image, meaning it can include images with multiple channels. The cropped initial foreground mask image is a single-channel image. Channel stitching can be performed on the multiple channels in the cropped target image and the cropped initial foreground mask image according to the channel dimension to obtain a stitched image. The number of image channels in the stitched image can be equal to the number of image channels in the cropped target image plus 1.

[0079] S303, perform feature learning on the stitched image to obtain the reference foreground mask image corresponding to the target image.

[0080] S304, optimize the reference foreground mask image to obtain the target foreground mask image corresponding to the target image.

[0081] In steps S303-S304, after image stitching processing is performed on the target image and the initial foreground mask image to obtain the stitched image, feature learning can be performed on the stitched image to obtain the reference foreground mask image corresponding to the target image; then the reference foreground mask image can be optimized to obtain the target foreground mask image corresponding to the target image; the target foreground mask image may contain the target foreground region corresponding to the target object, and the target foreground region is used to indicate the location of the target object in the target image.

[0082] The process of learning features from the stitched image to obtain the reference foreground mask image corresponding to the target image can include: performing M convolutional processes on the stitched image to obtain M convolutional feature maps; performing M upsampling processes on the M convolutional feature maps to obtain M reference foreground mask images; then determining N reference foreground mask images from the M reference foreground mask images. These N reference foreground mask images can be used for optimization processing. At least one of the N reference foreground mask images should be included, ensuring that the image size of the optimized target foreground mask image is the same as the image size of the target image in the stitched image. Here, N is an integer greater than 1, and M is an integer greater than or equal to N. After determining the N reference foreground mask images for optimization processing, N-1 optimization processes can be performed on these N reference foreground mask images to obtain the target foreground mask image corresponding to the target image.

[0083] After optimizing the reference foreground mask image to obtain the target foreground mask image corresponding to the target image, post-processing can be performed on the target image based on the target foreground mask image. Post-processing refers to the process of separating the target object in the target image using the target foreground mask image, and then combining the target object with a new background to generate a new image. The process of post-processing the target image based on the target foreground mask image can include:

[0084] (1) The target background mask image corresponding to the target foreground mask image can be determined. The process of determining the target background mask image can be understood as follows: the image obtained by subtracting the target foreground mask image from the template image with the same image size as the target foreground mask image. The pixel value of each pixel in the template image is the target pixel value (for example, it can be 1). That is to say, the pixel value of each pixel in the target background mask image is equal to the pixel value of the corresponding pixel in the template image minus the pixel value of the corresponding pixel in the target foreground mask image. (2) The target image can be processed by image separation based on the target foreground mask image to obtain a foreground image containing the target object. Image separation based on the target foreground mask image means: multiplying the pixel value of each pixel in the target foreground mask image with the pixel value of the corresponding pixel in the target image to obtain the pixel value of the corresponding pixel in the foreground image. After multiplying the target foreground mask image with the target image, the pixel value of the target object in the target image can be retained. (3) A reference image can be obtained, and the reference image can be separated from the target background mask image to obtain a background image. The reference image is the image that provides a new background for the target object. Similarly, the image separation process of the reference image based on the target background mask image means: multiplying the pixel value of each pixel in the target background mask image with the pixel value of the corresponding pixel in the reference image to obtain the pixel value of the corresponding pixel in the background image. After multiplying the target background mask image with the reference image, the new background provided for the target object can be retained. (4) The foreground image and the background image containing the target object can be overlaid to obtain a composite image. The image overlay process of the foreground image and the background image means: adding the pixel value of each pixel in the foreground image with the pixel value of the corresponding pixel in the background image to obtain the pixel value of the corresponding pixel in the composite image. The above image post-processing process can be expressed as the following formula 1:

[0085] Pred final =Pred*RAW+(1-Pred)*BACK formula 1;

[0086] The parameters in Formula 1 above are explained below: Pred finalThe target foreground mask image can be represented by Pred, RAW can be represented by RAW, (1-Pred) can be represented by (1-Pred) and BACK can be represented by BACK. It should be noted that in this embodiment, the target foreground mask image can be directly used to synthesize a new image, that is, the content described in (1)-(4) above; in this embodiment, the target foreground mask image can also be processed by image activation (for example, by using the sigmoid function to process the image activation), and the target foreground mask image after image activation is used to synthesize a new image, so that the target background mask image corresponding to the target foreground mask image after image activation is determined, and the target image is processed by image separation according to the target foreground mask image after image activation to obtain a foreground image containing the target object. Then, a reference image can be obtained, and the reference image is processed by image separation according to the target background mask image to obtain a background image. Finally, the foreground image containing the target object and the background image can be overlaid to obtain a synthesized image. By performing image activation processing on the target foreground mask image, the accuracy of the target foreground mask image can be further improved. The above image post-processing process can be referred to Figure 4b The example shown, Figure 4b Taking a person as an example, the illustration is as follows: Figure 4b It is easy to see that the target foreground mask image obtained through the optimization processing in the embodiments of this application can accurately cut out fine areas such as hair strands and clothing edges of people, and the separated target object can be highly integrated with the new background, effectively improving the cutout effect and the new image synthesis effect.

[0087] In this embodiment, feature learning is performed on the stitched image obtained by stitching the target image and the initial foreground mask image. During the feature learning process, the edge detail features of the target object in the target image can be fused into the initial foreground region of the initial foreground mask image. This can continuously improve the accuracy of the initial foreground region in the initial foreground mask image during the feature learning process. Furthermore, in the indirect image stitching process, image cropping is performed on the initial foreground mask image and the target image, which can eliminate large-area background missegmentation that may exist in the initial foreground mask image. This is beneficial for focusing on learning the relevant features of the foreground region during the feature learning process.

[0088] This application provides an image processing method that can be executed by the aforementioned computer device. This application mainly introduces the image processing method from a network application perspective, focusing on the feature learning process of the stitched image (i.e., the aforementioned M-fold convolution and M-fold upsampling processes) and the optimization process of the initial foreground mask image (i.e., the aforementioned N-1-fold optimization processes). Figure 5 As shown, the image processing method may include the following steps S501 to S506:

[0089] S501, acquire the target image and the corresponding initial foreground mask image.

[0090] The execution process of step S501 in this embodiment is the same as described above. Figure 3 The execution process of step S301 in the illustrated embodiment is the same, and can be found in the above description. Figure 3 The description of step S301 in the illustrated embodiment will not be repeated here.

[0091] S502, perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image.

[0092] The execution process of step S502 in this embodiment is the same as described above. Figure 3 The execution process of step S302 in the illustrated embodiment is the same; please refer to the above for details. Figure 3 The description of step S302 in the illustrated embodiment will not be repeated here.

[0093] Before describing steps S503-S506 in the embodiments of this application, it should be noted that both the feature learning process and the optimization process in the embodiments of this application are executed by calling the foreground mask optimization network. That is, steps S503-S506 in the embodiments of this application are executed by calling the foreground mask optimization network. To better understand the feature learning process and optimization process in the embodiments of this application, we will first combine... Figure 6a Foreground mask optimization networks (PFRMs) can include an encoder, a decoder, a normalization module (Spade module), and an optimization module (PRM module). Specifically: ① The encoder in the foreground mask optimization network can be constructed by stacking modules based on convolutions. The stitched image can be input into the encoder of the foreground mask optimization network for convolution processing. To facilitate the description of the feature learning process based on the foreground mask optimization network, the convolutional structure included in the encoder of the foreground mask optimization network can be represented as M layers. Figure 6aThe encoder in the foreground mask optimization network consists of 4 convolutional layers as an example. ② The decoder in the foreground mask optimization network can perform upsampling of the feature map layer by layer using deconvolutional structures. To facilitate the description of the feature learning process based on the foreground mask optimization network, the deconvolutional structure included in the decoder can be represented as M layers. Figure 6a An example is taken using a foreground mask optimization network whose decoder consists of four deconvolutional layers. Furthermore, the encoder and decoder of the foreground mask optimization network can transfer shallow and deep features via skip connections. This allows the decoder to fuse features from different stages, using semantic information from shallow features and structural information from deep features to guide the output of a reference foreground mask image corresponding to the target image. ③ The Spade module can be used to standardize the initial foreground mask image to obtain a standard foreground mask image. This standard foreground mask image can be added to the feature map output by one or more convolutional layers in the encoder of the foreground mask optimization network. The resulting feature map can be used as input to subsequent networks. Figure 6a An example is given by adding the normalized foreground mask image obtained through Spade module normalization to the feature map output from the second convolutional network structure in the encoder of the foreground mask optimization network. ④ The foreground mask optimization network may include one or more PRM modules. When the foreground mask optimization network includes multiple PRM modules, the multiple PRM modules can perform layer-by-layer optimization on the reference foreground mask image output by the decoder of the foreground mask optimization network, and determine the foreground mask image output by the last PRM module as the target foreground mask image corresponding to the target image.

[0094] S503 performs M convolutions on the stitched image to obtain M convolutional feature maps.

[0095] After stitching the target image and the initial foreground mask image to obtain the stitched image, M convolutional operations can be performed on the stitched image to obtain M convolutional feature maps. As mentioned above, the M convolutional operations are performed by calling the M-layer convolutional structure in the encoder of the foreground mask optimization network. The process of calling the M-layer convolutional structure to perform M convolutional operations on the stitched image to obtain M convolutional feature maps can include: calling the first convolutional structure to perform the first convolutional operation on the stitched image to obtain the first convolutional feature map; calling the second convolutional structure to perform the second convolutional operation on the first convolutional feature map to obtain the second convolutional feature map; continuing to perform subsequent convolutional operations until the convolutional feature map of the Mth convolutional operation is obtained.

[0096] As mentioned above, the Spade module can be used to standardize the initial foreground mask image to obtain a standard foreground mask image. The standard foreground mask image can be added to the feature map output by one or more convolutional structures in the encoder of the foreground mask optimization network. The feature map obtained by the addition can be used as the input of subsequent networks. This section uses the example of adding the feature map output from a certain convolutional layer (which can be represented as the (j-1)th convolutional layer) in the encoder of the foreground mask optimization network, and then inputting the resulting feature map into the j-th convolutional processing of the j-th convolutional layer, as an example: The Spade module can be called to standardize the initial foreground mask image to obtain a standard foreground mask image; the standard foreground mask image and the convolutional feature map from the (j-1)th convolutional processing can be overlaid to obtain a standard convolutional overlay image. Overlaying the standard foreground mask image and the convolutional feature map from the (j-1)th convolutional processing means adding the pixel value of each pixel in the standard foreground mask image to the corresponding pixel value in the (j-1)th convolutional feature map to obtain the pixel value of the corresponding pixel in the standard convolutional overlay image; then, the standard convolutional overlay image can be convolved to obtain the convolutional feature map from the j-th convolutional processing, where j is a positive integer greater than 1 and less than or equal to M.

[0097] Before introducing the standardized processing procedure of the Spade module, let's first combine... Figure 6b The structure of the Spade module is introduced below: The Spade (Spatially-Adaptive (DE)normalization) module, as a conditional normalization method, includes batch normalization and convolutional structures. Batch normalization normalizes the initial foreground mask image at the channel level, resulting in a normalized foreground mask image. The convolutional structure learns adjustment feature maps from the initial foreground mask image; these adjustment feature maps can include scaled feature maps. Figure 6b In the diagram, γ is represented as the bias adjustment feature map (bias, in...). Figure 6b The feature map (denoted as β) can be used to adjust the normalized foreground mask image to obtain a standard foreground mask image.

[0098] based on Figure 6bThe structure of the Spade module shown, and the process of calling the Spade module to standardize the initial foreground mask image to obtain a standard foreground mask image, may include: calling a batch normalization structure to normalize the initial foreground mask image to obtain a normalized foreground mask image; calling a convolution structure in the Spade module to convolve the initial foreground mask image to obtain an adjusted feature map; and adjusting the normalized foreground mask image according to the adjusted feature map to obtain a standard foreground mask image.

[0099] The adjustment feature map can include a scaling adjustment feature map and a bias adjustment feature map. The process of adjusting the normalized foreground mask image according to the adjustment feature map to obtain a standard foreground mask image can include: performing image fusion processing on the normalized foreground mask image and the scaling adjustment feature map to obtain a first adjustment feature map. The image fusion processing on the normalized foreground mask image and the scaling adjustment feature map means: multiplying the pixel value of each pixel in the normalized foreground mask image with the pixel value of each pixel in the scaling adjustment feature map to obtain the pixel value of the corresponding pixel in the first adjustment feature map. The scaling adjustment feature map can be used to scale the normalized foreground mask image. Scale adjustment means scaling the pixel values ​​in the normalized foreground mask image according to the scaling adjustment feature map. In other words, performing image fusion processing on the normalized foreground mask image and the scaling adjustment feature map means scaling the pixel values ​​in the normalized foreground mask image according to the scaling adjustment feature map. Then, the first adjustment feature map and the offset adjustment feature map can be overlaid to obtain a standard foreground mask image. Overlaying the first adjustment feature map and the offset adjustment feature map means adding the pixel value of each pixel in the first adjustment feature map to the pixel value of the corresponding pixel in the offset adjustment feature map to obtain the pixel value of the corresponding pixel in the standard foreground mask image. The offset adjustment feature map can be used to further adjust the offset of the first adjustment feature map. Offset adjustment means shifting the pixel value in the first adjustment feature map according to the offset adjustment feature map. In other words, the essence of overlaying the first adjustment feature map and the offset adjustment feature map is to shift the pixel value in the first adjustment feature map according to the offset adjustment feature map.

[0100] In this embodiment, the initial foreground mask image can reflect the semantic information in the target image (i.e., which pixels in the target image are foreground and which pixels are background). The Spade module can enhance the importance of the semantic information in the initial foreground mask image and introduce the standard foreground mask image containing enhanced semantic information into the encoder of the foreground mask optimization network. It combines the convolutional feature map output by the convolutional structure for feature learning, which can enhance the overall network structure of the foreground mask optimization network's ability to acquire the semantic information of the target image.

[0101] S504 performs M upsampling operations based on M convolutional feature maps to obtain M reference foreground mask images.

[0102] After performing M convolutions on the stitched image to obtain M convolutional feature maps, M upsampling processes can be performed on these M convolutional feature maps to obtain M reference foreground mask images. As mentioned earlier, the M upsampling processes are executed by calling the M-layer deconvolutional structure in the decoder of the foreground mask optimization network. The process of calling the M-layer deconvolutional structure to perform M upsampling processes on the M convolutional feature maps to obtain M reference foreground mask images can include: calling the first-layer deconvolutional structure to perform a first upsampling process on the convolutional feature map of the Mth convolution, obtaining the first upsampled reference foreground mask image; performing image overlay processing on the convolutional feature map of the (M-1)th convolution and the first upsampled reference foreground mask image to obtain a reference convolutional overlay image; calling the second-layer deconvolutional structure to perform a second upsampling process on the reference convolutional overlay image to obtain the second upsampled reference foreground mask image; continuing with subsequent upsampling processes until the Mth upsampled reference foreground mask image is obtained. The image overlay process for the convolution feature map of the (M-1)th convolution and the reference foreground mask image of the first upsampling process refers to adding the pixel value of each pixel in the convolution feature map of the (M-1)th convolution process to the pixel value of the corresponding pixel in the reference foreground mask image of the first upsampling process, so as to obtain the pixel value of the corresponding pixel in the reference convolution overlay image.

[0103] S505, determine N reference foreground mask images from M reference foreground mask images.

[0104] After performing M upsampling operations on M convolutional feature maps to obtain M reference foreground mask images, N reference foreground mask images for optimization can be determined from these M images. The N reference foreground mask images are some or all of the M reference foreground mask images. Specifically, the N reference foreground mask images must include at least the reference foreground mask image with the largest image size among the M reference foreground mask images; that is, the N reference foreground mask images must include at least the reference foreground mask image from the Mth upsampling operation.

[0105] S506, perform N-1 optimization processes on N reference foreground mask images to obtain the target foreground mask image corresponding to the target image.

[0106] After determining N reference foreground mask images from M reference foreground mask images, N-1 optimization processes can be performed on the N reference foreground mask images to obtain the target foreground mask image corresponding to the target image. As mentioned above, the foreground mask optimization network can include one or more PRM modules, and each optimization process can be performed by calling one PRM module. That is, N-1 optimization processes can be performed by calling N-1 PRM modules. The process of calling N-1 PRM modules to perform N-1 optimization processes on N reference foreground mask images to obtain the target foreground mask image corresponding to the target image can include: calling the first PRM module to perform a first optimization process on the first and second reference foreground mask images among the N reference foreground mask images to obtain a first intermediate foreground mask image, the image size of the first reference foreground mask image being smaller than the image size of the second reference foreground mask image; then calling the second PRM module to perform a second optimization process on the third reference foreground mask image and the first intermediate foreground mask image among the N reference foreground mask images to obtain a second intermediate foreground mask image, the image size of the third reference foreground mask image being larger than the image size of the first intermediate foreground mask image; then continuing to perform subsequent optimization processes until calling the N-1th PRM module to perform the N-1th optimization process to obtain the N-1th intermediate foreground mask image, and determining the N-1th intermediate foreground mask image as the target foreground mask image corresponding to the target image. As can be seen, the N-1 PRM modules can refine the reference foreground mask image layer by layer. Each optimization process can increase the edge details of the foreground region to a certain extent. As the number of optimizations gradually increases, the edge details of the foreground region become richer and the foreground region becomes more accurate.

[0107] It should be noted that the process of each optimization step in the N-1 optimization process is the same. Here, we will use the first optimization step as an example to describe the process. For the processes of other optimization steps in the N-1 optimization process besides the first one, please refer to the process of the first optimization step. The following will combine... Figure 6c The process of the first optimization is described. This involves calling the first PRM module to perform the first optimization on the first and second reference foreground mask images out of N reference foreground mask images, resulting in the first intermediate foreground mask image. This process may include:

[0108] (1) The first reference foreground mask image is upsampled according to the image size of the second reference foreground mask image to obtain the upsampled first reference foreground mask image. The image size of the upsampled first reference foreground mask image is the same as the image size of the second reference foreground mask image.

[0109] (2) Perform image transformation processing on the upsampled first reference foreground mask image to obtain a transformed foreground mask image, which includes ambiguous regions. Image transformation processing can be understood as a pixel value mapping process, that is, mapping the pixel values ​​of each pixel in the upsampled first reference foreground mask image to the corresponding pixel values ​​in the transformed foreground mask image through a mapping function. The mapping function can be found in Formula 2 below:

[0110]

[0111] The parameters in Formula 2 above are explained below: α can represent the pixel value of the target pixel (x, y) in the transformed foreground mask image, where the target pixel can be any pixel in the transformed foreground mask image; k-1 (x, y) can represent the pixel value of the target pixel (x, y) in the first reference foreground mask image after upsampling. The meaning of Formula 2 (i.e., the meaning of image transformation processing) is explained below: If the pixel value of a certain pixel in the first reference foreground mask image after upsampling belongs to the interval (0, 1), then the pixel value of that pixel can be mapped to 1; if the pixel value of a certain pixel in the first reference foreground mask image after upsampling does not belong to the interval (0, 1), for example, the pixel value is 0 or 1, then the pixel value of that pixel can be mapped to 0; after mapping each pixel in the first reference foreground mask image after upsampling according to the above formula, the transformed foreground mask image is obtained. It should be noted that the foreground region contained in the transformed foreground mask image is an ambiguous region with low confidence (i.e., the uncertain region of interest mentioned above). An ambiguous region is one in which the pixel values ​​of the pixels belonging to the ambiguous region belong to the interval (0, 1). The semantics of the pixel values ​​of these pixels are uncertain. That is, it cannot be determined whether the pixel values ​​of the pixels belonging to the ambiguous region represent the foreground or the background. Furthermore, ambiguous regions often contain edge details of the foreground region.

[0112] (3) Determine the transformed background mask image corresponding to the transformed foreground mask image. The process of determining the transformed background mask image can be understood as follows: Subtract the transformed background mask image from a template image with the same image size as the transformed foreground mask image. The pixel value of each pixel in the template image is the target pixel value (e.g., it can be 1). In other words, the pixel value of each pixel in the transformed background mask image is equal to the pixel value of the corresponding pixel in the template image minus the pixel value of the corresponding pixel in the transformed foreground mask image. The transformed background mask image can contain foreground regions with high confidence.

[0113] (4) Perform image fusion processing on the upsampled first reference foreground mask image and the transformed background mask image to obtain the first fused image. Image fusion processing on the upsampled first reference foreground mask image and the transformed background mask image means: multiplying the pixel value of each pixel in the upsampled first reference foreground mask image with the pixel value of the corresponding pixel in the transformed background mask image to obtain the pixel value of the corresponding pixel in the first fused image.

[0114] (5) Perform image fusion processing on the second reference foreground mask image and the transformed foreground mask image to obtain the second fused image. Performing image fusion processing on the second reference foreground mask image and the transformed foreground mask image to obtain the second fused image means multiplying the pixel value of each pixel in the second reference foreground mask image with the pixel value of the corresponding pixel in the transformed foreground mask image to obtain the pixel value of the corresponding pixel in the second fused image.

[0115] (6) Perform image overlay processing on the first fused image and the second fused image to obtain a first intermediate foreground mask image. Performing image overlay processing on the first fused image and the second fused image to obtain the first intermediate foreground mask image means: adding the pixel value of each pixel in the first fused image to the pixel value of the corresponding pixel in the second fused image to obtain the pixel value of the corresponding pixel in the first intermediate foreground mask image.

[0116] The processes described in (3)-(6) above can be found in the following formula description:

[0117] α k =α′ k g k +α k-1 (1-g k ) Formula 3;

[0118] The parameters in Formula 3 above are explained below: α k This can represent the first intermediate foreground mask image; α′ k It can represent a second reference foreground mask image; g k It can represent a transformed foreground mask image; α′ k g k This can represent the first fused image; α k-1 This can represent the first reference foreground mask image after upsampling; (1-g k ) can represent a transformed background mask image; α k-1 (1-g k ) can represent the second fused image.

[0119] As can be seen from Formula 3 and (1)-(6) above, the essence of the optimization process is to fuse and superimpose the foreground region with high confidence in the first reference foreground mask image and the ambiguous region with low confidence in the second reference foreground mask image. Since the ambiguous region often contains edge details of the foreground region, fusing the foreground region with high confidence and the ambiguous region with low confidence can increase the edge details in the foreground region and improve the accuracy of the foreground region. Furthermore, through multi-level optimization processing, the edge details of the foreground region can be continuously enriched, so that the final optimization process can obtain an accurate target foreground mask image.

[0120] In this embodiment, during the M convolution and M upsampling processes of the stitched image, the edge detail features of the target object in the target image are fused into the initial foreground region of the initial foreground mask image to obtain a reference foreground mask image. Compared to the foreground region in the initial foreground mask image, the foreground region in the reference foreground mask image can more accurately indicate the location of the target object in the target image. Through layer-by-layer optimization, foreground regions with high confidence and ambiguous regions with low confidence but containing more edge details of the target object can be continuously fused, thereby improving the accuracy of the foreground mask image layer by layer. Furthermore, for target objects such as people, the target foreground mask image output by the foreground mask optimization network in this embodiment can effectively define ambiguous regions containing edge details of the target object while ensuring accurate separation of the target object in the target image. This ensures clear edges even when the background and foreground RGB color spaces of the target image are similar or under extreme lighting conditions. In the embodiments of this application, the target foreground mask image output by the foreground mask optimization network can handle different characteristics of different objects, such as animal fur, smooth edges of buildings, and mesh of fabric, etc., and can automatically generate accurate matting results.

[0121] This application provides an image processing method that can be executed by the aforementioned computer device. This application mainly describes the image processing method from the perspective of network training, focusing on the training process of deep learning segmentation networks and foreground mask optimization networks, as well as the process of expanding the training samples in the training sample set used for network training. Figure 7 As shown, the image processing method may include the following steps S701-S708:

[0122] S701, acquire the target image and the corresponding initial foreground mask image.

[0123] As mentioned above, the initial foreground mask image can be obtained by segmenting the target object in the target image. This segmentation can be performed using a deep learning segmentation network. The training process of the deep learning segmentation network is described below: a training sample set can be obtained, which may include multiple training images; based on... Figure 4a The deep learning segmentation network structure shown can call an X-layer convolutional structure to perform X convolutional processing on the training image to obtain the convolutional feature map of the X-layer convolutional structure. Then, it can call an X-layer deconvolutional structure to perform X upsampling processing on the convolutional feature map of the X-layer convolutional structure to obtain the training foreground mask image corresponding to the training image. The network loss information of the deep learning segmentation network can be determined based on the difference between the training foreground mask image corresponding to the training image and the labeled foreground mask image corresponding to the training image. Then, the network parameters of the deep learning segmentation network can be optimized in the direction of reducing the network loss information of the deep learning segmentation network.

[0124] In this context, network loss information refers to the loss value. The phrase "reducing the network loss information of the deep learning segmentation network" mentioned in this application embodiment refers to a network optimization direction aimed at minimizing the network loss information (i.e., the network loss information is less than or equal to a first loss threshold). In other words, this direction refers to minimizing the network loss information as the goal of reducing the network loss information of the deep learning segmentation network. Optimizing the network parameters of the deep learning segmentation network in this direction, i.e., adjusting the size of the network parameters according to this direction, ensures that the network loss information generated by the deep learning segmentation network after each optimization is less than the network loss information generated by the deep learning segmentation network before optimization. For example, if the network loss information of the deep learning segmentation network calculated in this case is 0.85, then after optimizing the deep learning segmentation network in the direction of reducing network loss information, the network loss information generated by the deep learning segmentation network should be less than 0.85. In addition, when the overall network needs to determine the foreground mask image corresponding to an image containing a target type object (such as a person, bird, or vehicle), the training images in the training sample set can contain the target type object. This helps the deep learning segmentation network quickly learn the ability to segment target type objects from images and accelerates the convergence speed of the deep learning segmentation network.

[0125] After the deep learning segmentation network has been trained (i.e., converged), it can be used to segment the target objects in the target image, obtaining the initial foreground mask image corresponding to the target image. The process of segmenting the target objects in the target image using the trained deep learning segmentation network is similar to the process described above. Figure 3The execution process of step S301 in the illustrated embodiment is the same, and can be found in the above description. Figure 3 Description of step S301 in the illustrated embodiment.

[0126] As mentioned above, the feature learning and optimization processes of the initial foreground mask image can be performed by calling the foreground mask optimization network. Here, we introduce the process of obtaining the training sample set for the foreground mask optimization network. The training sample set can be expanded using the ABMix method, as described below: A training sample set for the foreground mask optimization network can be obtained, which includes at least a first sample image and a second sample image. The first sample image contains a first object, and the second sample image contains a second object. A foreground image containing the first object can be separated from the first sample image, and this foreground image is overlaid with the second sample image to obtain a third sample image. Similarly, a foreground image containing the second object can be separated from the second sample image, and this foreground image is overlaid with the first sample image to obtain a fourth sample image. The training sample set can then be updated based on the third and fourth sample images. The target image can be any image from the updated training sample set used to train the foreground mask optimization network. Expanding the training sample set using the ABMix method described above not only increases the amount of data used to train the foreground mask optimization network, but also makes the model more reliant on the semantic information provided by the preceding deep learning segmentation network, thus decoupling semantic information from the accurate foreground mask image.

[0127] S702, perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image.

[0128] S703 performs M convolutions on the stitched image to obtain M convolutional feature maps.

[0129] S704 performs M upsampling operations based on M convolutional feature maps to obtain M reference foreground mask images.

[0130] S705, determine N reference foreground mask images from M reference foreground mask images.

[0131] S706 performs N-1 optimization processes on N reference foreground mask images to obtain the target foreground mask image corresponding to the target image.

[0132] It should be noted that the processing flow for the target image and initial foreground mask image during the training of the foreground mask optimization network is the same as the processing flow for the target image and initial foreground mask image during the application of the foreground mask optimization network. See the above for details. Figure 3 and the above Figure 5 The illustrated embodiment describes the processing flow of the target image and the initial foreground mask image during the application of the foreground mask optimization network; that is, the execution process of step S702 in this embodiment can be found above. Figure 3 The description of step S302 in the illustrated embodiment and the execution process of step S703 can be found above. Figure 5 The description of step S503 in the illustrated embodiment and the execution process of step S704 can be found above. Figure 5 The description of step S504 in the illustrated embodiment and the execution process of step S705 can be found above. Figure 5 The description of step S506 in the illustrated embodiment, and the execution process of step S706 can be found above. Figure 5 Description of step S506 in the illustrated embodiment.

[0133] S707, obtain network loss information after N-1 optimization processes.

[0134] S708 uses network loss information from N-1 optimization processes for network training.

[0135] In steps S707-S708, the feature learning and optimization processes of the initial foreground mask image can be performed by calling the foreground mask optimization network. That is, the reference foreground mask image and the target foreground mask image are obtained through the foreground mask optimization network; combined with... Figure 8 It can be seen that in the process of optimizing the initial foreground mask image N-1 times based on N reference foreground mask images, the intermediate foreground mask image obtained in each of the N-1 optimization processes can be used to train the foreground mask optimization network. Here, we take the i-th optimization process in the N-1 optimization processes as an example to introduce the model training process based on the i-th intermediate foreground mask image obtained in the i-th optimization process. The process of training the foreground mask optimization network based on the i-th intermediate foreground mask image obtained in the N-1 optimization processes can include: obtaining the labeled foreground mask image (GroundTruth1, which can be represented as α) corresponding to the i-th intermediate foreground mask image. gt The network loss information for the i-th optimization process is determined based on the i-th intermediate foreground mask image, the labeled foreground mask image, and the labeled original image. The network parameters of the foreground mask optimization network are optimized in the direction of reducing the network loss information of the i-th optimization process.

[0136] Similar to the deep learning segmentation networks described above, the network loss information, or loss value, is used in this application. The phrase "in the direction of reducing the network loss information in the i-th optimization process" refers to a network optimization direction aimed at minimizing the network loss information (i.e., the network loss information is less than or equal to the second loss threshold). In other words, this direction refers to minimizing the network loss information as the goal of reducing the network loss information in the i-th optimization process. The network parameters of the foreground mask optimization network are optimized in this direction, meaning the size of the network parameters is adjusted accordingly, so that the network loss information generated by the foreground mask optimization network after each optimization is less than the network loss information generated before optimization. For example, if the calculated network loss information of the foreground mask optimization network is 0.85, then after optimizing the foreground mask optimization network in the direction of reducing network loss information, the network loss information generated by the foreground mask optimization network should be less than 0.85.

[0137] The network loss information for the i-th optimization process, determined based on the i-th intermediate foreground mask image, the labeled foreground mask image, and the labeled original image, may include:

[0138] (1) Based on the difference between the i-th intermediate foreground mask image and the labeled foreground mask image, the first loss information (Alpha Loss) is determined, which is L1 Loss. Specifically, the difference between the pixel value of each pixel in the i-th intermediate foreground mask image and the pixel value of the corresponding pixel in the labeled foreground mask image can be calculated, and then the first loss information can be calculated based on the difference between each pixel. The calculation process of the first loss information can be found in the following formula 4:

[0139]

[0140] The parameters in Formula 4 above are explained below: L a H can represent the first loss information (Alpha Loss); H can represent the number of pixels contained in the i-th intermediate foreground mask image and the labeled foreground mask image, where H is an integer greater than 1; α h α can represent the pixel value of the h-th pixel in the i-th intermediate foreground mask image, where h is a positive integer less than or equal to H; gt h It can represent the pixel value of the h-th pixel in the annotated foreground mask image.

[0141] (2) Perform image fusion processing on the i-th intermediate foreground mask image and the labeled original image to obtain the third fused image. Perform image fusion processing on the labeled foreground mask image and the labeled original image to obtain the fourth fused image. Based on the difference between the third fused image and the fourth fused image, determine the second loss information (Composition Loss). The second loss information is the L1 Loss under the constraint of the labeled original image.

[0142] Specifically, as mentioned above, the original labeled image is obtained by downsampling or scaling the target image. That is, the original labeled image can include an image with three channels: the R channel, the G channel, and the B channel. The i-th intermediate foreground mask image and the labeled foreground mask are both single-channel images. To perform image fusion processing before and between the i-th intermediate foreground mask image and the original labeled image, and to perform image fusion processing between the labeled foreground mask image and the original labeled image, channel expansion can be performed on both the i-th intermediate foreground mask image and the labeled foreground mask image. The i-th intermediate foreground mask image can be expanded to a three-channel image, and the labeled foreground mask image can also be expanded to a three-channel image. Then, the i-th intermediate foreground mask image can be further expanded to a three-channel image. Image fusion is performed pairwise between i intermediate foreground mask images and the original labeled images, resulting in a third fused image containing three channels. Similarly, image fusion is performed pairwise between the original labeled foreground mask images and the original labeled images, resulting in a fourth fused image containing three channels. Then, the first difference between the first channel images of the third fused image and the first channel images of the fourth fused image can be calculated, the second difference between the second channel images of the third fused image and the second channel images of the fourth fused image can be calculated, and the third difference between the third channel images of the third fused image and the third channel images of the fourth fused image can be calculated. Based on the first, second, and third differences, second loss information is calculated. Taking the first channel image as an example, the process of calculating the first difference between the first channel images of the third fused image and the first channel images of the fourth fused image can include: calculating the difference between the pixel values ​​of each pixel in the first channel image of the third fused image and the corresponding pixel values ​​in the first channel image of the fourth fused image, and then calculating the first difference based on the differences between each pixel. The formula for calculating the second loss information is shown in Formula 5 below.

[0143]

[0144] The parameters in Formula 5 above are explained below: L compIt can represent the second loss information (CompositionLoss); H can represent the number of pixels contained in the i-th intermediate foreground mask image and the labeled foreground mask image, 3 indicates 3 channels, and H is an integer greater than 1; α hc This can represent the pixel value of the h-th pixel in the c-th channel of the i-th intermediate foreground mask image, where h is a positive integer less than or equal to H, and c is a positive integer less than or equal to 3; I hc α can represent the pixel value of the h-th pixel in the c-th channel of the original image; gt hc It can represent the pixel value of the h-th pixel in the c-th channel of the annotated foreground mask image.

[0145] (3) Perform gradient processing on the i-th intermediate foreground mask image and the labeled foreground mask image. Based on the difference between the i-th intermediate foreground mask image after gradient processing and the labeled foreground mask image after gradient processing, determine the third loss information (Gradient Loss), which is L2 Loss. Specifically, the difference between the pixel value of each pixel in the i-th intermediate foreground mask image after gradient processing and the pixel value of the corresponding pixel in the labeled foreground mask image after gradient processing can be calculated. Then, the third loss information can be calculated based on the difference between each pixel. The calculation process of the third loss information can be found in Formula 6 below:

[0146]

[0147] The parameters in Formula 6 above are explained below: L grad G(α) can represent the third loss information (Gradient Loss); H can represent the number of pixels contained in the i-th intermediate foreground mask image and the labeled foreground mask image, where H is an integer greater than 1; h G(α) can represent the pixel value of the h-th pixel in the i-th intermediate foreground mask image after gradient processing, where h is a positive integer less than or equal to H; gt ) h S and Sh can represent the pixel value of the h-th pixel in the annotated foreground mask image after gradient processing; TThese are the Sobel operators in the x and y directions, respectively. The Sobel operator is an important processing method in computer vision, mainly used to obtain the first-order gradient of an image. Its common application and physical significance is edge detection. Gradient Loss extracts edge features from the i-th intermediate foreground mask image and the labeled foreground mask image using the gradient operator. It then uses the extracted edge features to apply L2 Loss constraints, making the predicted foreground region edge of the i-th intermediate foreground mask image closer to the foreground region edge of the labeled foreground mask image.

[0148] (4) Determine the network loss information for the i-th optimization process based on the first loss information, the second loss information, and the third loss information. The calculation process for the network loss information for the i-th optimization process can be found in Formula 7 below:

[0149] L = L a +L comp +L grad Formula 7;

[0150] The parameters in Formula 7 above are explained below. L can represent the network loss information of the i-th optimization process; L a This can represent the first loss information (Alpha Loss); L comp It can represent the second loss information (Composition Loss); L grad It can represent third-order loss information (Gradient Loss).

[0151] In this embodiment, during the training of the deep learning segmentation network, training samples can be selected based on the training objective (i.e., determining the foreground mask image corresponding to the image containing the target type object). This helps the deep learning segmentation network quickly learn the ability to segment target type objects from images, accelerating the convergence speed of the deep learning segmentation network. Secondly, the training sample set is expanded using the ABMix method, which not only increases the amount of data used to train the foreground mask optimization network, but also makes the model more dependent on the semantic information provided by the preceding deep learning segmentation network, achieving decoupling between semantic information and accurate foreground mask images. In addition, a gradient operator is introduced during the optimization of the network loss information of the foreground mask optimization network. The gradient operator uses the extracted edge features for L2 Loss constraints, making the predicted foreground region edges closer to the labeled foreground region edges.

[0152] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0153] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The image processing apparatus can be disposed in the computer device provided in this embodiment of the application. The image processing apparatus can be a computer program (including program code) running in the computer device, and can be used to execute... Figure 3 , Figure 5 or Figure 7 The corresponding steps in the method embodiments shown. Please refer to... Figure 9 The image processing apparatus may include the following units:

[0154] The acquisition unit 901 is used to acquire a target image and an initial foreground mask image corresponding to the target image; the target image contains a target object, and the initial foreground mask image is obtained by segmenting the target object in the target image;

[0155] The processing unit 902 is used to perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image; to perform feature learning on the stitched image to obtain a reference foreground mask image corresponding to the target image; to perform optimization processing on the reference foreground mask image to obtain a target foreground mask image corresponding to the target image; the target foreground mask image contains a target foreground region corresponding to the target object, and the target foreground region is used to indicate the location of the target object in the target image.

[0156] In one implementation, the number of reference foreground mask images used for optimization processing is N, and the number of optimization processing operations is N-1, where N is an integer greater than 1; the processing unit 902, when optimizing the reference foreground mask images to obtain the target foreground mask image corresponding to the target image, specifically performs the following steps:

[0157] The first and second reference foreground masks from the N reference foreground mask images are optimized for the first time to obtain the first intermediate foreground mask image. The third reference foreground mask image and the first intermediate foreground mask image from the N reference foreground mask images are optimized for the second time to obtain the second intermediate foreground mask image. Subsequent optimization processes are performed until the (N-1)th optimization process is performed to obtain the (N-1)th intermediate foreground mask image. The (N-1)th intermediate foreground mask image is determined as the target foreground mask image corresponding to the target image.

[0158] In one implementation, when processing unit 902 performs a first optimization process on the first reference foreground mask image and the second reference foreground mask image among N reference foreground mask images to obtain the first intermediate foreground mask image, it specifically performs the following steps:

[0159] Based on the image size of the second reference foreground mask image, the first reference foreground mask image is upsampled to obtain the upsampled first reference foreground mask image; the upsampled first reference foreground mask image is then transformed to obtain a transformed foreground mask image, which includes ambiguous regions; a transformed background mask image corresponding to the transformed foreground mask image is determined; the upsampled first reference foreground mask image and the transformed background mask image are then fused to obtain a first fused image; the second reference foreground mask image and the transformed foreground mask image are then fused to obtain a second fused image; and the first fused image and the second fused image are then overlaid to obtain a first intermediate foreground mask image.

[0160] In one implementation, the reference foreground mask image and the target foreground mask image are obtained through a foreground mask optimization network. The intermediate foreground mask image obtained in each of the N-1 optimization processes is used to train the foreground mask optimization network. The processing unit 902, when training the foreground mask optimization network based on the i-th intermediate foreground mask image obtained in the i-th optimization process of the N-1 optimization processes, specifically performs the following steps:

[0161] Obtain the labeled foreground mask image and labeled original image corresponding to the i-th intermediate foreground mask image, where i is a positive integer less than or equal to N-1; determine the network loss information for the i-th optimization process based on the i-th intermediate foreground mask image, labeled foreground mask image, and labeled original image; optimize the network parameters of the foreground mask optimization network in the direction of reducing network loss information.

[0162] In one implementation, the processing unit 902, when determining the network loss information for the i-th optimization process based on the i-th intermediate foreground mask image, the labeled foreground mask image, and the labeled original image, specifically performs the following steps:

[0163] Based on the difference between the i-th intermediate foreground mask image and the labeled foreground mask image, the first loss information is determined; image fusion processing is performed on the i-th intermediate foreground mask image and the labeled original image to obtain the third fused image, and image fusion processing is performed on the labeled foreground mask image and the labeled original image to obtain the fourth fused image, and the second loss information is determined based on the difference between the third fused image and the fourth fused image; gradient processing is performed on the i-th intermediate foreground mask image and the labeled foreground mask image, and the third loss information is determined based on the difference between the gradient-processed i-th intermediate foreground mask image and the gradient-processed labeled foreground mask image; based on the first loss information, the second loss information, and the third loss information, the network loss information for the i-th optimization process is determined.

[0164] In one implementation, the reference foreground mask image and the target foreground mask image are obtained through a foreground mask optimization network; the processing unit 902 is further configured to perform the following steps:

[0165] Obtain a training sample set for the foreground mask optimization network. The training sample set includes at least a first sample image and a second sample image. The first sample image contains a first object, and the second sample image contains a second object. Separate the foreground image containing the first object from the first sample image, and overlay the foreground image containing the first object with the second sample image to obtain a third sample image. Separate the foreground image containing the second object from the second sample image, and overlay the foreground image containing the second object with the first sample image to obtain a fourth sample image. Update the training sample set based on the third and fourth sample images. The target image is any image in the updated training sample set.

[0166] In one implementation, the number of reference foreground mask images used for optimization processing is N, where N is an integer greater than 1; the processing unit 902, when performing feature learning on the stitched image to obtain the reference foreground mask image corresponding to the target image, specifically performs the following steps:

[0167] M convolutions are performed on the stitched image to obtain M convolutional feature maps; M upsampling processes are performed on the M convolutional feature maps to obtain M reference foreground mask images; N reference foreground mask images are determined from the M reference foreground mask images, and the N reference foreground mask images include at least the reference foreground mask image with the largest image size among the M reference foreground mask images, where M is an integer greater than or equal to N.

[0168] In one implementation, when processing unit 902 performs M convolutional processes on the stitched image to obtain M convolutional feature maps, it specifically executes the following steps:

[0169] The stitched image is subjected to a first convolution process to obtain a convolution feature map of the first convolution process; the convolution feature map of the first convolution process is subjected to a second convolution process to obtain a convolution feature map of the second convolution process; subsequent convolution processes are continued until the convolution feature map of the Mth convolution process is obtained.

[0170] In one implementation, the M-th convolutional processing includes the j-th convolutional processing. Before performing the j-th convolutional processing, the processing unit 902 is further configured to perform the following steps: standardize the initial foreground mask image to obtain a standard foreground mask image.

[0171] When processing unit 902 performs the j-th convolution processing, it specifically performs the following steps: image superposition processing is performed on the standard foreground mask image and the convolution feature map of the (j-1)-th convolution processing to obtain a standard convolution superimposed image; convolution processing is performed on the standard convolution superimposed image to obtain the convolution feature map of the j-th convolution processing, where j is a positive integer greater than 1 and less than or equal to M.

[0172] In one implementation, the processing unit 902 is used to perform normalization processing on the initial foreground mask image to obtain a standard foreground mask image, specifically for executing:

[0173] The initial foreground mask image is normalized to obtain a normalized foreground mask image; the initial foreground mask image is convolved to obtain an adjusted feature map; the normalized foreground mask image is adjusted based on the adjusted feature map to obtain a standard foreground mask image.

[0174] In one implementation, the adjusted feature map includes a scaling feature map and a bias adjustment feature map; the processing unit 902, when adjusting the normalized foreground mask image according to the adjusted feature map to obtain a standard foreground mask image, specifically performs the following steps:

[0175] The normalized foreground mask image and the scaling feature map are fused together to obtain the first scaling feature map; the first scaling feature map and the offset scaling feature map are overlaid to obtain the standard foreground mask image.

[0176] In one implementation, when processing unit 902 performs M upsampling operations based on M convolutional feature maps to obtain M reference foreground mask images, it specifically performs the following steps:

[0177] The convolution feature map of the Mth convolution process is upsampled for the first time to obtain the reference foreground mask image of the first upsampled process. The convolution feature map of the (M-1)th convolution process and the reference foreground mask image of the first upsampled process are overlaid to obtain the reference convolution overlay image. The reference convolution overlay image is then upsampled for the second time to obtain the reference foreground mask image of the second upsampled process. Subsequent upsampling processes are continued until the reference foreground mask image of the Mth upsampled process is obtained.

[0178] In one implementation, the initial foreground mask image includes an initial foreground region corresponding to the target object; the processing unit 902 is used to perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image, specifically by performing the following steps:

[0179] The image cropping region is determined based on the initial foreground region; the target image is cropped based on the image cropping region to obtain the cropped target image; the initial foreground mask image is cropped based on the image cropping region to obtain the cropped initial foreground mask image; the cropped target image and the cropped initial foreground mask image are stitched together to obtain the stitched image.

[0180] In one implementation, the processing unit 902 is used to optimize the reference foreground mask image to obtain the target foreground mask image corresponding to the target image, and then performs the following steps:

[0181] Determine the target background mask image corresponding to the target foreground mask image; perform image separation processing on the target image based on the target foreground mask image to obtain a foreground image containing the target object; obtain a reference image, and perform image separation processing on the reference image based on the target background mask image to obtain a background image; perform image overlay processing on the foreground image containing the target object and the background image to obtain a composite image.

[0182] According to another embodiment of this application, Figure 9 The various units in the image processing apparatus shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0183] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 3 , Figure 5 or Figure 7 The computer program (including program code) involved in each step of the corresponding method shown is used to construct, for example... Figure 9 The image processing apparatus shown herein, and the image processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and executed therein.

[0184] In this embodiment, a target image and an initial foreground mask image corresponding to the target image can be obtained. The initial foreground mask image is obtained by segmenting the target object in the target image. Next, the target image and the initial foreground mask image can be stitched together, and feature learning can be performed on the stitched image to obtain a reference foreground mask image corresponding to the target image. Then, the reference foreground mask image can be optimized. The target foreground region in the optimized target foreground mask image can be used to accurately indicate the location of the target object in the target image. That is, after performing feature learning on the stitched image obtained by stitching the target image and the initial foreground mask image to obtain the reference foreground mask image, the accuracy of the target foreground mask region obtained by optimizing the reference foreground mask image is improved compared to both the reference foreground mask image and the initial foreground mask region. Thus, this embodiment can accurately determine the foreground mask image corresponding to the target image.

[0185] Based on the above methods and apparatus embodiments, this application provides a computer device. Please refer to [link to relevant documentation]. Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 10 The computer device shown includes at least a processor 1001, an input interface 1002, an output interface 1003, and a computer-readable storage medium 1004. The processor 1001, input interface 1002, output interface 1003, and computer-readable storage medium 1004 can be connected via a bus or other means.

[0186] The input interface 1002 can be used to acquire the target image and the corresponding initial foreground mask image, acquire the reference image, etc.; the output interface 1003 can be used to output the target foreground mask image, the composite image, etc.

[0187] The computer-readable storage medium 1004 can be stored in the memory of a computer device. The computer-readable storage medium 1004 is used to store computer programs, including computer instructions. The processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1004. The processor 1001 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, suitable for implementing one or more computer instructions, specifically suitable for loading and executing one or more computer instructions to achieve corresponding method flows or corresponding functions.

[0188] This application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the computer device. Furthermore, the storage space also stores one or more computer instructions suitable for loading and execution by a processor. These computer instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0189] In one embodiment, the processor 1001 may load and execute one or more computer instructions stored in the computer-readable storage medium 1004 to implement the aforementioned related... Figure 3 , Figure 5 or Figure 7 The corresponding steps of the image processing method shown. In a specific implementation, the computer instructions in the computer-readable storage medium 1004 are loaded by the processor 1001 and executed as follows:

[0190] Obtain the target image and its corresponding initial foreground mask image; the target image contains a target object, and the initial foreground mask image is obtained by segmenting the target object in the target image;

[0191] The target image and the initial foreground mask image are stitched together to obtain a stitched image. Feature learning is performed on the stitched image to obtain a reference foreground mask image corresponding to the target image. The reference foreground mask image is optimized to obtain a target foreground mask image corresponding to the target image. The target foreground mask image contains the target foreground region corresponding to the target object, and the target foreground region is used to indicate the location of the target object in the target image.

[0192] In one implementation, the number of reference foreground mask images used for optimization processing is N, and the number of optimization processing operations is N-1, where N is an integer greater than 1. When the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to optimize the reference foreground mask images and obtain the target foreground mask image corresponding to the target image, the following steps are specifically performed:

[0193] The first and second reference foreground masks from the N reference foreground mask images are optimized for the first time to obtain the first intermediate foreground mask image. The third reference foreground mask image and the first intermediate foreground mask image from the N reference foreground mask images are optimized for the second time to obtain the second intermediate foreground mask image. Subsequent optimization processes are performed until the (N-1)th optimization process is performed to obtain the (N-1)th intermediate foreground mask image. The (N-1)th intermediate foreground mask image is determined as the target foreground mask image corresponding to the target image.

[0194] In one implementation, when the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to perform a first optimization process on the first reference foreground mask image and the second reference foreground mask image among N reference foreground mask images to obtain the first intermediate foreground mask image, the instructions specifically perform the following steps:

[0195] Based on the image size of the second reference foreground mask image, the first reference foreground mask image is upsampled to obtain the upsampled first reference foreground mask image; the upsampled first reference foreground mask image is then transformed to obtain a transformed foreground mask image, which includes ambiguous regions; a transformed background mask image corresponding to the transformed foreground mask image is determined; the upsampled first reference foreground mask image and the transformed background mask image are then fused to obtain a first fused image; the second reference foreground mask image and the transformed foreground mask image are then fused to obtain a second fused image; and the first fused image and the second fused image are then overlaid to obtain a first intermediate foreground mask image.

[0196] In one implementation, the reference foreground mask image and the target foreground mask image are obtained through a foreground mask optimization network. The intermediate foreground mask images obtained in each of the N-1 optimization processes are used to train the foreground mask optimization network. The computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001. When training the foreground mask optimization network based on the i-th intermediate foreground mask image obtained in the i-th optimization process of the N-1 optimization processes, the following steps are specifically performed:

[0197] Obtain the labeled foreground mask image and labeled original image corresponding to the i-th intermediate foreground mask image, where i is a positive integer less than or equal to N-1; determine the network loss information for the i-th optimization process based on the i-th intermediate foreground mask image, labeled foreground mask image, and labeled original image; optimize the network parameters of the foreground mask optimization network in the direction of reducing network loss information.

[0198] In one implementation, when the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to determine the network loss information for the i-th optimization process based on the i-th intermediate foreground mask image, the labeled foreground mask image, and the labeled original image, they are specifically used to perform the following steps:

[0199] Based on the difference between the i-th intermediate foreground mask image and the labeled foreground mask image, the first loss information is determined; image fusion processing is performed on the i-th intermediate foreground mask image and the labeled original image to obtain the third fused image, and image fusion processing is performed on the labeled foreground mask image and the labeled original image to obtain the fourth fused image, and the second loss information is determined based on the difference between the third fused image and the fourth fused image; gradient processing is performed on the i-th intermediate foreground mask image and the labeled foreground mask image, and the third loss information is determined based on the difference between the gradient-processed i-th intermediate foreground mask image and the gradient-processed labeled foreground mask image; based on the first loss information, the second loss information, and the third loss information, the network loss information for the i-th optimization process is determined.

[0200] In one implementation, the reference foreground mask image and the target foreground mask image are obtained through a foreground mask optimization network; the computer instructions in the computer-readable storage medium 1004 are loaded by the processor 1001 and are also used to perform the following steps:

[0201] Obtain a training sample set for the foreground mask optimization network. The training sample set includes at least a first sample image and a second sample image. The first sample image contains a first object, and the second sample image contains a second object. Separate the foreground image containing the first object from the first sample image, and overlay the foreground image containing the first object with the second sample image to obtain a third sample image. Separate the foreground image containing the second object from the second sample image, and overlay the foreground image containing the second object with the first sample image to obtain a fourth sample image. Update the training sample set based on the third and fourth sample images. The target image is any image in the updated training sample set.

[0202] In one implementation, the number of reference foreground mask images used for optimization processing is N, where N is an integer greater than 1; when the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to perform feature learning on the stitched image to obtain the reference foreground mask image corresponding to the target image, they are specifically used to perform the following steps:

[0203] M convolutions are performed on the stitched image to obtain M convolutional feature maps; M upsampling processes are performed on the M convolutional feature maps to obtain M reference foreground mask images; N reference foreground mask images are determined from the M reference foreground mask images, and the N reference foreground mask images include at least the reference foreground mask image with the largest image size among the M reference foreground mask images, where M is an integer greater than or equal to N.

[0204] In one implementation, when the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to perform M convolutional processes based on the stitched image to obtain M convolutional feature maps, the instructions specifically perform the following steps:

[0205] The stitched image is subjected to a first convolution process to obtain a convolution feature map of the first convolution process; the convolution feature map of the first convolution process is subjected to a second convolution process to obtain a convolution feature map of the second convolution process; subsequent convolution processes are continued until the convolution feature map of the Mth convolution process is obtained.

[0206] In one implementation, the M-th convolutional processing includes the j-th convolutional processing. Before performing the j-th convolutional processing, the processing unit is further configured to perform the following steps: standardize the initial foreground mask image to obtain a standard foreground mask image.

[0207] When the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 for the j-th convolution process, they are specifically used to perform the following steps: performing image overlay processing on the standard foreground mask image and the convolution feature map of the (j-1)-th convolution process to obtain a standard convolution overlay image; performing convolution processing on the standard convolution overlay image to obtain the convolution feature map of the j-th convolution process, where j is a positive integer greater than 1 and less than or equal to M.

[0208] In one implementation, when the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to standardize the initial foreground mask image to obtain a standard foreground mask image, the instructions specifically perform the following steps:

[0209] The initial foreground mask image is normalized to obtain a normalized foreground mask image; the initial foreground mask image is convolved to obtain an adjusted feature map; the normalized foreground mask image is adjusted based on the adjusted feature map to obtain a standard foreground mask image.

[0210] In one implementation, the adjusted feature map includes a scaling feature map and a biasing feature map; when the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to adjust the normalized foreground mask image according to the adjusted feature map to obtain a standard foreground mask image, the instructions specifically perform the following steps:

[0211] The normalized foreground mask image and the scaling feature map are fused together to obtain the first scaling feature map; the first scaling feature map and the offset scaling feature map are overlaid to obtain the standard foreground mask image.

[0212] In one implementation, when the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to perform M upsampling processes based on M convolutional feature maps to obtain M reference foreground mask images, the instructions specifically perform the following steps:

[0213] The convolution feature map of the Mth convolution process is upsampled for the first time to obtain the reference foreground mask image of the first upsampled process. The convolution feature map of the (M-1)th convolution process and the reference foreground mask image of the first upsampled process are overlaid to obtain the reference convolution overlay image. The reference convolution overlay image is then upsampled for the second time to obtain the reference foreground mask image of the second upsampled process. Subsequent upsampling processes are continued until the reference foreground mask image of the Mth upsampled process is obtained.

[0214] In one implementation, the initial foreground mask image includes an initial foreground region corresponding to the target object; the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to perform image stitching processing on the target image and the initial foreground mask image to obtain a stitched image, specifically for performing the following steps:

[0215] The image cropping region is determined based on the initial foreground region; the target image is cropped based on the image cropping region to obtain the cropped target image; the initial foreground mask image is cropped based on the image cropping region to obtain the cropped initial foreground mask image; the cropped target image and the cropped initial foreground mask image are stitched together to obtain the stitched image.

[0216] In one implementation, the computer instructions in the computer-readable storage medium 1004 are loaded and executed by the processor 1001 to optimize the reference foreground mask image to obtain the target foreground mask image corresponding to the target image. They are then further used to perform the following steps:

[0217] Determine the target background mask image corresponding to the target foreground mask image; perform image separation processing on the target image based on the target foreground mask image to obtain a foreground image containing the target object; obtain a reference image, and perform image separation processing on the reference image based on the target background mask image to obtain a background image; perform image overlay processing on the foreground image containing the target object and the background image to obtain a composite image.

[0218] In this embodiment, a target image and an initial foreground mask image corresponding to the target image can be obtained. The initial foreground mask image is obtained by segmenting the target object in the target image. Next, the target image and the initial foreground mask image can be stitched together, and feature learning can be performed on the stitched image to obtain a reference foreground mask image corresponding to the target image. Then, the reference foreground mask image can be optimized. The target foreground region in the optimized target foreground mask image can be used to accurately indicate the location of the target object in the target image. That is, after performing feature learning on the stitched image obtained by stitching the target image and the initial foreground mask image to obtain the reference foreground mask image, the accuracy of the target foreground mask region obtained by optimizing the reference foreground mask image is improved compared to both the reference foreground mask image and the initial foreground mask region. Thus, this embodiment can accurately determine the foreground mask image corresponding to the target image.

[0219] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing methods provided in the various alternative embodiments described above.

[0220] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a target image and an initial foreground mask image corresponding to the target image; the target image contains a target object, and the initial foreground mask image is obtained by performing segmentation processing on the target object in the target image; performing image stitching processing on the target image and the initial foreground mask image to obtain a stitched image; performing M times of convolution processing based on the stitched image to obtain M convolution feature maps; performing M times of up-sampling processing based on the M convolution feature maps to obtain M reference foreground mask images; determining N reference foreground mask images from the M reference foreground mask images, N being an integer greater than 1 and M being an integer greater than or equal to N; performing first optimization processing on a first reference foreground mask image and a second reference foreground mask image in the N reference foreground mask images to obtain a first intermediate foreground mask image, the first optimization processing being used for fusing and superimposing a foreground region with high confidence in the first reference foreground mask image and an ambiguous region with low confidence in the second reference foreground mask image; performing second optimization processing on a third reference foreground mask image in the N reference foreground mask images and the first intermediate foreground mask image to obtain a second intermediate foreground mask image; and continuously performing subsequent optimization processing until an N-1 intermediate foreground mask image is obtained through N-1 times of optimization processing; and determining the N-1 intermediate foreground mask image as a target foreground mask image corresponding to the target image, the target foreground mask image containing a target foreground region corresponding to the target object, the target foreground region being used for indicating a region of the target object in the target image.

2. The method of claim 1, wherein, The first optimization processing on the first reference foreground mask image and the second reference foreground mask image in the N reference foreground mask images to obtain the first intermediate foreground mask image comprises: performing up-sampling processing on the first reference foreground mask image according to an image size of the second reference foreground mask image to obtain an up-sampled first reference foreground mask image; performing image transformation processing on the up-sampled first reference foreground mask image to obtain a transformed foreground mask image, the transformed foreground mask image containing an ambiguous region; determining a transformed background mask image corresponding to the transformed foreground mask image; performing image fusion processing on the up-sampled first reference foreground mask image and the transformed background mask image to obtain a first fusion image; performing image fusion processing on the second reference foreground mask image and the transformed foreground mask image to obtain a second fusion image; performing image superimposition processing on the first fusion image and the second fusion image to obtain the first intermediate foreground mask image.

3. The method of claim 1, wherein, The reference foreground mask image and the target foreground mask image are obtained through a foreground mask optimization network, and an intermediate foreground mask image obtained through each optimization processing in the N-1 times of optimization processing is used for training the foreground mask optimization network. The process of training the foreground mask optimization network based on the i-th intermediate foreground mask image obtained in the i-th optimization process in the N-1 optimization processes comprises: Obtaining a labeled foreground mask image and a labeled original image corresponding to the i-th intermediate foreground mask image, i being a positive integer less than or equal to N-1; Determining network loss information of the i-th optimization process according to the i-th intermediate foreground mask image, the labeled foreground mask image and the labeled original image; Optimizing network parameters of the foreground mask optimization network in a direction of reducing the network loss information.

4. The method of claim 3, wherein, The process of determining the network loss information of the i-th optimization process according to the i-th intermediate foreground mask image, the labeled foreground mask image and the labeled original image comprises: Determining first loss information based on a difference between the i-th intermediate foreground mask image and the labeled foreground mask image; Performing image fusion processing on the i-th intermediate foreground mask image and the labeled original image to obtain a third fusion image, performing image fusion processing on the labeled foreground mask image and the labeled original image to obtain a fourth fusion image, and determining second loss information based on a difference between the third fusion image and the fourth fusion image; Performing gradient processing on the i-th intermediate foreground mask image and the labeled foreground mask image, and determining third loss information based on a difference between the gradient-processed i-th intermediate foreground mask image and the gradient-processed labeled foreground mask image; Determining the network loss information of the i-th optimization process according to the first loss information, the second loss information and the third loss information.

5. The method of claim 1, wherein, The reference foreground mask image and the target foreground mask image are obtained by a foreground mask optimization network; the method further comprises: Obtaining a training sample set of the foreground mask optimization network, the training sample set comprising at least a first sample image and a second sample image, the first sample image containing a first object, and the second sample image containing a second object; Separating a foreground image containing the first object from the first sample image, and performing image superposition processing on the foreground image containing the first object and the second sample image to obtain a third sample image; Separating a foreground image containing the second object from the second sample image, and performing image superposition processing on the foreground image containing the second object and the first sample image to obtain a fourth sample image; Updating the training sample set according to the third sample image and the fourth sample image, the target image being any image in the updated training sample set.

6. The method of claim 1, wherein, The N reference foreground mask images comprise at least a reference foreground mask image with the largest image size in the M reference foreground mask images.

7. The method of claim 1, wherein, The process of performing M times of convolution processing based on the spliced image comprises: Performing first-time convolution processing on the spliced image to obtain a convolution feature map of the first-time convolution processing; performing second convolution processing on the convolution feature map of the first convolution processing to obtain a convolution feature map of the second convolution processing; the convolution processing is continuously performed until a convolution feature map of the Mth convolution processing is obtained.

8. The method of claim 7, wherein, The M convolution processing includes the jth convolution processing, and before the jth convolution processing is performed, the method further includes: performing standardization processing on the initial foreground mask image to obtain a standard foreground mask image; The process of the jth convolution processing includes: performing image superposition processing on the standard foreground mask image and the convolution feature map of the j-1th convolution processing to obtain a standard convolution superposition image; and performing convolution processing on the standard convolution superposition image to obtain the convolution feature map of the jth convolution processing, where j is a positive integer greater than 1 and less than or equal to M.

9. The method of claim 8, wherein, The standardization processing on the initial foreground mask image to obtain a standard foreground mask image includes: performing normalization processing on the initial foreground mask image to obtain a normalized foreground mask image; performing convolution processing on the initial foreground mask image to obtain an adjustment feature map; adjusting the normalized foreground mask image according to the adjustment feature map to obtain a standard foreground mask image.

10. The method of claim 9, wherein, The adjustment feature map includes a proportional adjustment feature map and a bias adjustment feature map; and the adjusting the normalized foreground mask image according to the adjustment feature map to obtain a standard foreground mask image includes: performing image fusion processing on the normalized foreground mask image and the proportional adjustment feature map to obtain a first adjustment feature map; performing image superposition processing on the first adjustment feature map and the bias adjustment feature map to obtain the standard foreground mask image.

11. The method of claim 7, wherein, The M times of upsampling processing based on the M convolution feature maps to obtain M reference foreground mask images includes: performing first upsampling processing on the convolution feature map of the Mth convolution processing to obtain a reference foreground mask image of the first upsampling processing; performing image superposition processing on the convolution feature map of the M-1th convolution processing and the reference foreground mask image of the first upsampling processing to obtain a reference convolution superposition image, and performing second upsampling processing on the reference convolution superposition image to obtain a reference foreground mask image of the second upsampling processing; the subsequent times of upsampling processing are continuously performed until a reference foreground mask image of the Mth upsampling processing is obtained.

12. The method of claim 1, wherein, The initial foreground mask image includes an initial foreground region corresponding to the target object; The image splicing processing on the target image and the initial foreground mask image to obtain a spliced image includes: determining an image cropping region according to the initial foreground region; performing image cropping processing on the target image according to the image cropping region to obtain a cropped target image; performing image cropping processing on the initial foreground mask image according to the image cropping region to obtain a cropped initial foreground mask image; performing image splicing processing on the cropped target image and the cropped initial foreground mask image to obtain the spliced image.

13. The method of claim 1, wherein, After the optimization processing on the reference foreground mask image is performed to obtain the target foreground mask image corresponding to the target image, the method further includes: determining a target background mask image corresponding to the target foreground mask image; performing image separation processing on the target image according to the target foreground mask image to obtain a foreground image containing the target object; obtaining a reference image and performing image separation processing on the reference image according to the target background mask image to obtain a background image; performing image superposition processing on the foreground image containing the target object and the background image to obtain a composite image.

14. An image processing apparatus characterized by comprising: The image processing apparatus includes: an acquisition unit configured to acquire a target image and an initial foreground mask image corresponding to the target image, the target image containing a target object, and the initial foreground mask image being obtained by performing segmentation processing on the target object in the target image; a processing unit configured to perform image splicing processing on the target image and the initial foreground mask image to obtain a spliced image, perform M times of convolution processing based on the spliced image to obtain M convolution feature maps, perform M times of up-sampling processing based on the M convolution feature maps to obtain M reference foreground mask images, determine N reference foreground mask images from the M reference foreground mask images, N being an integer greater than 1 and M being an integer greater than or equal to N, perform first-time optimization processing on a first reference foreground mask image and a second reference foreground mask image in the N reference foreground mask images to obtain a first intermediate foreground mask image, the first-time optimization processing being configured to perform fusion superposition on a foreground region with high confidence in the first reference foreground mask image and an ambiguous region with low confidence in the second reference foreground mask image, perform second-time optimization processing on a third reference foreground mask image in the N reference foreground mask images and the first intermediate foreground mask image to obtain a second intermediate foreground mask image, and continue to perform subsequent-time optimization processing until an N-1 intermediate foreground mask image is obtained through N-1 times of optimization processing, and determine the N-1 intermediate foreground mask image as a target foreground mask image corresponding to the target image, the target foreground mask image containing a target foreground region corresponding to the target object, and the target foreground region being configured to indicate a region of the target object in the target image.

15. A computer device, comprising: The computer device includes: a processor adapted to implement a computer program; a computer readable storage medium storing a computer program, the computer program being adapted to be loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a computer program, the computer program being adapted to be loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 13.

17. A computer program product, characterised in that, The computer program product includes computer instructions, the computer instructions being executed by the processor to implement the image processing method according to any one of claims 1 to 13.

Images