Image processing method and device, image processing model training method and device

Abstract

The present application relates to an image processing method, device, computer equipment, storage medium and computer program product, which are applied to cloud technology, artificial intelligence, smart transportation, assisted driving, conference, live broadcast, video, art design and other scenarios. The method includes: performing feature encoding processing on the original image to obtain multiple feature maps of different scales, performing a first decoding process based on the feature maps of multiple different scales to obtain a mask image, the mask image is used to segment the target subject from the original image at the semantic level, performing a second decoding process based on the feature maps of multiple different scales and the mask image to obtain multiple candidate mask maps of different levels, updating the candidate mask maps of multiple different levels according to the edge transition features between the target subject and the background area included in each candidate mask map, obtaining a target mask map, and determining the cutout result corresponding to the target subject according to the target mask map. The use of this method can improve the accuracy of cutout.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, storage medium and computer program product, as well as an image processing model training method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, image segmentation technology has emerged, which can separate the subject or background from the original image to meet different usage requirements. For example, a portrait in an image or video can be separated separately to be synthesized into other images or videos.

[0003] Traditional image segmentation technology generally uses deep convolutional neural networks to extract high-level semantic feature information of images, and then uses this high-level semantic information to detect and segment objects. However, relying solely on high-level semantic information for object detection and segmentation can easily lead to the loss of some detailed features and inaccurate image segmentation. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of cutouts in response to the above technical problems.

[0005] In one aspect, the present application provides an image processing method. The method comprises:

[0006] Perform feature encoding processing on the original image to obtain multiple feature maps of different scales;

[0007] Performing a first decoding process based on the multiple feature maps of different scales to obtain a mask image, wherein the mask image is used to segment a target subject from the original image at a semantic level;

[0008] Performing a second decoding process based on the multiple feature maps of different scales and the mask image to obtain multiple candidate mask maps of different levels;

[0009] According to the edge transition features between the target subject and the background area included in each candidate mask image, the candidate mask images at different levels are updated to obtain a target mask image;

[0010] A cutout result corresponding to the target subject is determined according to the target mask image.

[0011] In one embodiment, the present application further provides an image processing device. The device comprises:

[0012] The encoding module is used to perform feature encoding processing on the original image to obtain multiple feature maps of different scales;

[0013] A first decoding module, configured to perform a first decoding process based on the plurality of feature maps of different scales to obtain a mask image, wherein the mask image is used to segment a target subject from the original image at a semantic level;

[0014] A second decoding module, configured to perform a second decoding process based on the plurality of feature maps of different scales and the mask image to obtain a plurality of candidate mask maps of different levels;

[0015] An updating module, configured to update the candidate mask images at different levels according to edge transition features between the target subject and the background area included in each candidate mask image, to obtain a target mask image;

[0016] A result determination module is used to determine a cutout result corresponding to the target subject according to the target mask image.

[0017] In one embodiment, the device is executed by an image processing model, which includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units; wherein the coding structure is used to perform feature coding processing on the original image, the first decoding structure is used to perform first decoding processing, and the second decoding structure is used to perform second decoding processing.

[0018] In one of the embodiments, the encoding module is also used to obtain an original image and input the original image into a coding structure; multiple encoding units in the coding structure perform encoding processing in turn based on the input data corresponding to each encoding unit, to obtain feature maps output by each encoding unit; wherein the input data of the subsequent encoding unit includes the feature map output by the previous encoding unit, the input data of the initial encoding unit in the coding structure is the original image, and the scales of the feature maps output by different encoding units in the coding structure are different.

[0019] In one embodiment, the first decoding module is further used to perform first decoding processing based on the input data corresponding to each of the first decoding units in sequence based on the feature maps of the multiple different scales, through the multiple first decoding units in the first decoding structure, until the mask image is output through the last first decoding unit, wherein the input data of the current first decoding unit in the first decoding structure includes the decoding result corresponding to the previous first decoding unit and the feature map output by the encoding unit corresponding to the current first decoding unit, and the decoding result corresponding to the current first decoding unit is used to constitute the input data of the subsequent first decoding unit.

[0020] In one embodiment, the second decoding module is also used to determine the input data corresponding to each second decoding unit in the second decoding structure, wherein the input data corresponding to the second decoding unit includes the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes the mask image; the second decoding processing is performed in turn based on the input data corresponding to each second decoding unit by multiple second decoding units in the second decoding structure, so as to obtain the candidate mask maps output by each second decoding unit, wherein the levels of the candidate mask maps output by each second decoding unit are different.

[0021] In one embodiment, the updating module is further used to enlarge the candidate mask image of the current level to the same scale as the candidate mask image of the next adjacent level to obtain an enlarged candidate mask image; extract the edge transition image between the target subject and the background area in the enlarged candidate mask image, wherein the edge transition image includes edge transition features; perform fusion processing based on the edge transition image and the candidate mask image of the next adjacent level to obtain an updated mask image; use the updated mask image as the candidate mask image of the current level in the next round, and return to the step of enlarging the candidate mask image of the current level to the same scale as the candidate mask image of the next adjacent level and continue to execute until the updated mask image corresponding to the candidate mask image of the last level is obtained; and use the updated mask image corresponding to the candidate mask image of the last level as the target mask image.

[0022] In one embodiment, the updating module is further used to perform morphological processing on the edge transition image to obtain a processed edge transition image, wherein the morphological processing includes at least one of an erosion operation and a dilation operation;

[0023] The processed edge transition image and the candidate mask image of the adjacent next level are fused to obtain an updated mask image.

[0024] In one embodiment, the second decoding module is further used to perform a second decoding process based on the multiple feature maps of different scales and the mask image to obtain a foreground color image including the target subject;

[0025] The result determination module is further used to fuse the foreground color image and the target mask image to obtain a cutout result corresponding to the target subject.

[0026] In one embodiment, the apparatus is performed by an image processing model, and the apparatus further comprises:

[0027] an acquisition module, configured to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image;

[0028] A model determination module, used to determine an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure;

[0029] A first training module, used for performing a first training on the initial coding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is satisfied, thereby obtaining a trained coding structure and a first decoding structure;

[0030] A second training module is used to perform a second training on the initial second decoding structure through the second sample image set and based on the trained encoding structure and the first decoding structure, until a second stopping condition is met, to obtain a trained second decoding structure, so as to obtain a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

[0031] In one embodiment, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0032] Perform feature encoding processing on the original image to obtain multiple feature maps of different scales;

[0033] Performing a first decoding process based on the multiple feature maps of different scales to obtain a mask image, wherein the mask image is used to segment a target subject from the original image at a semantic level;

[0034] Performing a second decoding process based on the multiple feature maps of different scales and the mask image to obtain multiple candidate mask maps of different levels;

[0035] According to the edge transition features between the target subject and the background area included in each candidate mask image, the candidate mask images at different levels are updated to obtain a target mask image;

[0036] A cutout result corresponding to the target subject is determined according to the target mask image.

[0037] In one embodiment, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0038] Perform feature encoding processing on the original image to obtain multiple feature maps of different scales;

[0039] Performing a first decoding process based on the multiple feature maps of different scales to obtain a mask image, wherein the mask image is used to segment a target subject from the original image at a semantic level;

[0040] Performing a second decoding process based on the multiple feature maps of different scales and the mask image to obtain multiple candidate mask maps of different levels;

[0041] According to the edge transition features between the target subject and the background area included in each candidate mask image, the candidate mask images at different levels are updated to obtain a target mask image;

[0042] A cutout result corresponding to the target subject is determined according to the target mask image.

[0043] In one embodiment, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Perform feature encoding processing on the original image to obtain multiple feature maps of different scales;

[0045] Performing a first decoding process based on the multiple feature maps of different scales to obtain a mask image, wherein the mask image is used to segment a target subject from the original image at a semantic level;

[0046] Performing a second decoding process based on the multiple feature maps of different scales and the mask image to obtain multiple candidate mask maps of different levels;

[0047] According to the edge transition features between the target subject and the background area included in each candidate mask image, the candidate mask images at different levels are updated to obtain a target mask image;

[0048] A cutout result corresponding to the target subject is determined according to the target mask image.

[0049] The above-mentioned image processing method, apparatus, computer equipment, storage medium and computer program product obtain multiple feature maps of different scales by performing feature encoding processing on the original image. The large-scale feature map contains more low-level semantic features, and the small-scale feature map contains more high-level semantic features. The first decoding process is performed based on the feature maps of multiple different scales, and the target subject in the original image can be identified based on the semantic features of different levels, and the position of the target subject in the original image is represented by the mask image at the semantic level to preliminarily segment the target subject. The mask image can provide stronger semantic information, and the second decoding process is performed based on the feature maps and mask images of multiple different scales, which can effectively guide the layer-by-layer interpretation of the edge transition area between the target subject and the background area. Moreover, each candidate mask map has a different level. A high-level candidate mask map contains more edge transition information between the target subject and the background area. According to the edge transition features in each candidate mask map, multiple different levels of candidate mask maps are updated, which can effectively achieve strong attention regression on low-level features, thereby obtaining a target mask map with clearer texture and finer edge transition details of the target subject. Furthermore, the cutout result corresponding to the target subject can be accurately obtained according to the target mask map, further improving the accuracy of the cutout.

[0050] On the other hand, the present application also provides a method for training an image processing model, the method comprising:

[0051] Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image;

[0052] Determining an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure;

[0053] Performing a first training on the initial encoding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure;

[0054] The initial second decoding structure is subjected to second training through the second sample image set and based on the trained encoding structure and the first decoding structure, and the training is stopped when the second stopping condition is met, thereby obtaining a trained second decoding structure and obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

[0055] In one embodiment, the present application also provides a training device for an image processing model. The device comprises:

[0056] an acquisition module, configured to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image;

[0057] A model determination module, used to determine an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure;

[0058] A first training module, configured to perform a first training on the initial coding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is satisfied, thereby obtaining a trained coding structure and a first decoding structure;

[0059] The second training module is used to perform a second training on the initial second decoding structure through the second sample image set and based on the trained encoding structure and the first decoding structure, until the second stopping condition is met, to obtain the trained second decoding structure, so as to obtain a trained image processing model; the trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

[0060] In one embodiment, the first training module is further used to perform feature encoding processing on the first sample image through the initial encoding structure to obtain multiple first feature maps of different scales; perform first decoding processing based on the multiple first feature maps of different scales through the initial first decoding structure to obtain a first mask image corresponding to the first sample image; and train the initial encoding structure and the initial first decoding structure according to the difference between the segmentation labels corresponding to the first mask image and the first sample image until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure.

[0061] In one embodiment, the second training module is further used to perform feature encoding processing on the second sample image through the encoding structure to obtain multiple second feature maps of different scales; perform first decoding processing based on the multiple second feature maps of different scales through the first decoding structure to obtain a second mask image corresponding to the second sample image; perform second decoding processing based on the multiple second feature maps of different scales and the second mask image through the initial second decoding structure to obtain multiple sample mask maps of different levels; update the multiple sample mask maps of different levels based on the sample edge transition features between the sample body and the background area included in each sample mask map through the initial second decoding structure to obtain a predicted mask map; train the initial second decoding structure based on the mask loss between the predicted mask map and the cutout label corresponding to the second sample image until the second stop condition is met to obtain a trained second decoding structure.

[0062] In one embodiment, the mask loss includes a global loss and a scale loss; the second training module is further used to determine the global loss between the predicted mask image and the cutout label corresponding to the second sample image; the predicted mask image is adjusted to different scales, and the scale loss of the predicted mask image at the different scales is determined; based on the global loss and the scale loss, a target loss function is constructed; the initial second decoding structure is trained by the target loss function until a second stopping condition is met, and a trained second decoding structure is obtained.

[0063] In one embodiment, the second training module is further used to determine the mask loss between the second mask image and the segmentation label corresponding to the second sample image; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the mask loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

[0064] In one embodiment, the second sample image set also includes a foreground color label corresponding to the second sample image, and the second training module is further used to perform a second decoding process based on the second feature maps of multiple different scales and the second mask image through the initial second decoding structure to obtain a predicted foreground color image; determine the foreground color loss between the predicted foreground color image and the foreground color label corresponding to the second sample image; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image; construct a target loss function based on the mask loss and the foreground color loss; train the initial second decoding structure through the target loss function until the second stop condition is met, and obtain a trained second decoding structure.

[0065] In one embodiment, the second sample image includes a composite image; the second training module is further used to determine the channel loss of the predicted mask image and the corresponding cutout label corresponding to the composite image on the color channel; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the channel loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

[0066] In one embodiment, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0067] Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image;

[0068] Determining an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure;

[0069] Performing a first training on the initial encoding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure;

[0070] The initial second decoding structure is subjected to second training through the second sample image set and based on the trained encoding structure and the first decoding structure, and the training is stopped when the second stopping condition is met, thereby obtaining a trained second decoding structure and obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

[0071] In one embodiment, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0072] Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image;

[0073] Determining an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure;

[0074] Performing a first training on the initial encoding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure;

[0075] The initial second decoding structure is subjected to second training through the second sample image set and based on the trained encoding structure and the first decoding structure, and the training is stopped when the second stopping condition is met, thereby obtaining a trained second decoding structure and obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

[0076] In one embodiment, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0077] Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image;

[0078] Determining an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure;

[0079] Performing a first training on the initial encoding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure;

[0080] The initial second decoding structure is subjected to second training through the second sample image set and based on the trained encoding structure and the first decoding structure, and the training is stopped when the second stopping condition is met, thereby obtaining a trained second decoding structure and obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

[0081] The training method, apparatus, computer equipment, storage medium and computer program product of the above-mentioned image processing model, by obtaining a first sample image set and a second sample image set, the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image, to determine the image processing model to be trained, the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure, the initial encoding structure and the initial first decoding structure are first trained through the first sample image set until the first stop condition is met, and the trained encoding structure and first decoding structure are obtained, so that on the basis of ensuring the processing accuracy of the encoding structure and the first decoding structure, the second sample image set is used, and the initial second decoding structure is secondly trained based on the trained encoding structure and the first decoding structure, until the second stop condition is met, and the trained second decoding structure is obtained, so that the trained encoding structure, the first decoding structure and the second decoding structure constitute the trained image processing model, which can effectively provide the segmentation accuracy and cutout accuracy of the image processing model, so that the image processing model can accurately determine the cutout result corresponding to the target subject in the original image. Furthermore, by determining the cutout result corresponding to the target subject in the original image through the image processing model, the cutout efficiency can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0083] Figure 2 is a schematic flow chart of an image processing method in one embodiment;

[0084] Figure 3 A schematic diagram of a mask image provided in one embodiment;

[0085] Figure 4 A schematic diagram of a foreground color image provided in another embodiment;

[0086] Figure 5 is a schematic flow chart of an image processing method in one embodiment;

[0087] Figure 6 is an architectural diagram of an image processing model in one embodiment;

[0088] Figure 7 is a schematic diagram of comparison between a mask image and a corresponding edge transition image in one embodiment;

[0089] Figure 8 A schematic diagram of a flow chart of a training method for an image processing model in an embodiment;

[0090] Fig. 9 A schematic diagram showing a comparison between a segmentation result and a cutout result in one embodiment;

[0091] Fig.10 A schematic diagram of a flow chart of segmentation task training in one embodiment;

[0092] Fig.11 A schematic diagram of a process of training a cutout task in one embodiment;

[0093] Fig.12 A schematic diagram showing a comparison between a cutout result obtained by an image processing model in an embodiment and a processing result obtained by a traditional method;

[0094] Fig.13 A schematic diagram for comparing a case without foreground color synthesis and a case with foreground color synthesis in one embodiment;

[0095] Fig.14 is a structural block diagram of an image processing device in one embodiment;

[0096] Fig.15 is a structural block diagram of a training device for an image processing model in one embodiment;

[0097] Fig.16 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0099] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. For example, it is applied to the field of artificial intelligence (AI) technology, where artificial intelligence is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. The solution provided in the embodiments of the present application relates to an image processing method of artificial intelligence, which is specifically described by the following embodiments.

[0100] The image processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 and the server 104 can both independently execute the image processing method provided in the embodiment of the present application. The terminal 102 and the server 104 can also be used in collaboration to execute the image processing method provided in the embodiment of the present application. When the terminal 102 and the server 104 are used in collaboration to execute the image processing method provided in the embodiment of the present application, the terminal 102 obtains the original image and sends the original image to the server 104. The server 104 performs feature encoding processing on the original image to obtain a plurality of feature maps of different scales, performs a first decoding processing based on the feature maps of multiple different scales, and obtains a mask image, and the mask image is used to segment the target subject from the original image at the semantic level. The server 104 performs a second decoding process based on multiple feature maps and mask images of different scales to obtain multiple candidate mask maps at different levels; according to the edge transition features between the target subject and the background area included in each candidate mask map, the candidate mask maps at different levels are updated to obtain the target mask map. The server 104 determines the cutout result corresponding to the target subject according to the target mask map, and the server 104 returns the cutout result to the terminal 102. Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The terminal 102 can run an application, which can be a communication application, an email application, a video application, a music application, an image processing application, etc. The server 104 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 and the server 104 may be connected directly or indirectly via wired or wireless communication, which is not limited in the present application.

[0101] It should be noted that the number of "multiple" etc. mentioned in the embodiments of the present application refers to the number of "at least two".

[0102] In one embodiment, the training method of the image processing model can also be applied to Figure 1In the application environment shown. Both the terminal 102 and the server 104 can independently execute the training method of the image processing model provided in the embodiment of the present application. The terminal 102 and the server 104 can also be used in conjunction to execute the training method of the image processing model provided in the embodiment of the present application. When the terminal 102 and the server 104 are used in conjunction to execute the training method of the image processing model provided in the embodiment of the present application, the terminal 102 obtains a first sample image set and a second sample image set, the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image. The terminal 102 sends the first sample image set and the second sample image set to the server 104. The server 104 determines the image processing model to be trained, and the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure. The server 104 performs a first training on the initial encoding structure and the initial first decoding structure through the first sample image set until the first stop condition is met, and obtains the trained encoding structure and the first decoding structure. The server 104 uses the second sample image set and performs a second training on the initial second decoding structure based on the trained encoding structure and the first decoding structure until the second stopping condition is met, thereby obtaining a trained second decoding structure to obtain a trained image processing model. The trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

[0103] In one embodiment, Figure 2 As shown, an image processing method is provided, which is applied to a computer device (the computer device may be Figure 1 The terminal or server in the example is used to illustrate, including the following steps:

[0104] Step S202: performing feature encoding processing on the original image to obtain a plurality of feature maps of different scales.

[0105] The original image is the image to be cut out, and the original image can be any one of RGB (Red, Green, Blue) images, grayscale images, depth images, images corresponding to the Y component in YUV images, etc., but not limited thereto. In the YUV image, "Y" represents brightness (Luminance or Luma), that is, grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel. The original image can be an image obtained by shooting any scene, such as a person image, a landscape image, or an industrial device image, but not limited thereto. Cutting out refers to separating a part of an image or video from the original image or original video into a separate layer for subsequent image synthesis or video synthesis, such as cutting out the target subject from the original image. Subject refers to various objects, such as people, flowers, cats, dogs, cows, blue sky, white clouds, background, etc. The target subject refers to the required subject, which can be selected according to needs. Salient object detection refers to automatically processing the area of ​​interest and selectively ignoring the area of ​​no interest when facing a scene. The area of ​​interest is called the salient area.

[0106] A feature map is an image obtained by encoding the features of an original image and contains key information of the original image. The original image can be composed of a foreground area and a background area. The foreground area refers to the area where the target subject is located in the original image, and the background area refers to the rest of the area in the original image except the area where the target subject is located.

[0107] Specifically, the computer device may obtain an original image, and perform feature encoding processing on the original image to obtain multiple feature maps of different scales. The feature encoding processing may include convolution processing, and may also include at least one of pooling processing and activation processing. Further, the computer device may perform convolution processing on the original image, and perform the next convolution processing on the feature map obtained by the current convolution processing, and multiple feature maps of different scales may be obtained based on multiple convolution processing. Alternatively, the computer device may perform at least one of pooling processing or activation processing on the feature map obtained by the current convolution processing and then perform the next convolution processing to obtain multiple feature maps of different scales.

[0108] In one embodiment, the feature map obtained by the convolution process may be subjected to pooling process, and the feature map obtained by the pooling process may be subjected to the next convolution process. Based on multiple convolution processes and multiple pooling processes, multiple feature maps of different scales may be obtained.

[0109] In one embodiment, the feature map obtained by the convolution processing can be pooled, and the feature map obtained by the pooling processing can be activated, and the feature map obtained by the activation processing can be subjected to the next convolution processing. Based on multiple convolution processing, multiple pooling processing and multiple activation processing, multiple feature maps of different scales can be obtained.

[0110] It is understandable that pooling is performed after each convolution process, or pooling is performed after multiple convolution processes before the next convolution process is performed. For example, five convolution processes are required, and the feature map of each convolution process is first pooled before the next convolution process, or the feature map obtained after two convolution processes is pooled, and the feature map after the pooling process is then subjected to the next convolution process.

[0111] Similarly, activation processing can be performed after each pooling processing, or activation processing can be performed once after multiple pooling processing before performing the next convolution processing.

[0112] Step S204, performing a first decoding process based on the feature maps of multiple different scales to obtain a mask image, where the mask image is used to segment the target subject from the original image at a semantic level.

[0113] The mask image is an image filter template used to identify the subject in the original image, which can block other parts of the original image and filter out the subject in the image. The first decoding process is to preliminarily identify the target subject from the original image.

[0114] Specifically, the computer device may perform a first decoding process based on a plurality of feature maps of different scales to identify the target subject in the original image and the position of the target subject in the original image. Feature maps of different scales represent semantic features at different levels, and a mask image may be obtained after a first decoding process of the semantic features at different levels, and the mask image may be used to determine the position of the target subject in the original image. The computer device may mask the remaining portion of the original image except the target subject based on the mask image to filter out the target subject in the original image, so that the target subject is preliminarily segmented from the original image at the semantic level through the mask image.

[0115] Step S206, performing a second decoding process based on the feature maps and mask images of multiple different scales to obtain candidate mask maps of multiple different levels.

[0116] Among them, the mask image refers to a single-channel grayscale image, which is the same size as the original image and is used to indicate the position of the subject in the original image. The mask image can be an Alpha image. The area with a value of 1 (corresponding to a pixel value of 255) in the Alpha image represents the foreground subject, the area with a value of 0 (corresponding to a pixel value of 0) represents the background, and the area between 0-1 (corresponding to a pixel value between 0-255) represents the semi-transparent part of the foreground subject, which is generally common in the edges of people, such as hair, hand-held transparent objects, etc. The candidate mask image refers to the mask image obtained by the second decoding process. Multiple candidate mask images of different levels refer to multiple candidate mask images of different levels, and specifically, the scales of multiple candidate mask images are different. The second decoding process is to cut out the target subject from the original image based on the mask image obtained by the first decoding process.

[0117] Specifically, the computer device can perform a second decoding process based on multiple feature maps and mask images of different scales to identify the target subject in the original image, as well as the edge transition area between the target subject and the background area, to obtain multiple candidate mask maps of different levels. The different levels of the multiple candidate mask maps indicate that the scales of the multiple candidate mask maps are different. The large-scale candidate mask maps contain more low-level information. For example, the large-scale candidate mask maps contain more texture information of the target subject. The small-scale candidate mask maps contain more high-level semantic information. For example, the small-scale candidate mask maps contain more edge contour information of the target subject, that is, the edge transition information between the target subject and the background area.

[0118] Step S208, updating the candidate mask images at different levels according to the edge transition features between the target subject and the background area included in each candidate mask image, to obtain a target mask image.

[0119] Specifically, the computer device may determine edge transition features between the target subject and the background area in each candidate mask image, and update the candidate mask images at different levels according to the edge transition features to obtain the target mask image.

[0120] In this embodiment, the computer device can determine the edge transition features between the target subject and the background area in the candidate mask map of the current level, and update the candidate mask map of the next level based on the edge transition features in the candidate mask map of the current level to obtain an updated mask map. The updated mask map is used as the candidate mask map of the current level in the next round, and the steps of determining the edge transition features and updating the candidate mask map of the next level are continued until the update of the candidate mask map of the last level is completed, and the target mask map is obtained.

[0121] In one embodiment, the scales of candidate mask images at different levels are different. A candidate mask image at a higher level has a smaller scale, and a candidate mask image at a lower level has a larger scale.

[0122] In one embodiment, a computer device may update a low-level candidate mask map through a high-level candidate mask map. The scale corresponding to the high-level candidate mask map is smaller, and the scale corresponding to the low-level candidate mask map is larger, that is, a large-scale candidate mask map is updated through a small-scale candidate mask map.

[0123] Step S210, determining a cutout result corresponding to the target subject according to the target mask image.

[0124] The cutout result refers to the image of the target subject cut out from the original image.

[0125] Specifically, the computer device can fuse the target mask image and the original image, so as to cut out the target subject from the original image.

[0126] In one embodiment, the computer device may perform a second decoding process based on a plurality of feature maps and mask images of different scales to obtain a foreground color image including the target subject. The foreground color image and the target mask image are fused to obtain a cutout result corresponding to the target subject. The foreground color image is an image of the same size as the original image, and is used to represent the color components of the pixels of the original image that belong to the foreground. If the target subject belongs to the foreground, the foreground color image is used to represent the color components of the pixels of the original image that belong to the target subject.

[0127] In this embodiment, the original image is subjected to feature encoding processing to obtain multiple feature maps of different scales, the large-scale feature map contains more low-level semantic features, and the small-scale feature map contains more high-level semantic features. Based on the feature maps of multiple different scales, the first decoding processing can identify the target subject in the original image based on the semantic features of different levels, and the position of the target subject in the original image is represented by the mask image at the semantic level to preliminarily segment the target subject. The mask image can provide stronger semantic information, and the second decoding processing based on the feature maps and mask images of multiple different scales can effectively guide the layer-by-layer interpretation of the edge transition area between the target subject and the background area. In addition, the levels of each candidate mask map are different, and the high-level candidate mask map contains more edge transition information between the target subject and the background area. According to the edge transition features in each candidate mask map, the candidate mask maps of multiple different levels are updated, which can effectively realize the strong focus regression on the low-level features, thereby obtaining a target mask map with clearer texture and more delicate edge transition details of the target subject, and then accurately obtain the cutout result corresponding to the target subject according to the target mask map, further improving the accuracy of the cutout.

[0128] In one embodiment, the method is executed by an image processing model, which includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units; wherein the coding structure is used to perform feature coding processing on the original image, the first decoding structure is used to perform a first decoding process, and the second decoding structure is used to perform a second decoding process.

[0129] Specifically, the image processing method can be executed by an image processing model, and the image processing model can be deployed and run on a computer device, such as a terminal or a server. The image processing model includes a coding structure, a first decoding structure, and a second decoding structure. The coding structure includes a plurality of coding units, the first decoding structure includes a plurality of first decoding units, and the second decoding structure includes a plurality of second decoding units.

[0130] The computer device inputs the acquired original image into the encoding structure of the image processing model, performs feature encoding processing on the original image through the encoding structure, and obtains multiple feature maps of different scales. The multiple feature maps of different scales are input into the first decoding structure, and a first decoding process is performed based on the multiple feature maps of different scales to obtain a mask image, which is used to segment the target subject from the original image at the semantic level. The mask image and the multiple feature maps of different scales are used as input data of the second decoding structure, and a second decoding process is performed based on the multiple feature maps of different scales and the mask image through the second decoding structure to obtain multiple candidate mask images of different levels.

[0131] The image processing model updates multiple candidate mask images at different levels according to the edge transition features between the target subject and the background area included in each candidate mask image to obtain a target mask image. The image processing model determines the cutout result corresponding to the target subject according to the target mask image and outputs the cutout result.

[0132] In this embodiment, the multiple encoding units in the encoding structure correspond to the multiple first decoding units in the first decoding structure, and the multiple encoding units in the encoding structure correspond to the multiple second decoding units in the second decoding structure. It can be understood that the feature map output by each encoding unit is used as input data of the corresponding first decoding unit, and the feature map output by each encoding unit is used as input data of the corresponding second decoding unit.

[0133] In one embodiment, the scales of the feature maps output by the encoding units in the encoding structure are different.

[0134] In this embodiment, the image processing method is executed by an image processing model, which includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units. By performing feature coding processing on the original image through the coding structure, semantic features of different levels can be obtained, so that the first decoding structure performs a first decoding process based on semantic features of different levels, preliminarily predicts the position of the target subject in the original image, and reflects it through a mask image. The mask image reminds the rough position of the target subject in the original image, and then the second decoding structure uses semantic features of different levels and mask images to perform a second decoding process, which can effectively guide the second decoding structure to focus on the interpretation of the edge transition area between the target subject and the background area, thereby decoding to obtain candidate mask images of different levels, and realizing strong attention regression on low-level features. The candidate mask image not only contains the position of the target subject in the original image, but also contains the feature information of the edge transition area between the target subject and the background area, so that the target mask image with finer edge transition details can be obtained through candidate mask images of different levels.

[0135] In one embodiment, feature encoding is performed on the original image to obtain multiple feature maps of different scales, including:

[0136] An original image is obtained and input into a coding structure; encoding is performed in sequence by a plurality of coding units in the coding structure based on input data corresponding to each coding unit, so as to obtain feature maps output by each coding unit; wherein the input data of a subsequent coding unit includes the feature map output by a previous coding unit, the input data of an initial coding unit in the coding structure is the original image, and the scales of feature maps output by different coding units in the coding structure are different.

[0137] Specifically, the image processing model includes a coding structure, which may include multiple coding units. Each coding unit performs coding processing on its own input data to obtain a feature map output by each coding unit. In addition, the input data of the subsequent coding unit includes the feature map output by the previous coding unit, and the input data of the initial coding unit in the coding structure is the original image. The scale of the feature map output by each coding unit is different, and multiple feature maps of different scales can be obtained through multiple coding units of the coding structure.

[0138] For example, a computer device may obtain an original image and input the original image into an initial coding unit of the coding structure, and the initial coding unit performs coding processing on the original image to obtain a feature map output by the initial coding unit. The initial coding unit refers to the first coding unit in the coding structure, and the feature map output by the initial coding unit is used as input data of the second coding unit. From the second coding unit onwards, the feature map output by the previous coding unit is used as input data of the next coding unit, so that the coding units perform coding processing on their respective input data to obtain feature maps output by each coding unit.

[0139] In one embodiment, the encoding process includes convolution processing, and may also include at least one of pooling processing and activation processing. The encoding unit performs convolution processing on the input data, and performs at least one of pooling processing and activation processing on the feature map obtained by the convolution processing to obtain the feature map output by the encoding unit. The feature map output by the encoding unit is used as input data of the next encoding unit.

[0140] In one embodiment, the encoding unit may include a convolution layer, and may also include at least one of a pooling layer and an activation layer. The convolution layer is used to perform convolution processing, and the pooling layer is used to perform pooling processing on the feature map output by the convolution layer. The activation layer can be used to perform activation processing on the feature map output by the convolution layer, or to perform activation processing on the feature map output by the pooling layer.

[0141] In this embodiment, the coding structure includes multiple coding units, the input data of the initial coding unit in the coding structure is the original image, and the feature map output by the previous coding unit is used as the input data of the subsequent coding unit, so that the semantic information in the original image can be extracted layer by layer through different coding units. The scale of the feature map output by each coding unit is different, so that the semantic information of different levels in the original image can be represented by feature maps of different scales. The large-scale feature map contains rich low-level information, and the small-scale feature map is a deep feature map that can reflect high-level semantic information, so that it can be decoded in combination with semantic information of different levels in the subsequent decoding to improve the accuracy of decoding.

[0142] In one embodiment, a first decoding process is performed based on a plurality of feature maps of different scales to obtain a mask image, including:

[0143] Based on multiple feature maps of different scales, a first decoding process is performed in sequence based on input data corresponding to each of the first decoding units through multiple first decoding units in the first decoding structure, until a mask image is output through the last first decoding unit, wherein the input data of a current first decoding unit in the first decoding structure includes a decoding result corresponding to a previous first decoding unit and a feature map output by an encoding unit corresponding to the current first decoding unit, and the decoding result corresponding to the current first decoding unit is used to constitute the input data of a subsequent first decoding unit.

[0144] Specifically, the plurality of encoding units in the encoding structure correspond to the plurality of first decoding units in the first decoding structure, and the feature graph output by each encoding unit serves as input data of the corresponding first decoding unit. The input data of the current first decoding unit includes the feature graph output by the corresponding encoding unit and also includes the decoding result output by the previous decoding unit. The decoding result corresponding to the current first decoding unit is used to constitute the input data of the subsequent first decoding unit.

[0145] Specifically, the current first decoding unit performs the first decoding process based on the current input data to obtain the decoding result output by the current first decoding unit, and uses the decoding result output by the current first decoding unit as the input data of the next first decoding unit, so that the next first decoding unit performs the first decoding process based on the previous decoding result and the corresponding feature map, until the decoding result output by the last first decoding unit is obtained. The decoding result output by the last first decoding unit is the mask image.

[0146] In this embodiment, the input data of the current first decoding unit includes the decoding result of the previous first decoding unit and the feature map output by the corresponding encoding unit, so that the current first decoding unit decodes again based on the previous decoding result and the corresponding encoding feature map, and the feature map represents the semantic information of the original image, so that after multiple decodings, the target subject in the original image can be identified, and the position of the target subject in the original image can be identified through the mask image at the semantic level to preliminarily segment the target subject.

[0147] In one embodiment, a second decoding process is performed based on a plurality of feature maps and mask images of different scales to obtain a plurality of candidate mask maps of different levels, including:

[0148] Determine the input data corresponding to each second decoding unit in the second decoding structure, wherein the input data corresponding to the second decoding unit includes the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes a mask image; perform second decoding processing in sequence based on the input data corresponding to each second decoding unit by multiple second decoding units in the second decoding structure, to obtain the candidate mask maps output by each second decoding unit, wherein the levels of the candidate mask maps output by each second decoding unit are different.

[0149] Specifically, the second decoding structure includes a plurality of second decoding units, and the input data corresponding to each second decoding unit includes the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit before the second decoding unit. In addition, the input data corresponding to at least a part of the second decoding units also includes the mask image obtained by the first decoding process. Each second decoding unit performs a second decoding process based on its own input data to obtain a candidate mask map output by each second decoding processing unit. In addition, the levels of the candidate mask maps output by each second decoding unit are different.

[0150] Further, the image processing model can determine the input data corresponding to the current second decoding unit in the second decoding structure. When the input data of the current second decoding unit includes a feature map and a candidate mask map, a second decoding process is performed based on the feature map and the candidate mask map to obtain the candidate mask map output by the current second decoding unit, and the candidate mask map is used as the input data of the next second decoding unit. When the input data of the current second decoding unit includes a feature map, a candidate mask map and a mask image, a second decoding process is performed based on the feature map, the candidate mask map and the mask image to obtain the candidate mask map output by the current second decoding unit, and the candidate mask map is used as the input data of the next second decoding unit, until the candidate mask map output by each second decoding unit is obtained.

[0151] In one embodiment, when the input data of the second decoding unit includes a feature map and a candidate mask map, the feature map and the candidate mask map are fused to obtain the candidate mask map output by the second decoding unit. When the input data of the second decoding unit includes a feature map, a candidate mask map and a mask image, the feature map and the mask image can be fused to obtain a fused feature, and then the fused feature and the candidate mask map are fused to obtain the candidate mask map output by the second decoding unit. According to the same processing method, the candidate mask map output by each second decoding unit can be obtained.

[0152] In this embodiment, the input data corresponding to some of the second decoding units in the second decoding structure include the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit before the second decoding unit, so that some of the second decoding units further decode by combining the corresponding encoding features and the candidate mask map obtained by the previous decoding to obtain a more accurate candidate mask map. In addition, the input data corresponding to at least a part of the second decoding units also includes a mask image, which can provide stronger semantic information, guide the second decoding unit to focus on edge transition area decoding, and further improve the accuracy of the interpretation of the edge transition area between the target subject and the background area. In addition, the candidate mask maps output by each second decoding unit have different levels, the low-level candidate mask map contains more texture information of the target subject, and the high-level candidate mask map contains more edge transition information between the target subject and the background area, so that a target mask map with clearer texture and clearer edge transition can be obtained based on multiple different levels of candidate mask maps.

[0153] In one embodiment, according to the edge transition features between the target subject and the background area included in each candidate mask map, multiple candidate mask maps at different levels are updated to obtain a target mask map, including:

[0154] The candidate mask map of the current level is enlarged to the same scale as the candidate mask map of the next level to obtain the enlarged candidate mask map; the edge transition image between the target subject and the background area in the enlarged candidate mask map is extracted, and the edge transition image includes edge transition features; based on the edge transition image and the candidate mask map of the next level, a fusion process is performed to obtain an updated mask map; the updated mask map is used as the candidate mask map of the current level in the next round, and the step of enlarging the candidate mask map of the current level to the same scale as the candidate mask map of the next level is returned and continued until the updated mask map corresponding to the candidate mask map of the last level is obtained; the updated mask map corresponding to the candidate mask map of the last level is used as the target mask map.

[0155] Specifically, the scales of candidate mask images at different levels are different. The scale corresponding to a candidate mask image at a higher level is smaller, and the scale corresponding to a candidate mask image at a lower level is larger.

[0156] The levels corresponding to the candidate mask images of multiple different levels are successively reduced, and the corresponding scales are successively increased. The computer device can determine the candidate mask images of the current level and the candidate mask images of the next level adjacent to the candidate mask images of the current level, and the current level is higher than the next level, that is, the scale corresponding to the current level is smaller than the scale corresponding to the next level.

[0157] The computer device enlarges the candidate mask image of the current level to the same scale as the candidate mask image of the next level, and obtains the enlarged candidate mask image. The computer device determines the edge transition features between the target subject and the background area in the enlarged candidate mask image, and extracts the edge transition image between the target subject and the background area from the enlarged candidate mask image based on the edge transition features. The computer device fuses the edge transition image with the candidate mask image of the next level to obtain an updated mask image. Then, the computer device uses the updated mask image as the candidate mask image of the current level in the next round, and returns to the step of enlarging the candidate mask image of the current level to the same scale as the candidate mask image of the next level and continues to execute until the updated mask image corresponding to the candidate mask image of the last level is obtained. The computer device uses the updated mask image corresponding to the candidate mask image of the last level as the target mask image.

[0158] In one embodiment, a computer device may select a candidate mask map at the highest level as a candidate mask map at the current level, and gradually update candidate mask maps at lower levels starting from the candidate mask map at the highest level until the candidate mask map at the lowest level is updated, and use the mask map obtained by updating the candidate mask map at the lowest level as the target mask map.

[0159] In one embodiment, the candidate mask images of multiple different levels are candidate mask images of multiple different scales. The candidate mask image of the current scale is enlarged to the same scale as the candidate mask image of the next adjacent scale to obtain an enlarged candidate mask image, and the current scale is smaller than the next adjacent scale; the edge transition image between the target subject and the background area in the enlarged candidate mask image is extracted, and the edge transition image includes edge transition features; based on the edge transition image and the candidate mask image of the next adjacent scale, a fusion process is performed to obtain an updated mask image; the updated mask image is used as the candidate mask image of the current scale in the next round, and the step of enlarging the candidate mask image of the current scale to the same scale as the candidate mask image of the next adjacent scale is returned and continued to be executed until the updated mask image corresponding to the candidate mask image of the last scale is obtained and stopped; the updated mask image corresponding to the candidate mask image of the last scale is used as the target mask image.

[0160] In this embodiment, the candidate mask map at the lower level contains more bottom-level information, that is, it contains more texture information of the target subject, and the candidate mask map at the higher level contains more deep information, that is, it contains more edge transition information between the target subject and the background area. The candidate mask map at the current level is enlarged to the same scale as the candidate mask map at the next adjacent level to extract the edge transition image between the target subject and the background area in the enlarged candidate mask map, so that the edge transition information can be extracted from the candidate mask map at the higher level. Based on the edge transition image and the candidate mask map at the next adjacent level, the edge transition information in the candidate mask map at the lower level can be updated through the edge transition information at the higher level, so that the edge transition details between the target subject and the background area in the candidate mask map at the lower level after the update are clearer, and at the same time, enough texture information of the target subject can be retained, thereby improving the accuracy of the cutout.

[0161] In one embodiment, a fusion process is performed based on the edge transition image and the candidate mask image of the next adjacent layer to obtain an updated mask image, including:

[0162] The edge transition image is morphologically processed to obtain a processed edge transition image, wherein the morphological processing includes at least one of an erosion operation and a dilation operation; the processed edge transition image is fused with a candidate mask image of an adjacent next level to obtain an updated mask image.

[0163] Among them, morphological processing refers to processing the shape features of the image, which may include at least one of an erosion operation and a dilation operation. Morphological processing can also be an open operation or a closed operation. The open operation refers to first erosion and then dilation, and the closed operation refers to first dilation and then erosion. Dilation is to expand the boundaries of the pixel connected components of each 1 in the mask image by one layer to fill the edges or the holes inside the 0 pixels. Erosion is to remove the boundary points of the 1-pixel connected components of the mask image to reduce one layer to extract the backbone information, remove the burrs, and remove the isolated 0 pixels.

[0164] Specifically, the computer device may perform at least one of an erosion operation and a dilation operation on the edge transition image to obtain a processed edge transition image. The computer device may determine an edge transition region in a candidate mask image of an adjacent next level, and fuse the processed edge transition image with the edge transition region in the candidate mask image of an adjacent next level to obtain an updated mask image.

[0165] Furthermore, the computer device may perform a filtering operation on the edge transition image after morphological processing to obtain a processed edge transition image. The filtering operation may be a median filter.

[0166] In one embodiment, the computer device may first perform an erosion operation on the edge transition image and then perform an expansion operation to remove noise. The computer device may perform a guided filtering operation on the edge transition image after morphological processing to implement an edge filtering operation and obtain a processed edge transition image.

[0167] In this embodiment, morphological processing and filtering processing can ensure that the edge transition image obtained has less or no noise and a softer edge. The processed edge transition image is fused with the candidate mask image of the next adjacent level, so that the edge transition details of the target subject in the obtained updated mask image are more integrated.

[0168] In one embodiment, the method further includes: performing a second decoding process based on a plurality of feature maps and mask images of different scales to obtain a foreground color map including the target subject;

[0169] Determining a cutout result corresponding to the target subject according to the target mask image includes: fusing the foreground color image and the target mask image to obtain a cutout result corresponding to the target subject.

[0170] Specifically, the computer device can perform a second decoding process based on a plurality of feature maps and mask images of different scales to identify the color components belonging to the foreground in the original image and obtain a foreground color map. If the target subject belongs to the foreground area in the original image, the foreground color map includes the color components corresponding to the target subject in the original image. The computer device fuses the foreground color map and the target mask map to obtain a cutout result corresponding to the target subject.

[0171] The original image can be composed of the following parts:

[0172] C=A*F+(1-A)*B

[0173] Among them, C is the original image, A is the Alpha map, F is the foreground color map, and B is the background image. Figure 3 As shown, the corresponding foreground color image is as follows Figure 4 shown.

[0174] In one embodiment, the image processing model includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units; wherein the coding structure is used to perform feature coding processing on the original image, the first decoding structure is used to perform first decoding processing, and the second decoding structure is used to perform second decoding processing.

[0175] A second decoding process is performed based on feature maps and mask images of multiple different scales to obtain a foreground color map including a target subject, including: determining input data corresponding to each second decoding unit in a second decoding structure, wherein the input data corresponding to the second decoding unit includes a feature map output by a corresponding encoding unit and a foreground color feature output by a previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes a mask image; and a second decoding process is performed sequentially by multiple second decoding units in the second decoding structure based on the input data corresponding to each second decoding unit, until a foreground color map output by a final second decoding unit is obtained.

[0176] In one embodiment, when the input data of the second decoding unit includes the feature map and foreground color feature of the corresponding encoding unit, the feature map and the foreground color feature are fused to obtain the foreground color feature output by the second decoding unit, and the foreground color feature output by the second decoding unit is used as the input of the next second decoding unit. When the input data of the second decoding unit includes the feature map, the foreground color feature and the mask image, the feature map and the mask image can be fused to obtain the fused feature, and then the fused feature and the foreground color feature are fused to obtain the foreground color feature output by the second decoding unit, and the foreground color feature output by the second decoding unit is used as the input of the next second decoding unit, until the last second decoding unit outputs the foreground color map.

[0177] In one embodiment, the second decoding structure may include processing parameters corresponding to the mask image and processing parameters corresponding to the foreground color image. When it is necessary to obtain multiple candidate mask images of different levels, the processing parameters corresponding to the mask image can be selected so that the second decoding structure performs a second decoding process on multiple feature images and mask images of different scales based on the processing parameters corresponding to the mask image to obtain multiple candidate mask images of different levels. When it is necessary to obtain the foreground color image, the processing parameters corresponding to the foreground color image can be selected so that the second decoding structure performs a second decoding process on multiple feature images and mask images of different scales based on the processing parameters corresponding to the foreground color image to obtain multiple candidate mask images of different levels.

[0178] In one embodiment, the second decoding structure may include a first branch and a second branch, and the processing parameters of the first branch and the second branch are different. The first branch is used to perform a second decoding process based on a plurality of feature maps and mask images of different scales to obtain a plurality of candidate mask images of different levels. The second branch is used to perform a second decoding process based on a plurality of feature maps and mask images of different scales to obtain a foreground color image including the target subject.

[0179] In this embodiment, the first branch and the second branch have the same structure but different processing parameters, that is, the first branch and the second branch are both composed of multiple second decoding units, but the processing parameters of the second decoding units in the first branch are different from the processing parameters of the second decoding units in the second branch.

[0180] In this embodiment, a second decoding process is performed based on multiple feature maps and mask images of different scales to accurately extract the color components corresponding to the foreground area in the original image, especially accurately obtain the color components corresponding to the edge transition area between the foreground and the background, and obtain a foreground color image including the target subject. The foreground color image and the target mask image are fused so that the edge details of the obtained cutout result are more informative and the edge color is more accurate.

[0181] like Figure 5 As shown, it is a flow chart of an image processing method in one embodiment. The image processing model includes a shared feature encoder (Encoder), a segmentation decoder (Decoder) and a cutout decoder. The shared feature encoder is an encoding structure, the segmentation decoder is a first decoding structure, and the cutout decoder is a second decoding structure. The original RGB image (i.e., the original image) is input into the shared feature encoder, and the shared feature encoder performs feature encoding processing on the original RGB image to obtain feature maps of different scales. The feature maps of different scales are used as input data for the segmentation decoder and the cutout decoder, respectively. The segmentation decoder performs segmentation decoding processing on the feature maps of different scales to obtain a rough portrait segmentation Mask map, i.e., a mask image. Next, the rough portrait segmentation Mask map is used as the input data of the cutout decoder, and the cutout decoder performs cutout decoding processing based on the feature maps of different scales and the rough portrait segmentation Mask map. Finally, the cutout decoder outputs a fine cutout Alpha map and a portrait foreground color map. The fine cutout Alpha map is the target mask map. The architecture diagram of the image processing model can be shown as follows Figure 6 shown.

[0182] like Figure 6 As shown, the shared feature encoder consists of a convolution layer, a pooling layer, and an activation layer. During the convolution process, the size of the feature map will become smaller as the network deepens, and features of different scales represent semantic information at different levels. Large feature maps contain rich low-level information, such as color, edge, geometry, and texture information. Small-scale features, that is, deep feature maps, better reflect high-level semantic features, such as the distinction between foreground and background. The shared feature encoder in this embodiment is designed to have sufficient depth to ensure the ability to mine effective features.

[0183] In one embodiment, the encoding structure is composed of multiple encoding units, that is, the shared feature encoder is composed of multiple encoding units, and the encoding unit is composed of a convolutional layer, a pooling layer and an activation layer.

[0184] Since feature extraction is somewhat universal, and the segmentation task and the cutout task have certain similarities, this embodiment only needs to design an encoder so that the two tasks share features. The shared feature encoder can also reduce the network to redundant calculations, so that the reasoning of the two tasks can still ensure real-time performance. The shared feature encoder will finally be connected to an Atrous Spatial Pyramid Pooling (ASPP) unit to help segment objects of different sizes.

[0185] The segmentation decoder is also composed of convolutional layers, pooling layers, and activation layers. During the decoding process, the feature map will be enlarged layer by layer, and the feature maps of each layer will be connected to the feature map of the shared feature encoder through skip connections. In this embodiment, the segmentation output layer is restored to 1 / 2 the scale of the original image to reduce the network inference time. The segmentation decoder finally outputs a mask image, i.e., a mask image, where the position of 1 represents the position of the target body and 0 represents the position of the background area.

[0186] In one embodiment, the first decoding structure is composed of multiple first decoding units, that is, the segmentation decoder is composed of multiple first decoding units, and the first decoding unit is composed of a convolutional layer, a pooling layer and an activation layer.

[0187] The cutout decoder needs to take the mask map output by the segmentation decoder and the feature maps of different scales as input. The second decoding structure is composed of multiple second decoding units, that is, the cutout decoder is composed of multiple second decoding units, and the multiple second decoding units are connected to the multiple encoding units through the skip connection structure. The input data corresponding to the second decoding unit includes the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit of the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes the mask map output by the segmentation decoder. When the input data of the second decoding unit includes the feature map, the candidate mask map and the mask map, the feature map and the mask image can be fused to obtain the fused feature, and then the fused feature and the candidate mask map are fused to obtain the candidate mask map output by the second decoding unit. During the fusion process, the segmentation mask can provide strong semantic information to guide the cutout decoder to focus on the interpretation of edge position information. According to the same processing method, the candidate mask map output by each second decoding unit can be obtained. Based on each candidate mask map, a refined Alpha map, that is, a target mask map, can be obtained.

[0188] In addition, the cutout decoder takes the mask map output by the segmentation decoder and feature maps of different scales as input for decoding, and can also obtain the foreground color map of the target subject.

[0189] Deep feature maps provide deep semantic information, and shallow feature maps provide low-level semantic information. That is, the larger the scale, the clearer the texture, such as hair, and the smaller the scale, the more complete the outline of the subject, such as the portrait. In order to retain the outline and increase the fine texture of the edge when enlarging the size of the feature map, the following refined scale enlargement strategy can be used to enlarge the candidate alpha maps of different scales from small to large:

[0190] 1) The small-scale alpha image is enlarged to the same size as the large-scale alpha image to obtain the alpha s Figure, alpha s Figure Figure 7 As shown in (a);

[0191] 2) Extract small-scale alpha s The edge position alpha of the graph small , that is, 0<alpha small <1;

[0192] 3) Extract small-scale alpha s The edge position image of the image is morphologically processed and median filtered to obtain a complete, smooth, and expanded edge area mask. edge , that is, the edge transition image, such as Figure 7 As shown in (b);

[0193] 4) Update the large-scale alpha map using the following formula to obtain the updated mask map alpha big ;

[0194] alpha big =alpha big *mask edge +alpha small *(1-mask edge )

[0195] 5) The mask alpha will be updated big As the small-scale alpha map of the next round, continue to execute the above steps 1)-4) until the last large-scale alpha map is updated to obtain the target mask map.

[0196] In one embodiment, the method is performed by an image processing model obtained by a training step, the training step comprising:

[0197] A first sample image set and a second sample image set are obtained, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image; an image processing model to be trained is determined, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure; a first training is performed on the initial encoding structure and the initial first decoding structure through the first sample image set until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure; a second training is performed on the initial second decoding structure based on the trained encoding structure and the first decoding structure through the second sample image set, thereby obtaining a trained second decoding structure, thereby obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

[0198] The first sample image and the second sample image can be any one of RGB (Red, Green, Blue) images, grayscale images, depth images, images corresponding to the Y component in YUV images, etc., but are not limited thereto. The cutout label is the real mask image corresponding to the second sample image, or the real value representing the mask image corresponding to the second sample image. Meeting the first stop condition can be reaching a preset number of training times, a preset number of iterations, a loss value less than or equal to a loss threshold, etc.

[0199] Specifically, the computer device may obtain a first sample image set and a second sample image set. The first sample image set is a set including each first sample image and a segmentation label corresponding to each first sample image. The second sample image set is a set including each second sample image and a cutout label corresponding to each second sample image. The segmentation label is a real mask image corresponding to the first sample image, or a real value representing the mask image corresponding to the first sample image.

[0200] In one embodiment, the first sample image set and the second sample image set may contain the same image, and the second sample image set may include a composite image and a non-composite image. The composite image may be obtained by fusing a subject area in one image with a background area in another image.

[0201] The computer device may determine an image processing model to be trained, the image processing model to be trained comprising an initial coding structure, an initial first decoding structure and an initial second decoding structure. The initial coding structure is used to perform feature coding processing on the sample image, the initial first decoding structure is used to perform a first decoding processing, and the initial second decoding structure is used to perform a second decoding processing.

[0202] In one embodiment, the initial encoding structure is composed of a plurality of encoding units, the initial first decoding structure is composed of a plurality of first decoding units, and the initial second decoding structure is composed of a plurality of second decoding units.

[0203] The computer device performs a first training on the initial encoding structure and the initial first decoding structure through each first sample image in the first sample image set and the corresponding segmentation label, so as to adjust the parameters of the initial encoding structure and the initial first decoding structure in the first training process, and continue the training after adjusting the parameters until the first stopping condition is met, so as to obtain the trained encoding structure and the first decoding structure.

[0204] Specifically, the computer device performs a second training on the initial second decoding structure through each second sample image in the second sample image set and the corresponding cutout label, so as to adjust the parameters of the initial second decoding structure in the second training process, and continue the training after adjusting the parameters until the second stop condition is met, thereby obtaining a trained second decoding structure. The trained encoding structure, the first decoding structure and the second decoding structure constitute a trained image processing model. The trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

[0205] In this embodiment, a first sample image set and a second sample image set are obtained, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image. An image processing model to be trained is determined, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure. The initial encoding structure and the initial first decoding structure are first trained through the first sample image set until a first stop condition is satisfied, and a trained encoding structure and a first decoding structure are obtained. On the basis of ensuring the processing accuracy of the encoding structure and the first decoding structure, the second sample image set is used and the initial second decoding structure is secondly trained based on the trained encoding structure and the first decoding structure, and the second stop condition is satisfied, and a trained second decoding structure is obtained. The trained encoding structure, the first decoding structure and the second decoding structure constitute a trained image processing model, which can effectively provide the segmentation accuracy and the cutout accuracy of the image processing model, so that the cutout result corresponding to the target subject in the original image can be accurately determined through the image processing model. Furthermore, by determining the cutout result corresponding to the target subject in the original image through the image processing model, the cutout efficiency can be effectively improved.

[0206] In one embodiment, the image processing method can be applied to video conferencing or live broadcasting scenarios. By acquiring image frames in real time during a video conference, and cutting out the portraits in the image frames through the above-mentioned processing method, the cut-out portraits and the virtual background are synthesized in real time, so that the virtual background can be used to replace the real background of the characters during video conferencing or live broadcasting, which can effectively protect user privacy. The synthesized image frames obtained by the above-mentioned image processing method have more natural processing of the edge details of the portraits, especially the transition of the hair is more natural and delicate. Combined with the foreground color image, it is possible to effectively avoid the problem of a circle of artifacts appearing around the portrait when using a dark background image, which greatly improves the virtual background effect and enhances the user's visual experience. In addition, the use of virtual backgrounds in live broadcasting scenarios can also reduce costs and is not limited to venues.

[0207] In one embodiment, the image processing method can be applied to a cloud rendering platform, mainly to a studio and a host's home environment. The host's portrait can be cut out in real time through the image processing method, and the host's portrait and the background image to be used can be synthesized in real time, so that it can be used in any place and can effectively reduce costs.

[0208] In one embodiment, the image processing method can be applied to a video explanation scene, with the purpose of removing the background in the host video stream when mixing the host stream with the film and television stream, so that the host portrait can be perfectly integrated into the scene in the film and television stream.

[0209] In one embodiment, the image processing method can be applied to any scenario that requires cutout, such as poster design activities. Automatic cutout can be achieved through the above image processing method, which can help reduce manual processing time and improve work efficiency.

[0210] It should be noted that the application scenarios mentioned in the above embodiments are not used to limit the image processing method of the embodiments of the present application, but are only used to illustratively illustrate the application scenarios of the image processing method provided by the embodiments of the present application.

[0211] In one embodiment, Figure 8 As shown, a training method for an image processing model is provided, and the method is applied to a computer device (the computer device may be Figure 1 The terminal or server in the example is used to illustrate, including the following steps:

[0212] Step S802, obtaining a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image.

[0213] Specifically, the computer device may obtain a first sample image set and a second sample image set. The first sample image set is a set including each first sample image and a segmentation label corresponding to each first sample image. The second sample image set is a set including each second sample image and a cutout label corresponding to each second sample image. The segmentation label is a real mask image corresponding to the first sample image, or a real value representing the mask image corresponding to the first sample image.

[0214] In one embodiment, the first sample image set and the second sample image set may contain the same image, and the second sample image set may include a composite image and a non-composite image. The composite image may be obtained by fusing a subject area in one image with a background area in another image.

[0215] Step S804, determining an image processing model to be trained, where the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure.

[0216] Specifically, the computer device may determine an image processing model to be trained, the image processing model to be trained comprising an initial encoding structure, an initial first decoding structure and an initial second decoding structure. The initial encoding structure is used to perform feature encoding processing on the sample image, the initial first decoding structure is used to perform a first decoding processing, and the initial second decoding structure is used to perform a second decoding processing.

[0217] In one embodiment, the initial encoding structure is composed of a plurality of encoding units, the initial first decoding structure is composed of a plurality of first decoding units, and the initial second decoding structure is composed of a plurality of second decoding units.

[0218] Step S806: Perform first training on the initial coding structure and the initial first decoding structure using the first sample image set until a first stopping condition is met, thereby obtaining a trained coding structure and a first decoding structure.

[0219] Among them, satisfying the first stopping condition may be reaching a preset number of training times, a preset number of iterations, a loss value being less than or equal to a loss threshold, etc.

[0220] Specifically, the computer device performs a first training on the initial encoding structure and the initial first decoding structure through each first sample image in the first sample image set and the corresponding segmentation label, so as to adjust the parameters of the initial encoding structure and the initial first decoding structure during the first training process, and continue the training after adjusting the parameters until the first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure.

[0221] For example, when the training of the initial coding structure and the initial first decoding structure does not reach the preset training times, the parameters of the initial coding structure and the initial first decoding structure are adjusted and the training continues until the preset training times are reached, thereby obtaining the trained coding structure and the first decoding structure.

[0222] Step S808, using a second sample image set, and based on the trained encoding structure and the first decoding structure, the initial second decoding structure is subjected to a second training, and the second training is stopped when a second stopping condition is met, thereby obtaining a trained second decoding structure and obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

[0223] Among them, satisfying the second stop condition may be reaching a preset number of training times, a preset number of iterations, a loss value being less than or equal to a loss threshold, etc.

[0224] Specifically, the computer device performs a second training on the initial second decoding structure through each second sample image in the second sample image set and the corresponding cutout label, so as to adjust the parameters of the initial second decoding structure in the second training process, and continue the training after adjusting the parameters until the second stop condition is met, thereby obtaining a trained second decoding structure. The trained encoding structure, the first decoding structure and the second decoding structure constitute a trained image processing model. The trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

[0225] In this embodiment, a first sample image set and a second sample image set are obtained, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image. An image processing model to be trained is determined, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure. The initial encoding structure and the initial first decoding structure are first trained through the first sample image set until a first stop condition is satisfied, and a trained encoding structure and a first decoding structure are obtained. On the basis of ensuring the processing accuracy of the encoding structure and the first decoding structure, the second sample image set is used and the initial second decoding structure is secondly trained based on the trained encoding structure and the first decoding structure, and the second stop condition is satisfied, and a trained second decoding structure is obtained. The trained encoding structure, the first decoding structure and the second decoding structure constitute a trained image processing model, which can effectively provide the segmentation accuracy and the cutout accuracy of the image processing model, so that the cutout result corresponding to the target subject in the original image can be accurately determined through the image processing model. Furthermore, by determining the cutout result corresponding to the target subject in the original image through the image processing model, the cutout efficiency can be effectively improved.

[0226] In one embodiment, the initial encoding structure and the initial first decoding structure are first trained by using a first sample image set until a first stopping condition is satisfied, thereby obtaining a trained encoding structure and a first decoding structure, including:

[0227] The first sample image is subjected to feature encoding processing through the initial encoding structure to obtain a plurality of first feature maps of different scales; the first mask image corresponding to the first sample image is subjected to first decoding processing based on the plurality of first feature maps of different scales through the initial first decoding structure; the initial encoding structure and the initial first decoding structure are trained according to the difference between the segmentation labels corresponding to the first mask image and the first sample image until the first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure.

[0228] Specifically, the computer device may perform feature encoding processing on the first sample image through the initial encoding structure to obtain a first feature map, and perform the next feature encoding processing on the obtained first feature map to obtain a corresponding first feature map. After multiple feature encoding processes, first feature maps of different scales may be obtained. The initial first decoding structure performs a first decoding process according to the first feature maps of multiple different scales to identify the first subject in the first sample image and the position of the first subject in the first sample image. The first feature maps of different scales represent semantic features at different levels. After the first decoding process of the semantic features at different levels, a first mask image may be obtained, and the first mask image may be used to determine the position of the first subject in the first sample image. The computer device may calculate the difference between the segmentation labels corresponding to the first mask image and the first sample image, adjust the parameters of the initial encoding structure and the initial first decoding structure based on the difference, and continue training until the trained initial encoding structure and the initial first decoding structure meet the first stop condition, and obtain the trained encoding structure and the first decoding structure.

[0229] In one embodiment, the initial coding structure is composed of a plurality of coding units, and the initial first decoding structure is composed of a plurality of first decoding units. The plurality of coding units in the initial coding structure sequentially perform coding processing based on the input data corresponding to each coding unit, so as to obtain the first feature maps outputted by each coding unit; wherein the input data of the latter coding unit in the initial coding structure includes the first feature map outputted by the former coding unit, the input data of the initial coding unit in the initial coding structure is the first sample image, and the scales of the first feature maps outputted by different coding units in the initial coding structure are different.

[0230] Based on multiple first feature maps of different scales, first decoding processing is performed in sequence based on input data corresponding to each of the first decoding units through multiple first decoding units in the initial first decoding structure, until a mask image is output through the last first decoding unit, wherein the input data of the current first decoding unit in the initial first decoding structure includes a decoding result corresponding to the previous first decoding unit and a first feature map output by the encoding unit corresponding to the current first decoding unit, and the decoding result corresponding to the current first decoding unit is used to constitute the input data of the subsequent first decoding unit.

[0231] For example, the difference between the segmentation labels corresponding to the first mask image and the first sample image can be calculated by the cross entropy loss function:

[0232]

[0233] Among them, y iRepresents the true value of the first sample image i, that is, the segmentation label. The positive class is 1, and the negative class is 0. The positive class refers to the sample subject, such as a portrait, and the negative class refers to the background. i Represents the probability that the first sample image i is predicted to be a positive class.

[0234] In this embodiment, the first sample image is feature encoded by the initial encoding structure to obtain a plurality of first feature maps of different scales, and the first mask image corresponding to the first sample image is obtained by the initial first decoding structure based on the first feature maps of different scales. The initial encoding structure and the initial first decoding structure are trained according to the difference between the segmentation labels corresponding to the first mask image and the first sample image until the first stop condition is met, so that the trained encoding structure can accurately extract semantic information at different levels. The large-scale feature map contains rich low-level information, and the small-scale feature map can reflect high-level semantic information, so that the initial first decoding structure can combine semantic information at different levels for decoding, thereby improving the accuracy of decoding.

[0235] In one embodiment, the second sample image set is used to perform a second training on the initial second decoding structure based on the trained encoding structure and the first decoding structure until a second stopping condition is met, thereby obtaining a trained second decoding structure, including:

[0236] The second sample image is subjected to feature encoding processing by means of the encoding structure to obtain a plurality of second feature maps of different scales; the first decoding structure is subjected to a first decoding processing based on the second feature maps of different scales to obtain a second mask image corresponding to the second sample image; the initial second decoding structure is subjected to a second decoding processing based on the second feature maps of different scales and the second mask image to obtain a plurality of sample mask maps of different levels; the initial second decoding structure is subjected to an update processing on the sample mask maps of different levels based on the sample edge transition features between the sample body and the background area included in each sample mask map to obtain a predicted mask map; the initial second decoding structure is trained based on the mask loss between the cutout labels corresponding to the predicted mask map and the second sample image until the second stop condition is met to obtain a trained second decoding structure.

[0237] Specifically, the computer device can perform feature encoding processing on the second sample image through the encoding structure to obtain a second feature map, and perform the next feature encoding processing on the obtained second feature map to obtain a corresponding second feature map. After multiple feature encoding processes, second feature maps of different scales can be obtained. The second decoding structure performs a second decoding process based on multiple second feature maps of different scales to identify the sample body in the second sample image and the position of the sample body in the second sample image, and generates a second mask image for indicating the position of the sample body in the second sample image.

[0238] The initial second decoding structure can perform a second decoding process based on multiple feature maps and mask images of different scales to identify the sample body in the second sample image and the sample edge transition area between the sample body and the background area, and obtain multiple sample mask images of different levels. The initial second decoding structure can determine the sample edge transition features between the sample body and the background area in each sample mask image, and update the sample mask images of different levels according to each sample edge transition feature to obtain a predicted mask image.

[0239] Furthermore, the initial second decoding structure can determine the sample edge transition features between the sample body and the background area in the sample mask map of the current level, and update the sample mask map of the next adjacent level based on the sample edge transition features in the sample mask map of the current level to obtain an updated mask map. The updated mask map is used as the sample mask map of the current level in the next round, and the steps of determining the sample edge transition features and updating the sample mask map of the next adjacent level are continued until the update of the sample mask map of the last level is completed, and a predicted mask map is obtained.

[0240] The computer device can calculate the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, adjust the parameters of the initial second decoding structure based on the mask loss and continue training until the trained initial second decoding structure meets the second stopping condition, thereby obtaining a trained second decoding structure.

[0241] In one embodiment, the encoding structure is composed of a plurality of encoding units, the first decoding structure is composed of a plurality of first decoding units, and the initial second decoding structure is composed of a plurality of second decoding units. Determine the input data corresponding to each second decoding unit in the initial second decoding structure, wherein the input data corresponding to the second decoding unit includes the second feature map output by the corresponding encoding unit and the sample mask map output by the previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes a second mask image; the plurality of second decoding units in the initial second decoding structure sequentially perform second decoding processing based on the input data corresponding to each second decoding unit, and obtain the sample mask maps output by each second decoding unit, wherein the sample mask maps output by each second decoding unit have different levels.

[0242] In one embodiment, the initial second decoding structure includes initial parameters corresponding to the mask image and initial parameters corresponding to the foreground color image. The computer device performs a second training on the initial second decoding structure through a second sample image set and based on the trained encoding structure and the first decoding structure to adjust the initial parameters corresponding to the mask image in the second decoding structure until the second stop condition is met, thereby obtaining processing parameters corresponding to the mask image in the trained second decoding structure.

[0243] In one embodiment, the initial second decoding structure may include an initial first branch and an initial second branch. The computer device performs second training on the initial first branch in the initial second decoding structure based on the trained encoding structure and the first decoding structure through the second sample image set to adjust the initial parameters of the initial first branch until the second stop condition is met, and obtains the trained first branch and the processing parameters corresponding to the first branch. The processing parameters corresponding to the first branch refer to the processing parameters corresponding to the mask image.

[0244] In this embodiment, the initial second decoding structure performs a second decoding process based on the second feature maps and the second mask images of multiple different scales to obtain a plurality of sample mask maps of different levels, including: performing a second decoding process based on the second feature maps and the second mask images of multiple different scales through the initial first branch in the initial second decoding structure to obtain a plurality of sample mask maps of different levels;

[0245] Based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, the initial second decoding structure is trained until the second stopping condition is met, and the trained second decoding structure is obtained, including: based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, the initial first branch is trained until the second stopping condition is met, and the trained first branch and the processing parameters corresponding to the first branch are obtained.

[0246] In this embodiment, the initial second decoding structure uses different levels of semantic features and mask images to perform the second decoding process, which can effectively guide the second decoding structure to focus on the interpretation of the edge transition area between the target subject and the background area, thereby decoding to obtain sample mask maps of different levels, so as to achieve strong focus regression on low-level features during the training process. The sample mask map not only contains the position of the sample subject in the original image, but also contains the feature information of the edge transition area between the sample subject and the background area, so that a predicted mask map with finer edge transition details can be obtained through sample mask maps of different levels, and based on the mask loss between the predicted mask map and the cutout label corresponding to the second sample image, the initial second decoding structure is trained, and the parameters of the second decoding structure can be continuously adjusted, so that the trained second decoding structure can predict the edge transition details in the mask map more finely and delicately, making the cutout more accurate.

[0247] In one embodiment, the mask loss includes a global loss and a scale loss; based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, the initial second decoding structure is trained until a second stop condition is met, and the trained second decoding structure is obtained, including:

[0248] Determine the global loss between the predicted mask image and the cutout label corresponding to the second sample image; adjust the predicted mask image to different scales, and determine the scale loss of the predicted mask image at different scales; construct a target loss function based on the global loss and the scale loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

[0249] Specifically, the mask loss includes a global loss and a scale loss. The global loss refers to the overall difference between the predicted mask image and the corresponding cutout label, while the scale loss refers to the loss caused by the predicted mask image at different scales. The computer device can calculate the difference between the predicted mask image and the cutout label corresponding to the second sample image, and determine the global loss between the predicted mask image and the cutout label based on the difference. For example, the difference between the predicted mask image and the cutout label, or the absolute value of the difference, is used as the global loss corresponding to the second sample image.

[0250] It can be understood that the second sample image set includes multiple second sample images. In this way, the global loss of each second sample image can be calculated, summed and averaged, and the average is used as the global loss in the current training.

[0251] For example, the computing device may calculate the global loss using the following formula:

[0252]

[0253] Among them, y i represents the true alpha value of sample image i, which is between 0 and 1, p i Represents the predicted foreground opacity of sample image i, that is, the predicted alpha value.

[0254] The computer device may adjust the predicted mask image corresponding to the second sample image to different scales to obtain predicted mask images of different scales, and the computer device calculates the global loss between the predicted mask image at each scale and the cutout label corresponding to the second sample image to obtain the global loss at different scales corresponding to the second sample image. The computer device may calculate the scale loss corresponding to the second sample image according to the global loss at different scales corresponding to the second sample image and the number of predicted mask images at different scales.

[0255] For example, the computing device may calculate the scale loss by the following formula:

[0256]

[0257] Among them, j is the scale of the predicted mask image.

[0258] The computer device can use the summation formula of the global loss and the scale loss as the target loss function, or multiply the global loss and the scale loss by the corresponding weights and then sum them to obtain the target loss function. The initial second decoding structure is trained by the target loss function, and the parameters of the initial second decoding structure are adjusted based on the generated target loss during training and the training is continued until the trained initial second decoding structure meets the second stop condition, thereby obtaining a trained second decoding structure.

[0259] In this embodiment, the computer device can calculate the target loss value generated by the training through the target loss function in each training, and when the target loss value is greater than the loss threshold, adjust the parameters of the initial second decoding structure and continue training. When the target loss value is less than or equal to the loss threshold, stop and obtain the trained second decoding structure.

[0260] In this embodiment, the target loss function is constructed by combining the difference between the predicted mask image and the actual cutout label, as well as the difference between the predicted mask images of different scales and the actual cutout labels. This can take into account the impact of prediction results of different scales on the image processing model, so that the trained second decoding structure can be compatible with the processing of images of different scales, so that the generated predicted mask image is more accurate.

[0261] In one embodiment, the method further comprises: determining a mask loss between the second mask image and the segmentation labels corresponding to the second sample image;

[0262] Based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, the initial second decoding structure is trained until the second stop condition is met, and the trained second decoding structure is obtained, including:

[0263] Determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the mask loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

[0264] Specifically, the computer device can determine the difference between the segmentation labels corresponding to the second mask image and the second sample image, that is, the mask loss, through a cross entropy loss function. The computer device can calculate the difference between the predicted mask image and the cutout label corresponding to the second sample image, and determine the mask loss between the predicted mask image and the cutout label based on the difference. The computer device can use the summation formula of the mask loss and the mask loss as the target loss function, or multiply the mask loss and the mask loss by the corresponding weights respectively and then sum them to obtain the target loss function. The initial second decoding structure is trained through the target loss function, and the parameters of the initial second decoding structure are adjusted based on the generated target loss during training and the training is continued until the trained initial second decoding structure meets the second stop condition, thereby obtaining a trained second decoding structure.

[0265] In this embodiment, the mask loss can represent the loss generated by the trained encoding structure and the first decoding structure, and the mask loss can represent the loss generated by the initial second decoding structure. Combining the mask loss and the mask loss to train the initial second decoding structure can further improve the accuracy of the second decoding structure, so that the mask image predicted by the trained second decoding structure is more accurate.

[0266] In one embodiment, the second sample image set also includes a foreground color label corresponding to the second sample image, and the method further includes:

[0267] Performing a second decoding process based on a plurality of second feature maps of different scales and a second mask image through an initial second decoding structure to obtain a predicted foreground color image; determining a foreground color loss between the predicted foreground color image and a foreground color label corresponding to the second sample image;

[0268] Based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, the initial second decoding structure is trained until the second stop condition is met, and the trained second decoding structure is obtained, including:

[0269] Determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image; construct a target loss function based on the mask loss and the foreground color loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

[0270] Specifically, the second sample image set also includes a foreground color label corresponding to each second sample image. The computer device performs a second decoding process based on a plurality of second feature maps of different scales and a second mask image through an initial second decoding structure to identify the color components belonging to the foreground in the second sample image and obtain a predicted foreground color map. If the main body of the sample belongs to the foreground area in the second sample image, the foreground color map includes the color components corresponding to the main body of the sample in the second sample image. The computer device can calculate the difference between the predicted foreground color map corresponding to the second sample image and the corresponding foreground color label, and determine the foreground color loss between the predicted foreground color map and the corresponding foreground color label based on the difference.

[0271] For example, the foreground color loss can be calculated by the following formula:

[0272] Foreground color loss:

[0273]

[0274] Among them, y i represents the true value of the foreground color of sample image i, p i represents the foreground color prediction value of sample image i, and c represents the RGB channel index.

[0275] The computer device can calculate the difference between the predicted mask image and the cutout label corresponding to the second sample image, and determine the mask loss between the predicted mask image and the cutout label based on the difference. The computer device can use the summation formula of the mask loss and the foreground color loss as the target loss function, or multiply the mask loss and the foreground color loss by the corresponding weights and then sum them to obtain the target loss function. The initial second decoding structure is trained by the target loss function, and the parameters of the initial second decoding structure are adjusted based on the generated target loss during training and the training is continued until the trained initial second decoding structure meets the second stop condition, thereby obtaining a trained second decoding structure.

[0276] In one embodiment, the encoding structure is composed of a plurality of encoding units, the first decoding structure is composed of a plurality of first decoding units, and the initial second decoding structure is composed of a plurality of second decoding units. Determine the input data corresponding to each second decoding unit in the initial second decoding structure, wherein the input data corresponding to the second decoding unit includes the second feature map output by the corresponding encoding unit and the foreground color feature output by the previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes a second mask image; perform second decoding processing based on the input data corresponding to each second decoding unit in turn through the plurality of second decoding units in the initial second decoding structure until the predicted foreground color map output by the last second decoding unit is obtained.

[0277] In one embodiment, the initial second decoding structure includes initial parameters corresponding to the mask image and initial parameters corresponding to the foreground color image. The computer device performs a second training on the initial second decoding structure based on the trained encoding structure and the first decoding structure through the second sample image set to adjust the initial parameters corresponding to the foreground color image in the second decoding structure until the second stop condition is met, and obtains the processing parameters corresponding to the foreground color image in the trained second decoding structure.

[0278] In one embodiment, the initial second decoding structure may include an initial first branch and an initial second branch. The computer device performs second training on the initial second branch in the initial second decoding structure based on the trained encoding structure and the first decoding structure through the second sample image set to adjust the initial parameters of the initial second branch until the second stop condition is met, and obtains the trained second branch and the processing parameters corresponding to the second branch. The processing parameters corresponding to the second branch refer to the processing parameters corresponding to the foreground color image.

[0279] In this embodiment, performing a second decoding process based on a plurality of second feature maps of different scales and a second mask image through an initial second decoding structure to obtain a predicted foreground color image includes: performing a second decoding process based on a plurality of second feature maps of different scales and a second mask image through an initial second branch in the initial second decoding structure to obtain a predicted foreground color image;

[0280] The initial second decoding structure is trained by the target loss function until the second stop condition is met, and the trained second decoding structure is obtained, including: the initial second branch is trained by the target loss function until the second stop condition is met, and the trained second branch and the processing parameters corresponding to the second branch are obtained. The trained first branch and the second branch constitute the trained second decoding structure.

[0281] In this embodiment, a second decoding process is performed based on feature maps and mask images of multiple different scales to accurately extract the color components corresponding to the foreground area in the original image, especially accurately obtain the color components corresponding to the edge transition area between the foreground and the background, and obtain a predicted foreground color map including the main body of the sample. The initial second decoding structure is trained in combination with the foreground color loss and the mask loss, and the parameters of the foreground color map and the mask map corresponding to the predicted image of the second decoding structure can be trained simultaneously, so that the trained second decoding structure can share the output data of the encoding structure and the first decoding structure, and independently predict the foreground color map and the mask map corresponding to the image based on the shared output data, and make the edge details of the obtained cutout result more informative and the edge color more accurate.

[0282] In one embodiment, the second sample image includes a composite image; the method further includes: determining a channel loss of a predicted mask image and a corresponding cutout label corresponding to the composite image on a color channel;

[0283] Based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, the initial second decoding structure is trained until the second stop condition is met, and the trained second decoding structure is obtained, including:

[0284] Determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the channel loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

[0285] Specifically, the second sample image in the second sample image set may be a synthetic image or a non-synthetic image. When there is a synthetic image in the training, after obtaining the predicted mask image corresponding to the synthetic image through the initial second decoding structure, the channel loss of the predicted mask image and the corresponding cutout label on the color channel is determined. Channel loss refers to the sum of the losses on each color channel, for example, the sum of the losses on the red channel, the green channel, and the blue channel.

[0286] For example, the channel loss can be calculated by the following formula:

[0287]

[0288] Among them, L comp is the channel loss, y i represents the channel true value of sample image i, p i represents the channel prediction value of sample image i, and c represents the RGB channel index.

[0289] The computer device can calculate the difference between the predicted mask image and the cutout label corresponding to the second sample image, and determine the mask loss between the predicted mask image and the cutout label based on the difference. The computer device can use the summation formula of the mask loss and the channel loss as the target loss function, or multiply the mask loss and the channel loss by the corresponding weights respectively and then sum them to obtain the target loss function. The initial second decoding structure is trained by the target loss function, and the parameters of the initial second decoding structure are adjusted based on the generated target loss during training and the training is continued until the trained initial second decoding structure meets the second stop condition, thereby obtaining a trained second decoding structure.

[0290] In this embodiment, the use of synthetic images in training can improve the richness and complexity of training data, and can effectively improve the recognition ability of the image processing model. In addition, the combination of the loss on different color channels and the mask loss on the second decoding structure can further improve the accuracy of the second decoding structure decoding.

[0291] In one embodiment, the computer device may be based on the foreground color loss L fore , global loss L l1 and scale loss L lap Construct the target loss function L total , or based on the foreground color loss L fore , global loss L l1 , scale loss L lap and mask loss L seg Construct the target loss function L total , and can also be based on the global loss L l1 , scale loss L lap , mask loss L seg and channel loss L comp Construct the target loss function L total .

[0292] In other embodiments, the computer device may further calculate the foreground color loss L based on the foreground color loss L. fore , global loss L l1 , scale loss L lap , mask loss L seg and channel loss L comp , construct the target loss function L total , the constructed objective loss function L total As shown below:

[0293] L total =L seg +L l1 +L comp +L tap +L fore

[0294] In one embodiment, the training of the image processing model includes two stages: segmentation task training and cutout task training. The segmentation task training mainly trains the shared feature encoder and the segmentation decoder, and the cutout task training mainly trains the cutout decoder.

[0295] like Fig. 9 It can be seen from the segmentation result and the cutout result that the edge of the segmentation result is rough, the synthetic background has a sense of separation, and the cutout result is very natural and realistic. However, due to the difficulty in labeling the cutout data, the data set is very small, while the segmentation data is relatively large. The size of the training data set directly affects the generalization of the model, so the segmentation data is used to train in this embodiment to improve the generalization of the image processing model.

[0296] 1. Segmentation task training: Using the segmentation dataset, the gradient descent method is used to train the shared feature encoder and segmentation decoder. The specific process is as follows Fig.10 As shown:

[0297] 1) Build a segmentation dataset

[0298] There are two sources for training data: open source data download and data annotation. There are many open source portrait segmentation labels, and special portrait label data include Supervisely, Ai Fenjie, etc. Semantic segmentation datasets can also be used to extract portrait labels. The method is to set the portrait label to 1 and reset all other label categories to background 0. Available semantic segmentation labels include COCO, PascalVOC, etc. This method is also suitable for the Human Parsing dataset, which can reset all human body parts to 1. The number of data images collected by the above method can reach 100k, which far exceeds the order of magnitude of traditional cutout data and can greatly improve the robustness of the model.

[0299] 2) Random initialization of model parameters

[0300] Since we are training from scratch, we can initialize the model randomly.

[0301] 3) Batch data input

[0302] The computer cannot calculate all the data at once, so it is necessary to import the data into the memory in batches. Each batch of data will run the following 4) and 5) steps completely.

[0303] 4) Calculate segmentation loss

[0304] The data is input into the network, and after one forward propagation, the predicted mask image is output. In order to calculate the correctness of the predicted image, it needs to be compared with the true value mask image. The function used here to calculate the difference between the predicted image and the true value image is the loss function. The segmentation loss function to be used in this article is the binary cross entropy loss function, and the formula is as follows:

[0305]

[0306] 5) Update model parameters

[0307] After obtaining the loss based on the above steps, run a back propagation calculation to obtain the network parameter gradient, and update the model parameters according to the gradient descent method, that is, update the parameters of the shared feature encoder and the segmentation decoder, so that a complete training is completed. If the preset number of training rounds is not reached here, the process will continue to return to step 3) and start a new round of training. After each round of 3), 4), and 5), the model parameters will be updated once, so after multiple rounds of cyclic updates, the model gradually converges. When the number of training rounds reaches the preset number, the training ends and the process goes to step 6).

[0308] 6) Save model parameters

[0309] After the training is completed, the quality of the model is evaluated and the best model parameters are selected for saving. Since the cutout decoder is not involved in the training at this stage, the model parameters should be updated and saved as the parameters of the shared feature encoder and the segmentation decoder.

[0310] 2. Cutout Task Training

[0311] Using the cutout dataset, we use the gradient descent method to train the shared feature encoder and cutout decoder. The specific process is as follows: Fig.11 As shown:

[0312] 1) Construct a cutout training dataset

[0313] Open-source portrait cutout data is relatively scarce, and only a few hundred images are available. Since the purpose of the cutout decoder training is to force the network to learn low-level detailed texture features, the dataset should contain as much texture-rich data as possible, such as hair. In addition to foreground label data, we also need to collect more background data to increase the richness of the dataset through background synthesis during training to avoid overfitting of the model.

[0314] 2) Read the shared feature encoder and segmentation decoder parameters

[0315] After the segmentation training, the shared feature encoder has extracted some robust features, and the cutout training here will reuse these shared features. Therefore, the parameters of the shared feature encoder and the segmentation decoder will be read from the parameters saved after the segmentation task training is completed, while the cutout decoder parameters are still generated randomly.

[0316] 3) Set the learning rate

[0317] After training on the segmentation task, the shared feature encoder and the segmentation encoder already have relatively synthetic parameters. Therefore, a lower learning rate should be set for training in this stage, while a larger learning rate should be set for the cutout decoder to help it converge quickly.

[0318] 4) Batch data input

[0319] The computer cannot calculate all the data at once, so it is necessary to import the data into the memory in batches. Each batch of data will run the following 4) and 5) steps completely.

[0320] 5) Random background synthesis

[0321] Since the number of foreground data is not large enough, overfitting is very likely to occur during the training process. In this embodiment, data enhancement methods are increased as much as possible during the training process, and background synthesis is one of the enhancement methods. That is, after each batch of data is loaded into the memory, some sample images are randomly selected with a certain probability, and a background is randomly selected from the background image set to replace the background in the original image. In this way, the richness of the data can be effectively increased.

[0322] Image synthesis can be achieved through the following formula:

[0323] C=A*F+(1-A)*B

[0324] Among them, C is the composite image, A is the Alpha image, F is the foreground color image, and B is the background image.

[0325] 6) Calculate the foreground color label

[0326] Many cutout datasets only provide alpha images, but no foreground color images, which makes it impossible for the model to train the foreground color regression task on these data. This embodiment proposes to use traditional methods to estimate the foreground color, use it as a true value label, and add it to the network training to solve the pain point of missing labels. Among them, the traditional foreground color label estimation method can be Closed-form-matting or Multi-level method. This method does not need to be calculated in advance and saved as an image, but can be calculated in memory.

[0327] 7) Calculate the cutout loss

[0328] The cutout loss is used to calculate the difference between the prediction and the true value, which includes the Alpha loss and the foreground color loss. In this step, the segmentation loss (i.e., mask loss) is also calculated again.

[0329] Alpha loss:

[0330]

[0331]

[0332]

[0333] Foreground color loss:

[0334]

[0335] 8) Update the cutout decoder parameters

[0336] The network obtains the output mask image after one forward propagation. First, the segmentation loss L is calculated. seg Then the mask is fused with the encoding features and sent to the cutout decoder. The cutout decoder outputs the predicted alpha image and foreground color image. The loss is calculated according to the formula in 7). Finally, all the above losses are summarized to get the total loss L total . All parameters of the cutout decoder are updated according to the gradient descent method.

[0337] L total =L seg +L l1 +L comp +L lap +L fore

[0338] Steps 4), 5), 6), 7), and 8) are a complete process of updating the parameters of the cutout decoder. When the number of training rounds does not reach the preset number, the process will be repeated. When the number of training rounds reaches the preset number, the training ends and the process goes to step 9).

[0339] 9) Save model parameters

[0340] All structures participate in this round of training, so all model parameters are saved, that is, the parameters of the shared feature encoder, segmentation decoder, and cutout decoder are saved to obtain a trained image processing model.

[0341] In one embodiment, an image processing method is provided, including training and using an image processing model, the image processing method being applied to a computer device, including:

[0342] Training process:

[0343] A first sample image set and a second sample image set are obtained, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image.

[0344] An image processing model to be trained is determined, where the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure.

[0345] The first sample image is subjected to feature encoding processing through the initial encoding structure to obtain a plurality of first feature maps of different scales.

[0346] A first decoding process is performed based on a plurality of first feature maps of different scales through an initial first decoding structure to obtain a first mask image corresponding to the first sample image.

[0347] According to the difference between the segmentation labels corresponding to the first mask image and the first sample image, the initial encoding structure and the initial first decoding structure are trained until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure.

[0348] The second sample image is subjected to feature encoding processing by the encoding structure to obtain a plurality of second feature maps of different scales; and the first decoding structure is subjected to first decoding processing based on the plurality of second feature maps of different scales to obtain a second mask image corresponding to the second sample image.

[0349] A second decoding process is performed based on a plurality of second feature maps and second mask images of different scales through the initial second decoding structure to obtain a predicted foreground color map.

[0350] A second decoding process is performed based on a plurality of second feature maps and second mask images of different scales through the initial second decoding structure to obtain a plurality of sample mask maps of different levels.

[0351] The initial second decoding structure updates a plurality of different levels of sample mask maps based on the sample edge transition features between the sample body and the background area included in each sample mask map to obtain a predicted mask map.

[0352] Determine the foreground color loss between the predicted foreground color map and the foreground color label corresponding to the second sample image.

[0353] Determine the global loss between the predicted mask image and the corresponding matte label of the second sample image.

[0354] The predicted mask map is adjusted to different scales, and the scale loss of the predicted mask map at different scales is determined.

[0355] A mask loss between the second mask image and the segmentation label corresponding to the second sample image is determined.

[0356] When the second sample image is a composite image, channel losses of a predicted mask image and a corresponding cutout label on a color channel corresponding to the composite image are determined.

[0357] The target loss function is constructed based on foreground color loss, global loss, scale loss, mask loss and channel loss.

[0358] The initial second decoding structure is trained by the target loss function until the second stopping condition is met, and the trained second decoding structure is obtained to obtain the trained image processing model.

[0359] Use of image processing models:

[0360] The image processing model includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units; wherein the coding structure is used to perform feature coding processing on the original image, the first decoding structure is used to perform first decoding processing, and the second decoding structure is used to perform second decoding processing.

[0361] An original image is obtained and input into a coding structure; encoding is performed in sequence by a plurality of coding units in the coding structure based on input data corresponding to each coding unit, so as to obtain feature maps output by each coding unit; wherein the input data of a subsequent coding unit includes the feature map output by a previous coding unit, the input data of an initial coding unit in the coding structure is the original image, and the scales of feature maps output by different coding units in the coding structure are different.

[0362] Based on multiple feature maps of different scales, a first decoding process is performed in sequence based on input data corresponding to each of the first decoding units through multiple first decoding units in the first decoding structure, until a mask image is output through the last first decoding unit, wherein the input data of a current first decoding unit in the first decoding structure includes a decoding result corresponding to a previous first decoding unit and a feature map output by an encoding unit corresponding to the current first decoding unit, and the decoding result corresponding to the current first decoding unit is used to constitute the input data of a subsequent first decoding unit.

[0363] Determine input data corresponding to each second decoding unit in the second decoding structure, wherein the input data corresponding to the second decoding unit includes a feature map output by a corresponding encoding unit and a candidate mask map output by a previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes a mask image;

[0364] The second decoding processing is performed in sequence by multiple second decoding units in the second decoding structure based on the input data corresponding to each second decoding unit, so as to obtain candidate mask images output by each second decoding unit, wherein the candidate mask images output by each second decoding unit have different levels.

[0365] Furthermore, based on determining the input data corresponding to each second decoding unit in the second decoding structure, wherein the input data corresponding to the second decoding unit includes a feature map output by the corresponding encoding unit and a foreground color feature output by a previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes a mask image; second decoding processing is performed in sequence based on the input data corresponding to each second decoding unit through multiple second decoding units in the second decoding structure until the foreground color map output by the last second decoding unit is obtained.

[0366] The candidate mask map of the current level is enlarged to the same scale as the candidate mask map of the next level to obtain the enlarged candidate mask map; the edge transition image between the target body and the background area in the enlarged candidate mask map is extracted, and the edge transition image includes edge transition features; the edge transition image is morphologically processed to obtain a processed edge transition image, and the morphological processing includes at least one of an erosion operation and a dilation operation; the processed edge transition image and the candidate mask map of the next level are fused to obtain an updated mask map; the updated mask map is used as the candidate mask map of the current level in the next round, and the step of enlarging the candidate mask map of the current level to the same scale as the candidate mask map of the next level is returned and continued until the updated mask map corresponding to the candidate mask map of the last level is obtained; the updated mask map corresponding to the candidate mask map of the last level is used as the target mask map.

[0367] The foreground color image and the target mask image are fused to obtain a cutout result corresponding to the target subject.

[0368] The cutout result and the background image are synthesized to obtain a synthesized image.

[0369] In this embodiment, a target loss function is constructed based on foreground color loss, global loss, scale loss, mask loss and channel loss, and the initial second decoding structure is trained through the target loss function to obtain a trained second decoding structure. The trained encoding structure, the first decoding structure and the second decoding structure constitute a trained image processing model, which can effectively provide the segmentation accuracy and cutout accuracy of the image processing model.

[0370] The trained image processing model includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units. By performing feature coding processing on the original image through the coding structure, semantic features of different levels can be obtained, so that the first decoding structure performs a first decoding process based on semantic features of different levels, preliminarily predicts the position of the target subject in the original image, and reflects it through a mask image. The mask image represents the rough position of the target subject in the original image, and the second decoding structure uses semantic features of different levels and mask images to perform a second decoding process, which can effectively guide the second decoding structure to focus on the interpretation of the edge transition area between the target subject and the background area, thereby decoding and obtaining candidate mask images of different levels, and realizing strong attention regression to low-level features. The candidate mask map not only contains the position of the target subject in the original image, but also contains the feature information of the edge transition area between the target subject and the background area. By updating the candidate mask maps at different levels, it is possible to effectively achieve strong focus regression on low-level features, thereby obtaining a target mask map with clearer texture and finer edge transition details. Furthermore, the cutout result corresponding to the target subject can be accurately obtained according to the target mask map, further improving the accuracy of the cutout.

[0371] In addition, a second decoding process is performed based on multiple feature maps and mask images of different scales to accurately extract the color components corresponding to the foreground area in the original image, especially accurately obtain the color components corresponding to the edge transition area between the foreground and the background, and obtain a foreground color image including the target subject. The foreground color image and the target mask image are fused so that the edge details of the obtained cutout result are more informative and the edge color is more accurate. The cutout result and the background image are synthesized to obtain a composite image with clear and natural transition of edge details and natural color transition, making the composite image more natural and realistic.

[0372] like Fig.12 As shown in FIG. 1 , a schematic diagram comparing the image processing results obtained by the image processing model of this embodiment and the processing results obtained by the traditional method is provided. Fig.12 It can be seen that the image processing model of this embodiment processes the original image to obtain an intermediate segmentation result, i.e., a mask image, and then obtains a final result, i.e., a target mask image, based on the intermediate segmentation result. The hair texture of the final result is clear and delicate, achieving hair-level cutout. However, the traditional method directly obtains the segmentation result, resulting in background misdetection, and the area in the background area that is similar to the hair color is also segmented out, which is not accurate enough.

[0373] like Fig.13As shown, a conventional composite image without foreground color synthesis and a composite image with foreground color synthesis in the present application are provided. It can be clearly seen that the lack of foreground color will cause color artifacts to appear in the edge transition area during image synthesis, and the visual sensory effect is poor. In the present application, the foreground color image is obtained by cutting out the image, so that the color transition in the edge area is more natural and delicate during image synthesis, and the quality of the cutout synthesis is higher.

[0374] In addition, the images obtained by the image processing method of the present application are superior to the existing methods in various objective indicators. The comparison is shown in the following table:

[0375]

[0376] Among them, MODNet is a traditional real-time portrait background replacement algorithm, MAD (Mean Absolute Difference) refers to the mean absolute error, and MSE (Mean Squared Error) refers to the mean squared error. It can be seen from the above table that the mean absolute error and the mean squared error of this application are smaller than those of the traditional processing method, that is, the image processing method of this application can achieve higher precision and higher accuracy.

[0377] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0378] Based on the same inventive concept, the embodiments of the present application also provide an image processing device for implementing the above-mentioned image processing method, and a training device for an image processing model for implementing the above-mentioned training method of the image processing model. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more of the image processing device embodiments provided below can refer to the limitations on the image processing method above, and the specific limitations in one or more of the image processing model training device embodiments provided below can refer to the limitations on the image processing model training method above, which will not be repeated here.

[0379] In one embodiment, Fig.14As shown, an image processing device 1400 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an encoding module 1402, a first decoding module 1404, a second decoding module 1406, an updating module 1408 and a result determination module 1410, wherein:

[0380] The encoding module 1402 is used to perform feature encoding processing on the original image to obtain multiple feature maps of different scales.

[0381] The first decoding module 1404 is used to perform a first decoding process based on a plurality of feature maps of different scales to obtain a mask image, where the mask image is used to segment the target subject from the original image at a semantic level.

[0382] The second decoding module 1406 is used to perform a second decoding process based on the feature maps and mask images of multiple different scales to obtain candidate mask images of multiple different levels.

[0383] The updating module 1408 is used to update the candidate mask images at different levels according to the edge transition features between the target subject and the background area included in each candidate mask image, so as to obtain a target mask image.

[0384] The result determination module 1410 is used to determine the cutout result corresponding to the target subject according to the target mask image.

[0385] In this embodiment, the original image is subjected to feature encoding processing to obtain multiple feature maps of different scales, the large-scale feature map contains more low-level semantic features, and the small-scale feature map contains more high-level semantic features. Based on the feature maps of multiple different scales, the first decoding processing can identify the target subject in the original image based on the semantic features of different levels, and the position of the target subject in the original image is represented by the mask image at the semantic level to preliminarily segment the target subject. The mask image can provide stronger semantic information, and the second decoding processing based on the feature maps and mask images of multiple different scales can effectively guide the layer-by-layer interpretation of the edge transition area between the target subject and the background area. In addition, the levels of each candidate mask map are different, and the high-level candidate mask map contains more edge transition information between the target subject and the background area. According to the edge transition features in each candidate mask map, the candidate mask maps of multiple different levels are updated, which can effectively realize the strong focus regression on the low-level features, thereby obtaining a target mask map with clearer texture and more delicate edge transition details of the target subject, and then accurately obtain the cutout result corresponding to the target subject according to the target mask map, further improving the accuracy of the cutout.

[0386] In one embodiment, the device is executed by an image processing model, which includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units; wherein the coding structure is used to perform feature coding processing on the original image, the first decoding structure is used to perform a first decoding process, and the second decoding structure is used to perform a second decoding process.

[0387] In this embodiment, the image processing method is executed by an image processing model, which includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units. By performing feature coding processing on the original image through the coding structure, semantic features of different levels can be obtained, so that the first decoding structure performs a first decoding process based on semantic features of different levels, preliminarily predicts the position of the target subject in the original image, and reflects it through a mask image. The mask image indicates the rough position of the target subject in the original image, and then the second decoding structure uses semantic features of different levels and mask images to perform a second decoding process, which can effectively guide the second decoding structure to focus on the interpretation of the edge transition area between the target subject and the background area, thereby decoding to obtain candidate mask images of different levels, and realizing strong attention regression on low-level features. The candidate mask image not only contains the position of the target subject in the original image, but also contains the feature information of the edge transition area between the target subject and the background area, so that the target mask image with finer edge transition details can be obtained through candidate mask images of different levels.

[0388] In one embodiment, the encoding module 1402 is also used to obtain the original image and input the original image into the encoding structure; multiple encoding units in the encoding structure perform encoding processing based on the input data corresponding to each encoding unit in turn, to obtain feature maps output by each encoding unit; wherein the input data of the subsequent encoding unit includes the feature map output by the previous encoding unit, the input data of the initial encoding unit in the encoding structure is the original image, and the scales of the feature maps output by different encoding units in the encoding structure are different.

[0389] In this embodiment, the coding structure includes multiple coding units, the input data of the initial coding unit in the coding structure is the original image, and the feature map output by the previous coding unit is used as the input data of the subsequent coding unit, so that the semantic information in the original image can be extracted layer by layer through different coding units. The scale of the feature map output by each coding unit is different, so that the semantic information of different levels in the original image can be represented by feature maps of different scales. The large-scale feature map contains rich low-level information, and the small-scale feature map is a deep feature map that can reflect high-level semantic information, so that it can be decoded in combination with semantic information of different levels in the subsequent decoding to improve the accuracy of decoding.

[0390] In one embodiment, the first decoding module 1404 is also used to perform first decoding processing based on feature maps of multiple different scales, and in sequence through multiple first decoding units in the first decoding structure based on input data corresponding to each first decoding unit, until the mask image is output through the last first decoding unit, wherein the input data of the current first decoding unit in the first decoding structure includes a decoding result corresponding to the previous first decoding unit, and a feature map output by the encoding unit corresponding to the current first decoding unit, and the decoding result corresponding to the current first decoding unit is used to constitute the input data of the subsequent first decoding unit.

[0391] In this embodiment, the input data of the current first decoding unit includes the decoding result of the previous first decoding unit and the feature map output by the corresponding encoding unit, so that the current first decoding unit decodes again based on the previous decoding result and the corresponding encoding feature map, and the feature map represents the semantic information of the original image, so that after multiple decodings, the target subject in the original image can be identified, and the position of the target subject in the original image can be identified through the mask image at the semantic level to preliminarily segment the target subject.

[0392] In one embodiment, the second decoding module 1406 is also used to determine the input data corresponding to each second decoding unit in the second decoding structure, wherein the input data corresponding to the second decoding unit includes the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes a mask image; the second decoding processing is performed in turn based on the input data corresponding to each second decoding unit by multiple second decoding units in the second decoding structure, so as to obtain the candidate mask maps output by each second decoding unit, wherein the levels of the candidate mask maps output by each second decoding unit are different.

[0393] In this embodiment, the input data corresponding to some of the second decoding units in the second decoding structure include the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit before the second decoding unit, so that some of the second decoding units further decode by combining the corresponding encoding features and the candidate mask map obtained by the previous decoding to obtain a more accurate candidate mask map. In addition, the input data corresponding to at least a part of the second decoding units also includes a mask image, which can provide stronger semantic information, guide the second decoding unit to focus on edge transition area decoding, and further improve the accuracy of the interpretation of the edge transition area between the target subject and the background area. In addition, the candidate mask maps output by each second decoding unit have different levels, the low-level candidate mask map contains more texture information of the target subject, and the high-level candidate mask map contains more edge transition information between the target subject and the background area, so that a target mask map with clearer texture and clearer edge transition can be obtained based on multiple different levels of candidate mask maps.

[0394] In one embodiment, the update module 1408 is also used to enlarge the candidate mask image of the current level to the same scale as the candidate mask image of the next adjacent level to obtain an enlarged candidate mask image; extract the edge transition image between the target subject and the background area in the enlarged candidate mask image, wherein the edge transition image includes edge transition features; perform fusion processing based on the edge transition image and the candidate mask image of the next adjacent level to obtain an updated mask image; use the updated mask image as the candidate mask image of the current level in the next round, and return to the step of enlarging the candidate mask image of the current level to the same scale as the candidate mask image of the next adjacent level and continue to execute until the updated mask image corresponding to the candidate mask image of the last level is obtained; and use the updated mask image corresponding to the candidate mask image of the last level as the target mask image.

[0395] In this embodiment, the candidate mask map at the lower level contains more bottom-level information, that is, it contains more texture information of the target subject, and the candidate mask map at the higher level contains more deep information, that is, it contains more edge transition information between the target subject and the background area. The candidate mask map at the current level is enlarged to the same scale as the candidate mask map at the next adjacent level to extract the edge transition image between the target subject and the background area in the enlarged candidate mask map, so that the edge transition information can be extracted from the candidate mask map at the higher level. Based on the edge transition image and the candidate mask map at the next adjacent level, the edge transition information in the candidate mask map at the lower level can be updated through the edge transition information at the higher level, so that the edge transition details between the target subject and the background area in the candidate mask map at the lower level after the update are clearer, and at the same time, enough texture information of the target subject can be retained, thereby improving the accuracy of the cutout.

[0396] In one embodiment, the update module 1408 is also used to perform morphological processing on the edge transition image to obtain a processed edge transition image, and the morphological processing includes at least one of an erosion operation and a dilation operation; the processed edge transition image and the candidate mask image of the adjacent next level are fused to obtain an updated mask image.

[0397] In this embodiment, morphological processing and filtering processing can ensure that the edge transition image obtained has less or no noise and a softer edge. The processed edge transition image is fused with the candidate mask image of the next adjacent level, so that the edge transition details of the target subject in the obtained updated mask image are more integrated.

[0398] In one embodiment, the second decoding module 1406 is further used to perform a second decoding process based on a plurality of feature maps and mask images of different scales to obtain a foreground color image including the target subject;

[0399] The result determination module 1410 is further used to fuse the foreground color image and the target mask image to obtain a cutout result corresponding to the target subject.

[0400] In this embodiment, a second decoding process is performed based on multiple feature maps and mask images of different scales to accurately extract the color components corresponding to the foreground area in the original image, especially accurately obtain the color components corresponding to the edge transition area between the foreground and the background, and obtain a foreground color image including the target subject. The foreground color image and the target mask image are fused so that the edge details of the obtained cutout result are more informative and the edge color is more accurate.

[0401] In one embodiment, the apparatus is performed by an image processing model, and the apparatus further comprises:

[0402] The acquisition module is used to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image.

[0403] The model determination module is used to determine the image processing model to be trained, and the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure.

[0404] The first training module is used to perform first training on the initial coding structure and the initial first decoding structure through a first sample image set until a first stopping condition is met, thereby obtaining a trained coding structure and a first decoding structure.

[0405] The second training module is used to perform a second training on the initial second decoding structure through a second sample image set and based on the trained encoding structure and the first decoding structure, until a second stopping condition is met, to obtain a trained second decoding structure, so as to obtain a trained image processing model; the trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

[0406] In this embodiment, a first sample image set and a second sample image set are obtained, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image. An image processing model to be trained is determined, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure. The initial encoding structure and the initial first decoding structure are first trained through the first sample image set until a first stop condition is satisfied, and a trained encoding structure and a first decoding structure are obtained. On the basis of ensuring the processing accuracy of the encoding structure and the first decoding structure, the second sample image set is used and the initial second decoding structure is secondly trained based on the trained encoding structure and the first decoding structure, and the second stop condition is satisfied, and a trained second decoding structure is obtained. The trained encoding structure, the first decoding structure and the second decoding structure constitute a trained image processing model, which can effectively provide the segmentation accuracy and the cutout accuracy of the image processing model, so that the cutout result corresponding to the target subject in the original image can be accurately determined through the image processing model. Furthermore, by determining the cutout result corresponding to the target subject in the original image through the image processing model, the cutout efficiency can be effectively improved.

[0407] In one embodiment, Fig.15 As shown, a training device 1500 for an image processing model is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an acquisition module 1502, a model determination module 1504, a first training module 1506 and a second training module 1508, wherein:

[0408] The acquisition module 1502 is used to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image.

[0409] The model determination module 1504 is used to determine the image processing model to be trained, where the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure.

[0410] The first training module 1506 is used to perform first training on the initial coding structure and the initial first decoding structure through a first sample image set until a first stopping condition is met, thereby obtaining a trained coding structure and a first decoding structure.

[0411] The second training module 1508 is used to perform a second training on the initial second decoding structure through a second sample image set and based on the trained encoding structure and the first decoding structure, until a second stopping condition is met, to obtain a trained second decoding structure, so as to obtain a trained image processing model; the trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

[0412] In this embodiment, a first sample image set and a second sample image set are obtained, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image. An image processing model to be trained is determined, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure and an initial second decoding structure. The initial encoding structure and the initial first decoding structure are first trained through the first sample image set until a first stop condition is satisfied, and a trained encoding structure and a first decoding structure are obtained. On the basis of ensuring the processing accuracy of the encoding structure and the first decoding structure, the second sample image set is used and the initial second decoding structure is secondly trained based on the trained encoding structure and the first decoding structure, and the second stop condition is satisfied, and a trained second decoding structure is obtained. The trained encoding structure, the first decoding structure and the second decoding structure constitute a trained image processing model, which can effectively provide the segmentation accuracy and the cutout accuracy of the image processing model, so that the cutout result corresponding to the target subject in the original image can be accurately determined through the image processing model. Furthermore, by determining the cutout result corresponding to the target subject in the original image through the image processing model, the cutout efficiency can be effectively improved.

[0413] In one embodiment, the first training module 1506 is further used to perform feature encoding processing on the first sample image through the initial encoding structure to obtain multiple first feature maps of different scales; perform first decoding processing based on the multiple first feature maps of different scales through the initial first decoding structure to obtain a first mask image corresponding to the first sample image; according to the difference between the segmentation labels corresponding to the first mask image and the first sample image, the initial encoding structure and the initial first decoding structure are trained until the first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure.

[0414] In this embodiment, the first sample image is feature encoded by the initial encoding structure to obtain a plurality of first feature maps of different scales, and the first mask image corresponding to the first sample image is obtained by the initial first decoding structure based on the first feature maps of different scales. The initial encoding structure and the initial first decoding structure are trained according to the difference between the segmentation labels corresponding to the first mask image and the first sample image until the first stop condition is met, so that the trained encoding structure can accurately extract semantic information at different levels. The large-scale feature map contains rich low-level information, and the small-scale feature map can reflect high-level semantic information, so that the initial first decoding structure can combine semantic information at different levels for decoding, thereby improving the accuracy of decoding.

[0415] In one embodiment, the second training module 1508 is also used to perform feature encoding processing on the second sample image through the encoding structure to obtain multiple second feature maps of different scales; perform first decoding processing based on the second feature maps of multiple scales through the first decoding structure to obtain a second mask image corresponding to the second sample image; perform second decoding processing based on the second feature maps of multiple scales and the second mask image through the initial second decoding structure to obtain multiple different levels of sample mask maps; update the multiple different levels of sample mask maps based on the sample edge transition features between the sample body and the background area included in each sample mask map through the initial second decoding structure to obtain a predicted mask map; train the initial second decoding structure based on the mask loss between the predicted mask map and the cutout label corresponding to the second sample image until the second stop condition is met to obtain a trained second decoding structure.

[0416] In this embodiment, the initial second decoding structure uses different levels of semantic features and mask images to perform the second decoding process, which can effectively guide the second decoding structure to focus on the interpretation of the edge transition area between the target subject and the background area, thereby decoding to obtain sample mask maps of different levels, so as to achieve strong focus regression on low-level features during the training process. The sample mask map not only contains the position of the sample subject in the original image, but also contains the feature information of the edge transition area between the sample subject and the background area, so that a predicted mask map with finer edge transition details can be obtained through sample mask maps of different levels, and based on the mask loss between the predicted mask map and the cutout label corresponding to the second sample image, the initial second decoding structure is trained, and the parameters of the second decoding structure can be continuously adjusted, so that the trained second decoding structure can predict the edge transition details in the mask map more finely and delicately, making the cutout more accurate.

[0417] In one embodiment, the mask loss includes a global loss and a scale loss; the second training module 1508 is further used to determine the global loss between the predicted mask image and the cutout label corresponding to the second sample image; the predicted mask image is adjusted to different scales, and the scale loss of the predicted mask image at different scales is determined; based on the global loss and the scale loss, a target loss function is constructed; the initial second decoding structure is trained by the target loss function until the second stop condition is met, and a trained second decoding structure is obtained.

[0418] In this embodiment, the target loss function is constructed by combining the difference between the predicted mask image and the actual cutout label, as well as the difference between the predicted mask images of different scales and the actual cutout labels. This can take into account the impact of prediction results of different scales on the image processing model, so that the trained second decoding structure can be compatible with the processing of images of different scales, so that the generated predicted mask image is more accurate.

[0419] In one embodiment, the second training module 1508 is also used to determine the mask loss between the second mask image and the segmentation label corresponding to the second sample image; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the mask loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

[0420] In this embodiment, the mask loss can represent the loss generated by the trained encoding structure and the first decoding structure, and the mask loss can represent the loss generated by the initial second decoding structure. Combining the mask loss and the mask loss to train the initial second decoding structure can further improve the accuracy of the second decoding structure, so that the mask image predicted by the trained second decoding structure is more accurate.

[0421] In one embodiment, the second sample image set also includes a foreground color label corresponding to the second sample image. The second training module 1508 is further used to perform a second decoding process based on a plurality of second feature maps of different scales and a second mask image through the initial second decoding structure to obtain a predicted foreground color image; determine the foreground color loss between the predicted foreground color image and the foreground color label corresponding to the second sample image; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image; construct a target loss function based on the mask loss and the foreground color loss; train the initial second decoding structure through the target loss function until the second stop condition is met, and obtain a trained second decoding structure.

[0422] In this embodiment, a second decoding process is performed based on feature maps and mask images of multiple different scales to accurately extract the color components corresponding to the foreground area in the original image, especially accurately obtain the color components corresponding to the edge transition area between the foreground and the background, and obtain a predicted foreground color map including the main body of the sample. The initial second decoding structure is trained in combination with the foreground color loss and the mask loss, and the parameters of the foreground color map and the mask map corresponding to the predicted image of the second decoding structure can be trained simultaneously, so that the trained second decoding structure can share the output data of the encoding structure and the first decoding structure, and independently predict the foreground color map and the mask map corresponding to the image based on the shared output data, and make the edge details of the obtained cutout result more informative and the edge color more accurate.

[0423] In one embodiment, the second sample image includes a composite image; the second training module 1508 is further used to determine the channel loss of the predicted mask image and the corresponding cutout label corresponding to the composite image on the color channel; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the channel loss; train the initial second decoding structure through the target loss function until the second stop condition is met, and obtain a trained second decoding structure.

[0424] In this embodiment, the use of synthetic images in training can improve the richness and complexity of training data, and can effectively improve the recognition ability of the image processing model. In addition, the combination of the loss on different color channels and the mask loss on the second decoding structure can further improve the accuracy of the second decoding structure decoding.

[0425] Each module in the above-mentioned image processing device and image processing model training device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0426] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. This embodiment is described by taking the computer device as a server as an example. The internal structure diagram thereof may be as follows: Fig.16As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0427] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0428] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0429] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0430] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0431] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0432] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0433] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0434] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Perform feature encoding processing on the original image to obtain multiple feature maps of different scales; Performing a first decoding process based on the multiple feature maps of different scales to obtain a mask image, wherein the mask image is used to segment a target subject from the original image at a semantic level; Performing a second decoding process based on the multiple feature maps of different scales and the mask image to obtain multiple candidate mask maps of different levels; According to the edge transition features between the target subject and the background area included in each candidate mask image, the candidate mask images at different levels are updated to obtain a target mask image; A cutout result corresponding to the target subject is determined according to the target mask image.

2. The method according to claim 1, characterized in that The method is executed by an image processing model, which includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units; wherein the coding structure is used to perform feature coding processing on the original image, the first decoding structure is used to perform first decoding processing, and the second decoding structure is used to perform second decoding processing.

3. The method according to claim 2, characterized in that The feature encoding process is performed on the original image to obtain multiple feature maps of different scales, including: Acquire an original image, and input the original image into a coding structure; The encoding processing is performed in sequence on the input data corresponding to each encoding unit by the multiple encoding units in the encoding structure, so as to obtain feature maps output by each encoding unit; wherein the input data of the subsequent encoding unit includes the feature map output by the previous encoding unit, the input data of the initial encoding unit in the encoding structure is the original image, and the scales of the feature maps output by different encoding units in the encoding structure are different.

4. The method according to claim 2, characterized in that: The first decoding process is performed based on the feature maps of the multiple different scales to obtain a mask image, including: Based on the multiple feature maps of different scales, a first decoding process is performed in sequence based on the input data corresponding to each of the first decoding units through the multiple first decoding units in the first decoding structure, until the mask image is output through the last first decoding unit, wherein the input data of the current first decoding unit in the first decoding structure includes a decoding result corresponding to the previous first decoding unit and a feature map output by the encoding unit corresponding to the current first decoding unit, and the decoding result corresponding to the current first decoding unit is used to constitute the input data of the subsequent first decoding unit.

5. The method according to claim 2, characterized in that: The second decoding process is performed based on the feature maps of the multiple different scales and the mask image to obtain multiple candidate mask maps of different levels, including: Determine input data corresponding to each second decoding unit in the second decoding structure, wherein the input data corresponding to the second decoding unit includes a feature map output by a corresponding encoding unit and a candidate mask map output by a previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes the mask image; The second decoding processing is performed in sequence by multiple second decoding units in the second decoding structure based on the input data corresponding to each second decoding unit, so as to obtain candidate mask images output by each second decoding unit, wherein the candidate mask images output by each second decoding unit have different levels.

6. The method according to claim 1, characterized in that The updating process of the candidate mask images at different levels is performed according to the edge transition features between the target subject and the background area included in each candidate mask image to obtain the target mask image, including: Enlarging the candidate mask image of the current level to the same scale as the candidate mask image of the next adjacent level to obtain an enlarged candidate mask image; Extracting an edge transition image between the target subject and the background area in the enlarged candidate mask image, wherein the edge transition image includes edge transition features; Performing fusion processing based on the edge transition image and the candidate mask image of the adjacent next level to obtain an updated mask image; The updated mask image is used as the candidate mask image of the current level in the next round, and the step of enlarging the candidate mask image of the current level to the same scale as the candidate mask image of the next level is returned to and continued until the updated mask image corresponding to the candidate mask image of the last level is obtained; The updated mask image corresponding to the candidate mask image of the last level is used as the target mask image.

7. The method according to claim 6, characterized in that The step of fusing the edge transition image and the candidate mask image at the next adjacent level to obtain an updated mask image includes: Performing morphological processing on the edge transition image to obtain a processed edge transition image, wherein the morphological processing includes at least one of an erosion operation and a dilation operation; The processed edge transition image and the candidate mask image of the adjacent next level are fused to obtain an updated mask image.

8. The method according to claim 1, characterized in that The method further comprises: Performing a second decoding process based on the multiple feature maps of different scales and the mask image to obtain a foreground color map including the target subject; The step of determining a cutout result corresponding to the target subject according to the target mask image includes: The foreground color image and the target mask image are fused to obtain a cutout result corresponding to the target subject.

9. The method according to any one of claims 1 to 8, characterized in that: The method is performed by an image processing model, and the image processing model is obtained through a training step, and the training step includes: Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image; the segmentation label is a real mask image corresponding to the first sample image, and the cutout label is a real mask image corresponding to the second sample image; Determining an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure; Performing a first training on the initial encoding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure; The initial second decoding structure is subjected to second training through the second sample image set and based on the trained encoding structure and the first decoding structure, and the training is stopped when the second stopping condition is met, thereby obtaining a trained second decoding structure and obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

10. A training method for an image processing model, characterized in that: The method comprises: Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image; the segmentation label is a real mask image corresponding to the first sample image, and the cutout label is a real mask image corresponding to the second sample image; Determining an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure; Performing a first training on the initial encoding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure; The initial second decoding structure is subjected to second training through the second sample image set and based on the trained encoding structure and the first decoding structure, and the training is stopped when the second stopping condition is met, thereby obtaining a trained second decoding structure and obtaining a trained image processing model; the trained image processing model is used to determine a cutout result corresponding to a target subject in the original image.

11. The method according to claim 10, characterized in that The first training of the initial coding structure and the initial first decoding structure is performed by using the first sample image set until a first stopping condition is satisfied, thereby obtaining a trained coding structure and a first decoding structure, including: Performing feature encoding processing on the first sample image by using the initial encoding structure to obtain a plurality of first feature maps of different scales; Performing a first decoding process based on the plurality of first feature maps of different scales through the initial first decoding structure to obtain a first mask image corresponding to the first sample image; According to the difference between the segmentation labels corresponding to the first mask image and the first sample image, the initial encoding structure and the initial first decoding structure are trained until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure.

12. The method according to claim 10, characterized in that The second sample image set is used to perform a second training on the initial second decoding structure based on the trained encoding structure and the first decoding structure until a second stopping condition is met to obtain a trained second decoding structure, including: Performing feature encoding processing on the second sample image by using the encoding structure to obtain a plurality of second feature maps of different scales; Performing a first decoding process based on the plurality of second feature maps of different scales through the first decoding structure to obtain a second mask image corresponding to the second sample image; Performing a second decoding process based on the plurality of second feature maps of different scales and the second mask image through the initial second decoding structure to obtain a plurality of sample mask maps of different levels; The initial second decoding structure updates the sample mask images at different levels based on the sample edge transition features between the sample body and the background area included in each sample mask image to obtain a predicted mask image; Based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, the initial second decoding structure is trained until a second stopping condition is met, thereby obtaining a trained second decoding structure.

13. The method according to claim 12, characterized in that The mask loss includes a global loss and a scale loss; the initial second decoding structure is trained based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image until a second stop condition is met, thereby obtaining a trained second decoding structure, including: Determining a global loss between the predicted mask image and a matting label corresponding to the second sample image; Adjusting the predicted mask image to different scales, and determining scale losses of the predicted mask image at the different scales; Based on the global loss and the scale loss, construct a target loss function; The initial second decoding structure is trained by the target loss function until a second stopping condition is met, thereby obtaining a trained second decoding structure.

14. The method according to claim 12, characterized in that The method further comprises: Determining a mask loss between the second mask image and the segmentation label corresponding to the second sample image; The training of the initial second decoding structure based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image until a second stopping condition is satisfied to obtain a trained second decoding structure includes: Determine a mask loss between the predicted mask image and a matting label corresponding to the second sample image, and construct a target loss function based on the mask loss and the mask loss; The initial second decoding structure is trained by the target loss function until a second stopping condition is met, thereby obtaining a trained second decoding structure.

15. The method according to claim 12, characterized in that The second sample image set also includes a foreground color label corresponding to the second sample image, and the method further includes: Performing a second decoding process based on the plurality of second feature maps of different scales and the second mask image through the initial second decoding structure to obtain a predicted foreground color map; Determining a foreground color loss between the predicted foreground color image and a foreground color label corresponding to the second sample image; The training of the initial second decoding structure based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image until a second stopping condition is satisfied to obtain a trained second decoding structure includes: Determining a mask loss between the predicted mask image and a matting label corresponding to the second sample image; Based on the mask loss and the foreground color loss, construct a target loss function; The initial second decoding structure is trained by the target loss function until a second stopping condition is met, thereby obtaining a trained second decoding structure.

16. The method according to claim 12, characterized in that The second sample image includes a synthetic image; and the method further includes: Determine the channel loss of the predicted mask image and the corresponding cutout label corresponding to the composite image on the color channel; The training of the initial second decoding structure based on the mask loss between the predicted mask image and the cutout label corresponding to the second sample image until a second stopping condition is satisfied to obtain a trained second decoding structure includes: Determine a mask loss between the predicted mask image and the matting label corresponding to the second sample image, and construct a target loss function based on the mask loss and the channel loss; The initial second decoding structure is trained by the target loss function until a second stopping condition is met, thereby obtaining a trained second decoding structure.

17. An image processing device, characterized in that: The device comprises: The encoding module is used to perform feature encoding processing on the original image to obtain multiple feature maps of different scales; A first decoding module, configured to perform a first decoding process based on the plurality of feature maps of different scales to obtain a mask image, wherein the mask image is used to segment a target subject from the original image at a semantic level; A second decoding module, configured to perform a second decoding process based on the plurality of feature maps of different scales and the mask image to obtain a plurality of candidate mask maps of different levels; An updating module, configured to update the candidate mask images at different levels according to edge transition features between the target subject and the background area included in each candidate mask image, to obtain a target mask image; A result determination module is used to determine a cutout result corresponding to the target subject according to the target mask image.

18. The device according to claim 17, characterized in that The device is executed by an image processing model, which includes a coding structure composed of multiple coding units, a first decoding structure composed of multiple first decoding units, and a second decoding structure composed of multiple second decoding units; wherein the coding structure is used to perform feature coding processing on the original image, the first decoding structure is used to perform first decoding processing, and the second decoding structure is used to perform second decoding processing.

19. The device according to claim 18, characterized in that The encoding module is also used to obtain an original image and input the original image into a coding structure; multiple encoding units in the encoding structure perform encoding processing in turn based on the input data corresponding to each encoding unit, so as to obtain feature maps output by each encoding unit; wherein the input data of the subsequent encoding unit includes the feature map output by the previous encoding unit, the input data of the initial encoding unit in the encoding structure is the original image, and the scales of the feature maps output by different encoding units in the encoding structure are different.

20. The device according to claim 18, characterized in that The first decoding module is also used to perform first decoding processing based on the input data corresponding to each of the first decoding units in sequence based on the feature maps of the multiple different scales, and through the multiple first decoding units in the first decoding structure, until the mask image is output through the last first decoding unit, wherein the input data of the current first decoding unit in the first decoding structure includes a decoding result corresponding to the previous first decoding unit and a feature map output by the encoding unit corresponding to the current first decoding unit, and the decoding result corresponding to the current first decoding unit is used to constitute the input data of the subsequent first decoding unit.

21. The device according to claim 18, characterized in that The second decoding module is also used to determine the input data corresponding to each second decoding unit in the second decoding structure, wherein the input data corresponding to the second decoding unit includes the feature map output by the corresponding encoding unit and the candidate mask map output by the previous second decoding unit before the second decoding unit, and the input data corresponding to at least a part of the second decoding units also includes the mask image; the second decoding processing is performed in turn based on the input data corresponding to each second decoding unit by multiple second decoding units in the second decoding structure, so as to obtain the candidate mask maps output by each second decoding unit, wherein the levels of the candidate mask maps output by each second decoding unit are different.

22. The device according to claim 17, characterized in that The updating module is further used to enlarge the candidate mask image of the current level to the same scale as the candidate mask image of the next adjacent level to obtain an enlarged candidate mask image; extract an edge transition image between the target subject and the background area in the enlarged candidate mask image, wherein the edge transition image includes edge transition features; Based on the edge transition image and the candidate mask map of the adjacent next layer, a fusion process is performed to obtain an updated mask map; the updated mask map is used as the candidate mask map of the current layer in the next round, and the step of enlarging the candidate mask map of the current layer to the same scale as the candidate mask map of the adjacent next layer is returned and continued until the updated mask map corresponding to the candidate mask map of the last layer is obtained; the updated mask map corresponding to the candidate mask map of the last layer is used as the target mask map.

23. The device according to claim 22, characterized in that The updating module is further used to perform morphological processing on the edge transition image to obtain a processed edge transition image, wherein the morphological processing includes at least one of an erosion operation and a dilation operation; and to fuse the processed edge transition image with the candidate mask image of the adjacent next level to obtain an updated mask image.

24. The device according to claim 17, characterized in that The second decoding module is further used to perform a second decoding process based on the multiple feature maps of different scales and the mask image to obtain a foreground color map including the target subject; The result determination module is further used to fuse the foreground color image and the target mask image to obtain a cutout result corresponding to the target subject.

25. The device according to any one of claims 17 to 24, characterized in that The device is executed by an image processing model, and the device also includes: an acquisition module, configured to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image; the segmentation label is a real mask image corresponding to the first sample image, and the cutout label is a real mask image corresponding to the second sample image; A model determination module, used to determine an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure; A first training module, configured to perform a first training on the initial coding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is satisfied, thereby obtaining a trained coding structure and a first decoding structure; The second training module is used to perform a second training on the initial second decoding structure through the second sample image set and based on the trained encoding structure and the first decoding structure, until the second stopping condition is met, to obtain the trained second decoding structure, so as to obtain a trained image processing model; the trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

26. A training device for an image processing model, characterized in that: The device comprises: an acquisition module, configured to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a first sample image and a segmentation label corresponding to the first sample image, and the second sample image set includes a second sample image and a cutout label corresponding to the second sample image; the segmentation label is a real mask image corresponding to the first sample image, and the cutout label is a real mask image corresponding to the second sample image; A model determination module, used to determine an image processing model to be trained, wherein the image processing model to be trained includes an initial encoding structure, an initial first decoding structure, and an initial second decoding structure; A first training module, configured to perform a first training on the initial coding structure and the initial first decoding structure by using the first sample image set until a first stopping condition is satisfied, thereby obtaining a trained coding structure and a first decoding structure; The second training module is used to perform a second training on the initial second decoding structure through the second sample image set and based on the trained encoding structure and the first decoding structure, until the second stopping condition is met, to obtain the trained second decoding structure, so as to obtain a trained image processing model; the trained image processing model is used to determine the cutout result corresponding to the target subject in the original image.

27. The device according to claim 26, characterized in that The first training module is further used to perform feature encoding processing on the first sample image through the initial encoding structure to obtain multiple first feature maps of different scales; perform first decoding processing based on the multiple first feature maps of different scales through the initial first decoding structure to obtain a first mask image corresponding to the first sample image; and train the initial encoding structure and the initial first decoding structure according to the difference between the segmentation labels corresponding to the first mask image and the first sample image until a first stopping condition is met, thereby obtaining a trained encoding structure and a first decoding structure.

28. The device according to claim 26, characterized in that The second training module is further configured to perform feature encoding processing on the second sample image through the encoding structure to obtain a plurality of second feature maps of different scales; and perform first decoding processing based on the plurality of second feature maps of different scales through the first decoding structure to obtain a second mask image corresponding to the second sample image; The initial second decoding structure performs a second decoding process based on the multiple second feature maps of different scales and the second mask image to obtain sample mask maps of multiple different levels; the initial second decoding structure performs an update process on the sample mask maps of multiple different levels based on the sample edge transition features between the sample body and the background area included in each sample mask map to obtain a predicted mask map; the initial second decoding structure is trained based on the mask loss between the predicted mask map and the cutout label corresponding to the second sample image until a second stop condition is met, thereby obtaining a trained second decoding structure.

29. The device according to claim 28, characterized in that The mask loss includes a global loss and a scale loss; the second training module is further used to determine the global loss between the predicted mask image and the cutout label corresponding to the second sample image; the predicted mask image is adjusted to different scales, and the scale loss of the predicted mask image at the different scales is determined; based on the global loss and the scale loss, a target loss function is constructed; the initial second decoding structure is trained by the target loss function until the second stopping condition is met, and a trained second decoding structure is obtained.

30. The device according to claim 28, characterized in that The second training module is also used to determine the mask loss between the second mask image and the segmentation label corresponding to the second sample image; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the mask loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

31. The device according to claim 28, characterized in that The second sample image set also includes a foreground color label corresponding to the second sample image. The second training module is further used to perform a second decoding process based on the second feature maps of multiple different scales and the second mask image through the initial second decoding structure to obtain a predicted foreground color image; determine the foreground color loss between the predicted foreground color image and the foreground color label corresponding to the second sample image; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image; construct a target loss function based on the mask loss and the foreground color loss; train the initial second decoding structure through the target loss function until the second stop condition is met, and obtain a trained second decoding structure.

32. The device according to claim 28, characterized in that The second sample image includes a composite image; the second training module is also used to determine the channel loss of the predicted mask image and the corresponding cutout label corresponding to the composite image on the color channel; determine the mask loss between the predicted mask image and the cutout label corresponding to the second sample image, and construct a target loss function based on the mask loss and the channel loss; train the initial second decoding structure through the target loss function until the second stopping condition is met, and obtain a trained second decoding structure.

33. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

34. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

35. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images