Portrait segmentation processing method and device, medium and program product

Abstract

The invention provides a portrait segmentation processing method and device, a medium and a program product. The method comprises the following steps: acquiring a target input image; encoding the target input image through a pre-training encoder to obtain image features; carrying out portrait state recognition on the image features through a pre-trained portrait state recognizer to obtain a portrait orientation identifier; selecting a corresponding target decoder from a plurality of pre-training decoders according to the portrait orientation identifier, the plurality of pre-training decoders being used for performing portrait segmentation on the input images with different portrait orientations; and decoding the image features through a target decoder to obtain a target portrait mask. According to the embodiment of the invention, the accuracy of portrait segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of portrait segmentation technology, and in particular to a portrait segmentation processing method, device, medium and program product. Background Technology

[0002] The goal of portrait segmentation technology is to separate the foreground (usually a person) from the background in an image, thereby achieving accurate portrait recognition. Portrait segmentation technology can be applied to various scenarios, such as virtual backgrounds for 5G calls, video conferencing, and Region of Interest (ROI) encoding. It eliminates the need for green screen technology to replace the real background with a more professional or interesting one, thus hiding the user's actual environment. Alternatively, it can be used for ROI encoding, allocating more bitrate to the regions of the face that are of interest to the human eye, thereby enhancing subjective quality.

[0003] Despite the wide range of applications for human face segmentation technology, some problems and challenges still exist. In complex or dynamically changing backgrounds, missegmentation may still occur. For example, when a user is in a complex background, or when the background changes dynamically as the user moves, objects in the background may sometimes be incorrectly segmented as foreground objects. For scenes where the human face is not oriented forward, the accuracy of human face segmentation is relatively low. Summary of the Invention

[0004] This application provides a portrait segmentation processing method, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of portrait segmentation.

[0005] In a first aspect, embodiments of this application provide a portrait segmentation processing method, the method comprising: Obtain the target input image; The target input image is encoded using a pre-trained encoder to obtain image features; The image features are analyzed by a pre-trained human state recognizer to identify the human state and obtain the human orientation identifier. The target decoder is selected from multiple pre-trained decoders based on the portrait orientation identifier, wherein the multiple pre-trained decoders are used to perform portrait segmentation on input images with different portrait orientations; The target decoder decodes the image features to obtain the target human image mask.

[0006] Secondly, embodiments of this application provide an electronic device, including: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the portrait segmentation processing method as described in the first aspect.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the portrait segmentation processing method as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the portrait segmentation processing method as described in the first aspect.

[0009] In this embodiment, the target input image is first acquired, and then encoded by a pre-trained encoder to obtain image features. A pre-trained human face state recognizer then performs human face state recognition on the image features to obtain a human face orientation identifier. Finally, based on the human face orientation identifier, a corresponding target decoder is selected from multiple pre-trained decoders to decode the image features and obtain a target human face mask. By setting corresponding pre-trained decoders specifically responsible for outputting human face masks for different human face orientations, and using the human face orientation identifier as prior information to dynamically select different pre-trained decoder branches to output human face masks, the accuracy of human face segmentation can be improved when the human face orientation in the input image is unknown. Attached Figure Description

[0010] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0011] Figure 1 These are the results of forward human portrait segmentation and downward human portrait segmentation in existing technologies. Figure 2 The results of human portrait segmentation facing right and human portrait segmentation facing left are shown in the existing technology; Figure 3 A schematic block diagram of a third target portrait segmentation network applying the portrait segmentation processing method provided in the embodiments of this application; Figure 4 This is a schematic block diagram of the DSS module in an embodiment of this application; Figure 5 This is a schematic block diagram of the DSSM in the DSS module of the pre-trained encoder in the embodiments of this application; Figure 6 This is a schematic block diagram of the DSSM in the DSS module of the pre-trained decoder in the embodiments of this application; Figure 7A flowchart illustrating a portrait segmentation method provided in an embodiment of this application; Figure 8 for Figure 7 A detailed flowchart of step S500 is shown below; Figure 9 for Figure 7 A detailed flowchart of step S200 is shown below; Figure 10 for Figure 9 A detailed flowchart of step S210 is shown below; Figure 11 This is a schematic diagram of the model training process for the pre-trained encoder and pre-trained decoder in the embodiments of this application; Figure 12 This is a schematic diagram illustrating the process of obtaining face mask tags in an embodiment of this application; Figure 13 This is a schematic diagram of the model training process of the pre-trained human face state recognition device in the embodiments of this application; Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions provided in this application will be described in detail below with reference to the accompanying drawings.

[0013] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, the described exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this application.

[0014] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of a feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0016] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0017] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in the embodiments of this application.

[0018] To facilitate a better understanding of the solutions in the embodiments of this application, the relevant technologies will be introduced first below.

[0019] Human face segmentation is an image processing technique that aims to separate the human figure from the background in an image. It extracts the pixels representing the human figure from the original image, creating a binary image where pixels representing the human figure are set to white, and pixels representing the non-human figure are set to black. This technique is widely used in various scenarios, including portrait cutout enhancement, photo background replacement, ID photo creation, and privacy protection. Human face segmentation can identify the human figure's outline in an image and separate it from the background, adapting to single or multiple figures, complex backgrounds, and various human poses. It supports separating the human figure's outline from the image background, returning a segmented binary image, grayscale image, or a foreground image with a transparent background, thus achieving accurate extraction of the human figure. The application of this technique not only improves the efficiency of image processing but also facilitates subsequent image analysis and applications.

[0020] A mask is a tool used to cover a portion or all of an image being processed, controlling the area or process of image processing. The specific image or object used for covering is called a mask or template. In optical image processing, masks can be films, filters, etc. A mask is a binary image composed of 0s and 1s. When a mask is applied in a function, regions with 1s are processed, while regions with 0s are not included in the calculation. Image masks are defined by specifying data values, data ranges, finite or infinite values, regions of interest, and annotation files. Any combination of these options can be used as input to create a mask. A mask is a template for an image filter. Practical masks are frequently used in remote sensing images. When extracting roads, rivers, or buildings, an N*N matrix is ​​used to filter pixels in the image, highlighting the desired features or landmarks. This matrix is ​​a type of mask.

[0021] Digital slow shutter (DSS) technology is a camera technology designed to improve camera performance in low light conditions. By reducing the image frame refresh rate and increasing the camera's sensitivity, DSS reduces image noise while maintaining image sharpness and detail in low-light conditions. This technology reduces the camera's exposure time, allowing for sufficiently clear images to be captured even in low-light environments, thus improving overall image quality. Furthermore, DSS technology involves wide dynamic range (WDR) technology in image processing. This is a technique that captures two images with different sharp areas at different exposure speeds within the same time frame. By eliminating inferior pixels and retaining superior pixels, and then combining the two, a WDR of over 120dB is achieved. This technology makes details in dark areas of the image clearly visible while suppressing saturation in bright areas, thereby improving the overall visibility and detail of the image.

[0022] Deep learning models (Deep Structured Semantic Model, DSSM): Primarily used to learn semantic representations of text.

[0023] The goal of portrait segmentation technology is to separate the foreground (usually a person) from the background in an image, thereby achieving accurate portrait recognition. Portrait segmentation technology can be applied to various scenarios, such as virtual backgrounds for 5G calls, video conferencing, and Region of Interest (ROI) coding. It eliminates the need for green screen technology to replace the real background with a more professional or interesting one, thus hiding the user's actual environment. Alternatively, it can be used for ROI coding, allocating more bitrate to the regions of interest to the human eye, thereby enhancing subjective quality. Despite its wide range of applications, portrait segmentation technology still faces some challenges. In complex or dynamically changing backgrounds, missegmentation can still occur. For example, when a user is in a complex background, or when the background changes dynamically as the user moves, objects in the background may sometimes be incorrectly segmented as foreground. In scenarios where the orientation of the portrait cannot be determined, the segmentation accuracy is lower for other orientations (such as downward, left, right, etc.) compared to a facing portrait. Figure 1 and Figure 2 As shown, some pixels of the portrait portion facing down, left, and right are also set to black.

[0024] Based on this, this application proposes a portrait segmentation processing method, device, medium, and program product. By setting up a pre-trained decoder specifically responsible for outputting the portrait mask for different portrait orientations, and using the portrait orientation identifier as prior information to dynamically select different pre-trained decoder branches to output the portrait mask, the accuracy of portrait segmentation can be improved when the portrait orientation in the input image is unknown. Moreover, the computational complexity is low and it can meet the real-time requirements.

[0025] Please refer to Figure 3 , Figure 3 This is a schematic block diagram of a third target portrait segmentation network applying the portrait segmentation processing method provided in the embodiments of this application. Figure 3 As shown, the third target human image segmentation network includes a pre-trained encoder, a pre-trained human image state recognizer, and multiple pre-trained decoders; A pre-trained encoder is used to acquire the target input image, encode the target input image, and obtain image features; A pre-trained human state recognizer is used to recognize human state from image features and obtain human orientation indicators. Multiple pre-trained decoders are used to segment the human figure from input images with different orientations.

[0026] In this embodiment, the pre-trained encoder consists of a series of Nx and DSS modules (mode 0), where Nx is a downsampling module, such as... Figure 4 As shown, Figure 4 This is a schematic block diagram of the DSS module in an embodiment of this application. The DSS module includes channel separation, layer normalization, multiple linear mapping layers, depthwise separable convolution, oriented state-space model (DSSM), and channel concatenation. In the pre-trained encoder, as... Figure 5 As shown, Figure 5 This is a schematic block diagram of the DSSM in the DSS module of the pre-trained encoder in this embodiment. The DSSM includes a pre-trained feature scanning model, a pre-trained state space model, and channel stitching connected in sequence. The pre-trained feature scanning model includes feature scanning in four directions: left to right, top to bottom, right to left, and bottom to top. The scanning information in the four directions is fed into the pre-trained state space model and modeled using formula (1) to obtain feature information in the four directions. Channel stitching is then performed on the feature information in the four directions to obtain image features. Therefore, this pre-trained encoder structure can adapt to feature extraction when the portrait is facing different directions.

[0027] Formula (1) in, A Here is the state transition matrix. B This represents the matrix showing the influence of the control quantity on the state quantity.C This represents the matrix showing the influence of the current state variables on the output. D This is the matrix showing the effect of the current control input on the output. For the current state quantity, For the observed state variables, To control the quantity, This is the output of the state-space model.

[0028] A pre-trained human face state recognizer is positioned between a pre-trained encoder and multiple pre-trained decoders. The pre-trained human face state recognizer receives image features encoded by the pre-trained encoder from the target input image and generates a human face orientation identifier based on these features, indicating whether the target input image contains a human face and the direction of that human face. Based on the human face orientation identifier, the pre-trained decoder dynamically selects the corresponding mode to decode the image features encoded by the pre-trained encoder, outputting a target human face mask. The human face orientation identifier in this application indicates whether the target input image contains a human face and the direction of that human face. For example, if the target input image does not contain a human face, the human face orientation identifier is 0; if the target input image contains a human face and the human face is facing forward, the human face orientation identifier is 1; if the target input image contains a human face and the human face is facing right, the human face orientation identifier is 2; if the target input image contains a human face and the human face is facing downwards, the human face orientation identifier is 3; and if the target input image contains a human face and the human face is facing left, the human face orientation identifier is 4. The pre-trained human face state recognizer can be implemented by a classification network or a pose estimation network. For example, when a pre-trained human face state recognizer is implemented using a classification network, the classification network consists of a five-class classifier. The trained classification network can generate a human face orientation indicator based on the encoded image features to determine whether the target input image contains a human face and the direction of the human face.

[0029] The pre-trained decoder has a similar structure to the pre-trained encoder. It consists of a series of Nx and DSS modules (mode 1, mode 2, mode 3, mode 4), where Nx are upsampling modules, such as... Figure 4 As shown, the DSS module includes channel separation, layer normalization, multiple linear mapping layers, depthwise separable convolution, oriented state-space model (DSSM), and channel concatenation. To achieve dynamic selection of the pre-trained decoder, in the pre-trained decoder, as... Figure 6 As shown, Figure 6 This is a schematic block diagram of the DSSM in the DSS module of the pre-trained decoder in this embodiment of the application. The DSSM includes four unidirectional feature scanning models and corresponding first state space model, second state space model, third state space model, and fourth state space model (i.e., Figure 6(Modes 1, 2, 3, and 4). The direction of feature scanning directly affects the effect of the target portrait mask output by the pre-trained decoder. However, omnidirectional feature scanning like the pre-trained encoder introduces redundant computation. By obtaining the portrait orientation identifier of the target input image through the pre-trained portrait state recognizer, the pre-trained decoder dynamically selects the corresponding pre-trained decoder output target portrait mask directly based on the portrait orientation identifier during feature scanning, which can improve the efficiency and accuracy of portrait segmentation. When the feature scanning direction is sequentially scanned from the top of the image, the portrait segmentation effect tends to be better. Therefore, for... Figure 6 Mode 1, with feature scanning direction from left to right, corresponds to the frontal scene of the human figure in the target input image; for Figure 6 Mode 2, with the feature scanning direction from top to bottom, corresponds to a scene in the target input image where the person is facing right; for Figure 6 Mode 3, with the feature scanning direction from right to left, corresponds to a scene where the person in the target input image is facing downwards; for Figure 6 Mode 4, the feature scanning direction is from bottom to top, corresponding to a scene in the target input image where the human figure faces left.

[0030] The third target human face segmentation network in this application adopts a Recurrent Video Model (RVM) as its algorithm baseline, improves the feature extraction encoder of the neural network model (MobileNetv3), changes the design of the Direction State Space Recognition (DSS) module to effectively extract features when adapting to various human face orientations, and dynamically selects decoder branches of different modes in the decoder part.

[0031] It should be noted that the relationships and working principles between the downsampling module, upsampling module, channel separation, layer normalization, multiple linear mapping layers, depthwise separable convolution, directional state space model (DSSM), and channel splicing are existing technologies known to those skilled in the art, and will not be elaborated here.

[0032] This embodiment first acquires the target input image, encodes it using a pre-trained encoder, scans it from multiple directions to obtain image features, and then uses a pre-trained human face state recognizer to perform human face state recognition on the image features to obtain a human face orientation identifier. Based on this human face orientation identifier, it is determined whether there is a human face in the target input image. If there is no human face, a target human face mask with all zeros is output; if there is a human face, the corresponding target decoder is selected from multiple pre-trained decoders based on the human face orientation identifier to decode the image features and obtain the target human face mask. By setting a corresponding pre-trained decoder to be dedicated to outputting the human face mask for different human face orientations, and using the human face orientation identifier as prior information to dynamically select different pre-trained decoder branches to output the human face mask, the accuracy of human face segmentation can be improved when the human face orientation in the input image is unknown.

[0033] Please refer to Figure 7 This is a schematic flowchart of a portrait segmentation processing method provided in an embodiment of this application. The execution subject of this method can be... Figure 3 The third target human image segmentation network is shown. (See example...) Figure 7 As shown, the portrait segmentation processing method provided in this application embodiment includes, but is not limited to, steps S100 to S500: Step S100: Obtain the target input image; Step S200: Encode the target input image using a pre-trained encoder to obtain image features; Step S300: Perform image state recognition on image features using a pre-trained human state recognizer to obtain the human orientation identifier; Step S400: Select the corresponding target decoder from multiple pre-trained decoders according to the portrait orientation identifier, wherein the multiple pre-trained decoders are used to perform portrait segmentation on input images with different portrait orientations; Step S500: Decode the image features using the target decoder to obtain the target human image mask.

[0034] In steps S100 to S500 of this embodiment, the target input image is first acquired, and the target input image is encoded by a pre-trained encoder to obtain image features. Then, the image features are used to perform image state recognition by a pre-trained human face state recognizer to obtain a human face orientation identifier. Finally, the corresponding target decoder is selected from multiple pre-trained decoders to decode the image features based on the human face orientation identifier to obtain the target human face mask. By setting a corresponding pre-trained decoder to be dedicated to outputting the human face mask for different human face orientations, and using the human face orientation identifier as prior information to dynamically select different pre-trained decoder branches to output the human face mask, the accuracy of human face segmentation when the human face orientation in the input image is unknown can be improved.

[0035] It is understandable that the target input image includes a foreground image and a background image. The foreground image generally represents a human figure, while the target input image may also only contain a background image, i.e. a pure background image without a human figure.

[0036] In this embodiment, the pre-trained encoder includes a pre-trained feature scanning model and a pre-trained state space model connected in sequence. The pre-trained feature scanning model includes feature scanning in four directions: left to right, top to bottom, right to left, and bottom to top. The scanning information in each of the four directions is fed into the pre-trained state space model and modeled using formula (1) to obtain feature information in the four directions. The feature information in the four directions is then concatenated to obtain image features. Thus, this pre-trained encoder structure can adapt to feature extraction when the portrait is facing different directions.

[0037] The pre-trained human face state recognizer receives image features encoded by a pre-trained encoder from a target input image. Based on these encoded features, it generates a human face orientation identifier indicating whether the target input image contains a human face and the direction of that human face. The corresponding pre-trained decoder can be dynamically selected based on the human face orientation identifier to decode the image features encoded by the pre-trained encoder, outputting a target human face mask. The human face orientation identifier in this application indicates whether the target input image contains a human face and the direction of that human face. For example, if the target input image does not contain a human face, the human face orientation identifier is 0; if the target input image contains a human face and the human face is facing forward, the human face orientation identifier is 1; if the target input image contains a human face and the human face is facing right, the human face orientation identifier is 2; if the target input image contains a human face and the human face is facing downwards, the human face orientation identifier is 3; and if the target input image contains a human face and the human face is facing left, the human face orientation identifier is 4.

[0038] It should be noted that the specific value of the portrait orientation indicator can be changed, as long as it can distinguish the five portrait states, and should not be regarded as a limitation of this application.

[0039] The direction of feature scanning directly affects the effectiveness of the target portrait mask output by the pre-trained decoder. However, omnidirectional feature scanning by the pre-trained encoder introduces redundant computation. By obtaining the portrait orientation identifier of the target input image through a pre-trained portrait state recognizer, the pre-trained decoder can dynamically select the corresponding output target portrait mask based on the portrait orientation identifier during feature scanning, thus improving the efficiency and accuracy of portrait segmentation. Sequential feature scanning from the top of the image tends to yield better portrait segmentation results. Figure 6 Mode 1, with feature scanning direction from left to right, corresponds to the frontal scene of the human figure in the target input image; for Figure 6 Mode 2, with the feature scanning direction from top to bottom, corresponds to a scene in the target input image where the person is facing right; for Figure 6 Mode 3, with the feature scanning direction from right to left, corresponds to a scene where the person in the target input image is facing downwards; for Figure 6 Mode 4, the feature scanning direction is from bottom to top, corresponding to a scene in the target input image where the human figure faces left.

[0040] After step S500, the generated target portrait mask can be used to superimpose the foreground and background using formula (2) to achieve functions such as background replacement.

[0041] [0,1]Formula (2) in, For training images, Foreground image, Background image, This represents the probability that the pixel belongs to the foreground image.

[0042] Please refer to Figure 6 and Figure 8 , Figure 6 This is a schematic block diagram of the DSSM in the DSS module of the pre-trained decoder in this embodiment of the application. Figure 8 for Figure 7 The detailed flowchart of step S500 is shown below. In some embodiments, step S500 specifically includes, but is not limited to, performing the following steps S510 to S520 through the target decoder: Step S510: Scan the image features according to the scanning direction corresponding to the orientation mark of the human face to obtain the scan features; Step S520: Perform human image segmentation on the scanned features and output the target human image mask.

[0043] In some embodiments of this application, the human face orientation markers include a forward orientation marker, a right-facing marker, a downward orientation marker, and a left-facing marker, and the scanning features include one of the following: a forward scanning feature, a right-facing scanning feature, a downward scanning feature, and a left-facing scanning feature. Step S510 specifically includes, but is not limited to, the following steps: When the human face orientation indicator is positive, the image features are scanned sequentially from left to right to obtain the positive scan features; When the portrait orientation is marked as right, the image features are scanned sequentially from top to bottom to obtain the right-facing scan features. When the portrait orientation indicator is downward, the image features are scanned sequentially from right to left to obtain the downward scan features. When the orientation of the human figure is marked as "facing left", the image features are scanned sequentially from bottom to top to obtain the "facing left" scan features.

[0044] The direction of feature scanning directly affects the effectiveness of the target portrait mask output by the pre-trained decoder. However, omnidirectional feature scanning by the pre-trained encoder introduces redundant computation. By obtaining the portrait orientation identifier of the target input image through a pre-trained portrait state recognizer, the pre-trained decoder can dynamically select the corresponding output target portrait mask based on the portrait orientation identifier during feature scanning, thus improving the efficiency and accuracy of portrait segmentation. Sequential feature scanning from the top of the image tends to yield better portrait segmentation results. Figure 6 Mode 1, with feature scanning direction from left to right, corresponds to the frontal scene of the human figure in the target input image; for Figure 6 Mode 2, with the feature scanning direction from top to bottom, corresponds to a scene in the target input image where the person is facing right; for Figure 6Mode 3, with the feature scanning direction from right to left, corresponds to a scene where the person in the target input image is facing downwards; for Figure 6 Mode 4, the feature scanning direction is from bottom to top, corresponding to a scene in the target input image where the human figure faces left.

[0045] In some embodiments of this application, the human face orientation markers include a forward orientation marker, a right-facing marker, a downward orientation marker, and a left-facing marker, and the multiple pre-trained decoders are respectively a forward decoder, a right-facing decoder, a downward-facing decoder, and a left-facing decoder. Step S400 specifically includes, but is not limited to, the following steps: If the portrait orientation indicator is positive, select the positive decoder; If the portrait orientation indicator is set to "facing right", select the "facing right" decoder. If the portrait orientation indicator is "facing down", select the "facing down" decoder; If the portrait orientation indicator is set to "facing left", select the "facing left" decoder.

[0046] The human face orientation identifier in this application is used to indicate whether the target input image contains a human face and the orientation of the human face. For example, when the target input image does not contain a human face, the human face orientation identifier is 0; when the target input image contains a human face and the human face is facing forward, the human face orientation identifier is 1; when the target input image contains a human face and the human face is facing right, the human face orientation identifier is 2; when the target input image contains a human face and the human face is facing downward, the human face orientation identifier is 3; when the target input image contains a human face and the human face is facing left, the human face orientation identifier is 4. Based on the human face orientation identifier, it is determined whether there is a human face in the target input image. If there is no human face, a target human face mask with all zeros is output; if there is a human face, the corresponding target decoder is selected from multiple pre-trained decoders to decode the image features according to the human face orientation identifier to obtain the target human face mask. By setting up pre-trained decoders specifically responsible for outputting portrait masks for different portrait orientations, and using the portrait orientation identifier as prior information to dynamically select different pre-trained decoder branches to output portrait masks, the accuracy of portrait segmentation can be improved when the portrait orientation in the input image is unknown.

[0047] Please refer to Figure 9 ,for Figure 7 In some embodiments, the pre-trained encoder includes a pre-trained feature scanning model and a pre-trained state space model connected in sequence. Step S200 specifically includes, but is not limited to, steps S210 to S230: Step S210: Scan the target input image according to multiple different scanning directions using a pre-trained feature scanning model to obtain multiple corresponding scanning information; Step S220: Input multiple scan information into the pre-trained state space model respectively, and extract the corresponding multiple feature information; Step S230: Concatenate multiple feature information channels to obtain image features.

[0048] In one embodiment of this application, the pre-trained feature scanning model includes feature scanning in four directions: left to right, top to bottom, right to left, and bottom to top. The scanning information in each of the four directions is fed into the pre-trained state space model and modeled using formula (1) to obtain feature information in the four directions. The feature information in the four directions is then concatenated to obtain image features. Thus, this pre-trained encoder structure can adapt to feature extraction when the portrait is facing different directions.

[0049] Please refer to Figure 10 ,for Figure 9 In some embodiments, the specific process of step S210 includes, but is not limited to, steps S211 to S214: (The text abruptly ends here, so the translation stops as well.) Step S211: The target input image is sequentially scanned from left to right using a pre-trained feature scanning model to obtain forward scanning information; Step S212: The target input image is sequentially scanned from top to bottom using a pre-trained feature scanning model to obtain rightward scanning information; Step S213: The target input image is sequentially scanned from right to left using a pre-trained feature scanning model to obtain downward scanning information; Step S214: The target input image is sequentially scanned from bottom to top using a pre-trained feature scanning model to obtain leftward scanning information.

[0050] In some embodiments of this application, the image segmentation effect tends to be better when the feature scanning direction is sequentially scanned from the top of the image. Therefore, the feature scanning direction is set from left to right, corresponding to the frontal scene of the image in the target input image; the feature scanning direction is set from top to bottom, corresponding to the right-facing scene of the image in the target input image; the feature scanning direction is set from right to left, corresponding to the downward-facing scene of the image in the target input image; and the feature scanning direction is set from bottom to top, corresponding to the left-facing scene of the image in the target input image.

[0051] Please refer to Figure 11 This is a schematic diagram of the model training process for the pre-trained encoder and pre-trained decoder in the embodiments of this application. In some embodiments, the pre-trained encoder and multiple pre-trained decoders are obtained through the following steps S610 to S660: Step S610: Construct multiple training sample sets corresponding to different portrait orientations; Step S620: Construct the original human image segmentation network based on the original encoder and the original decoder; Step S630: Use all training sample sets to perform the first training on the original human image segmentation network to obtain the first target human image segmentation network; Step S640: Use the original encoder in the first target human image segmentation network as a pre-trained encoder; Step S650: For each portrait orientation, the first target portrait segmentation network is trained a second time using the training sample set corresponding to the portrait orientation to obtain the second target portrait segmentation network corresponding to the portrait orientation. During the second training process, the parameters of the pre-trained encoder in the first target portrait segmentation network are frozen, and the parameters of the original decoder in the first target portrait segmentation network are adjusted. Step S660: Use the original decoder in the second target human image segmentation network as the pre-trained decoder corresponding to the human image orientation.

[0052] In one embodiment of this application, the image segmentation training loss consists of the L1 loss function, the Laplacian loss function, and the time-series correlation loss function, which are respectively Equations (3), (4), and (5).

[0053] Formula (3) in, For L1 loss function, Representing the true GT, The alpha plot represents the prediction.

[0054] Formula (4) in, Let Laplace loss function be used. S This is a Gaussian pyramid level. Let L1 loss function be defined at a certain level of the Gaussian pyramid. Representing the true GT, The alpha graph represents the predicted alpha. The alpha graph is decomposed into 5 levels of Gaussian pyramids, and an L1 loss function is applied at each level in a weighted manner to supervise the local and global alpha output.

[0055] Formula (5) in, Let be the temporal correlation loss function, which represents the loss between the true ground truth (GT) and the predicted alpha map for consecutive t frames.

[0056] The embodiments of this application involve training the human image segmentation network in a multi-stage manner. First, multiple training sample sets corresponding to different human image orientations are constructed, including a forward training sample set, a right-facing training sample set, a down-facing training sample set, and a left-facing training sample set. Then, a pre-trained encoder and multiple pre-trained decoders are trained using the human image segmentation training loss method. The multiple pre-trained decoders are a forward decoder, a right-facing decoder, a down-facing decoder, and a left-facing decoder.

[0057] In some embodiments, a pre-trained encoder is first trained using a forward training sample set, a right-facing training sample set, a down-facing training sample set, and a left-facing training sample set. Then, the parameters of the pre-trained encoder are frozen, and the parameters of the original decoder in the first target image segmentation network are adjusted using the forward training sample set to train a forward decoder. This ensures that the pre-trained encoder and forward decoder are suitable for segmenting forward-facing images. Similarly, the parameters of the pre-trained encoder are frozen, and the parameters of the original decoder in the first target image segmentation network are adjusted using the right-facing training sample set to train a right-facing decoder. To make the pre-trained encoder and right-facing decoder suitable for segmenting portraits facing right; the parameters of the pre-trained encoder are frozen, and the parameters of the original decoder in the first target portrait segmentation network are adjusted using a downward training sample set to train a downward-facing decoder, so that the pre-trained encoder and downward-facing decoder are suitable for segmenting portraits facing downward; the parameters of the pre-trained encoder are frozen, and the parameters of the original decoder in the first target portrait segmentation network are adjusted using a left-facing training sample set to train a left-facing decoder, so that the pre-trained encoder and left-facing decoder are suitable for segmenting portraits facing left.

[0058] In this embodiment of the application, the parameters of the pre-trained encoder are frozen during the training of the pre-trained decoder, which can ensure the specificity of the data, so that each pre-trained decoder can be specifically responsible for the output of the portrait mask corresponding to the portrait orientation.

[0059] It should be noted that the principles and processes of human face segmentation training loss are existing technologies known to those skilled in the art, and will not be elaborated upon here.

[0060] Please refer to Figure 12 This is a schematic diagram illustrating the process of obtaining face mask labels in an embodiment of this application. In one embodiment of this application, the training sample set includes multiple training images and face mask labels corresponding to the training images. The face mask label corresponding to each training image is obtained through the following steps S611 to S614: Step S611: Combine the original portrait mask and background image to obtain the training image; Step S612: Identify and extract the object mask from the training image; Step S613: Determine the overlap area information between the object mask and the original human face mask; Step S614: Determine the face mask label based on the overlapping area information and the preset overlapping threshold.

[0061] In step S611 of this application embodiment, the foreground image (i.e. the original portrait mask) is usually separated from the background image, and a new training image is synthesized by superimposing the portrait mask according to formula (2).

[0062] In step S612 of this application embodiment, the identification and extraction of the item mask can be done by manual annotation or by generating it through a general segmentation network. The specific principles and processes are existing technologies known to those skilled in the art and will not be described in detail here.

[0063] In steps S613 and S614 of this application embodiment, when the user moves the camera and the background is in motion or in a complex background situation, objects in the background may be mistakenly classified as foreground. In order to improve the above defects, based on the training image, it is determined whether the object belongs to the foreground or the background by judging whether the overlap area information between the object mask and the original portrait mask is greater than a preset overlap threshold, and then it is determined whether to adjust the face mask label. This can reduce the occurrence of missegmentation in complex backgrounds or dynamically changing background situations and improve the accuracy of portrait segmentation.

[0064] In one embodiment of this application, step S614 specifically includes, but is not limited to, the following steps: When the overlap area information is greater than the overlap threshold, the object mask and the original portrait mask are merged to obtain the face mask label; If the information of the overlapping area is less than or equal to the overlap threshold, the original portrait mask is used as the portrait mask label.

[0065] The overlap threshold can be set to 2000 pixels. If the overlap area is greater than 2000 pixels, the simulated object is in front of the person and belongs to the foreground. The object mask is then merged with the original person mask to obtain the face mask label. If the overlap area is less than or equal to 2000 pixels, the simulated object is behind the person and belongs to the background. In this case, the original person mask remains unchanged and is used as the face mask label.

[0066] It should be noted that the overlap threshold can be set according to the actual situation, at least enough to simulate whether the object is in front of or behind the person. The specific values ​​given above should not be regarded as limitations on this application.

[0067] Please refer to Figure 13 This is a schematic diagram of the model training process of the pre-trained human face state recognizer in the embodiments of this application. In some embodiments, the pre-trained human face state recognizer is trained through the following steps S670 to S690: Step S670: Construct a corresponding human face state recognition network based on each second target human face segmentation network. Each human face state recognition network includes a pre-trained encoder, an original human face state recognizer, and a pre-trained decoder. Step S680: Use all training sample sets to perform a third training on each portrait state recognition network to obtain the corresponding third target portrait segmentation network. During the third training process, freeze the parameters of the pre-trained encoder and pre-trained decoder in the portrait state recognition network, and adjust the parameters of the original portrait state recognizer in the portrait state recognition network. Step S690: Obtain a pre-trained human state recognizer based on the third target human image segmentation network.

[0068] In one embodiment of this application, the parameters of the pre-trained encoder and pre-trained decoder are first frozen, and then a pre-trained human state recognizer is trained. The pre-trained human state recognizer can be associated with the pre-trained encoder and each pre-trained decoder, so that the human face orientation identifier can be used as prior information to dynamically select different pre-trained decoder branches to output human face masks.

[0069] The training sample set also includes a pure background training sample set with no human figures. The orientation of the human figures in the pure background training sample set is labeled as 0, the orientation of the human figures in the positive training sample set is labeled as 1, the orientation of the human figures in the right-facing training sample set is labeled as 2, the orientation of the human figures in the downward-facing training sample set is labeled as 3, and the orientation of the human figures in the left-facing training sample set is labeled as 4. The third training uses cross-entropy loss iterative training.

[0070] It should be noted that the principles and processes of cross-entropy loss iterative training are existing technologies known to those skilled in the art, and will not be elaborated upon here. Furthermore, the specific values ​​of the portrait orientation identifier can be changed, as long as it can distinguish the five portrait states, and should not be considered a limitation of this application.

[0071] This application embodiment improves the accuracy of portrait segmentation when the portrait orientation in the input image is unknown by setting a pre-trained decoder specifically responsible for outputting the portrait mask for the corresponding portrait orientation and using the portrait orientation identifier as prior information to dynamically select different pre-trained decoder branches to output the portrait mask.

[0072] This application also provides an electronic device, such as... Figure 14 As shown, the electronic device 1400 includes: One or more processors 1410; The memory 1420 stores one or more programs, which, when executed by one or more processors 1410, enable the one or more processors 1410 to implement the portrait segmentation processing method provided in any embodiment of this application.

[0073] Memory 1420, as a non-transitory network system, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 1420 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1420 may optionally include remotely located memories 1420 relative to processor 1410, which can be connected to processor 1410 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0074] The memory 1420 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1420 and is called and executed by the processor 1410.

[0075] The processor 1410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0076] In some embodiments, the electronic device further includes: Input / output interfaces are used to implement information input and output; The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus transmits information between various components of the device (e.g., processor 1410, memory 1420, input / output interfaces, and communication interfaces); The processor 1410, memory 1420, input / output interface, and communication interface can communicate with each other within the device via a bus.

[0077] One embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for executing the portrait segmentation processing method provided in any embodiment of this application.

[0078] An embodiment of this application also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the portrait segmentation processing method provided in any embodiment of this application.

[0079] The system architecture and application scenarios described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will understand that as system architectures evolve and new application scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems.

[0080] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0081] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0082] The above description, with reference to the accompanying drawings, illustrates some embodiments of this application, but does not limit the scope of the invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this invention should be considered within the scope of this application.

Claims

1. A method for human image segmentation, characterized in that, The method includes: Obtain the target input image; The target input image is encoded using a pre-trained encoder to obtain image features; The image features are analyzed by a pre-trained human state recognizer to identify the human state and obtain the human orientation identifier. The target decoder is selected from multiple pre-trained decoders based on the portrait orientation identifier, wherein the multiple pre-trained decoders are used to perform portrait segmentation on input images with different portrait orientations; The target decoder decodes the image features to obtain the target human image mask.

2. The method according to claim 1, characterized in that, Decoding the image features using the target decoder to obtain the target portrait mask includes performing the following processing using the target decoder: The image features are scanned according to the scanning direction corresponding to the orientation of the human face to obtain the scan features; The scanned features are used for human face segmentation, and the target human face mask is output.

3. The method according to claim 2, characterized in that, The portrait orientation markers include a forward orientation marker, a right orientation marker, a downward orientation marker, and a left orientation marker. The scanning features include one of the following: a forward scanning feature, a right scanning feature, a downward scanning feature, and a left scanning feature. The scanning of the image features according to the scanning direction corresponding to the portrait orientation markers to obtain the scanning features includes: When the portrait orientation identifier is the positive orientation identifier, the image features are sequentially scanned from left to right to obtain the positive scan features; When the portrait orientation identifier is the right-facing identifier, the image features are sequentially scanned from top to bottom to obtain the right-facing scan features; When the portrait orientation identifier is the downward identifier, the image features are sequentially scanned from right to left to obtain the downward scan features; When the orientation of the human figure is identified as the left-facing feature, the image features are scanned sequentially from bottom to top to obtain the left-facing scan features.

4. The method according to claim 1, characterized in that, The portrait orientation identifier includes a forward orientation identifier, a right orientation identifier, a downward orientation identifier, and a left orientation identifier. The multiple pre-trained decoders are respectively a forward decoder, a right-facing decoder, a downward-facing decoder, and a left-facing decoder. The step of selecting the corresponding target decoder from the multiple pre-trained decoders based on the portrait orientation identifier includes: If the portrait orientation identifier is the positive identifier, select the positive decoder; If the portrait orientation indicator is the right-facing indicator, select the right-facing decoder; If the portrait orientation indicator is the downward orientation indicator, select the downward orientation decoder; If the portrait orientation indicator is the left-facing indicator, select the left-facing decoder.

5. The method according to claim 1, characterized in that, The pre-trained encoder includes a pre-trained feature scanning model and a pre-trained state space model connected in sequence. Encoding the target input image using the pre-trained encoder to obtain image features includes: The pre-trained feature scanning model scans the target input image according to multiple different scanning directions to obtain corresponding multiple scanning information. The scanning information is input into the pre-trained state space model to extract the corresponding feature information. The image features are obtained by concatenating multiple feature information channels.

6. The method according to claim 5, characterized in that, The multiple scanning information includes forward scanning information, rightward scanning information, downward scanning information, and leftward scanning information. The pre-trained feature scanning model scans the target input image according to multiple different scanning directions to obtain the corresponding multiple scanning information, including: The target input image is sequentially scanned from left to right using the pre-trained feature scanning model to obtain the forward scanning information. The target input image is sequentially scanned from top to bottom using the pre-trained feature scanning model to obtain the rightward scanning information. The target input image is sequentially scanned from right to left using the pre-trained feature scanning model to obtain the downward scanning information. The target input image is scanned sequentially from bottom to top using the pre-trained feature scanning model to obtain the leftward scanning information.

7. The method according to claim 1, characterized in that, The pre-trained encoder and the plurality of pre-trained decoders are obtained through the following steps: Construct multiple training sample sets corresponding to different portrait orientations; An original human portrait segmentation network was constructed based on the original encoder and original decoder; The original human image segmentation network is trained using all of the training sample sets to obtain a first target human image segmentation network; The original encoder in the first target human image segmentation network is used as the pre-trained encoder; For each portrait orientation, the first target portrait segmentation network is trained a second time using the training sample set corresponding to the portrait orientation to obtain the second target portrait segmentation network corresponding to the portrait orientation. During the second training process, the parameters of the pre-trained encoder in the first target portrait segmentation network are frozen, and the parameters of the original decoder in the first target portrait segmentation network are adjusted. The original decoder in the second target portrait segmentation network is used as the pre-trained decoder corresponding to the portrait orientation.

8. The method according to claim 7, characterized in that, The training sample set includes multiple training images and corresponding face mask labels for the training images. The face mask label for each training image is obtained through the following steps: The training image is obtained by combining the original portrait mask and the background image; Identify and extract object masks from the training images; Determine the overlap area information between the object mask and the original human image mask; The face mask label is determined based on the overlapping area information and the preset overlapping threshold.

9. The method according to claim 8, characterized in that, Determining the face mask label based on the overlapping region information and a preset overlapping threshold includes: If the overlap area information is greater than the overlap threshold, the object mask and the original portrait mask are merged to obtain the face mask label; If the overlapping area information is less than or equal to the overlapping threshold, the original portrait mask is used as the portrait mask label.

10. The method according to claim 7, characterized in that, The pre-trained human face state recognition device is trained through the following steps: A corresponding human face state recognition network is constructed based on each of the second target human face segmentation networks. Each of the human face state recognition networks includes the pre-trained encoder, the original human face state recognizer, and the pre-trained decoder. The third training is performed on each of the human face state recognition networks using all the training sample sets to obtain the corresponding third target human face segmentation network. During the third training process, the parameters of the pre-trained encoder and the pre-trained decoder in the human face state recognition network are frozen, and the parameters of the original human face state recognizer in the human face state recognition network are adjusted. The pre-trained human state recognizer is obtained based on the third target human image segmentation network.

11. An electronic device, comprising: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the portrait segmentation processing method as described in any one of claims 1-10.

12. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement: the portrait segmentation processing method as described in any one of claims 1-10.

13. A computer program product comprising a computer program that, when executed by a processor, implements the portrait segmentation processing method as described in any one of claims 1-10.

Images