Skin image segmentation method and device, live broadcast system, equipment and medium

Abstract

The invention relates to a skin image segmentation method and device, a live broadcast system, electronic equipment and a computer readable storage medium. The method comprises the following steps: applying affine transformation simulation to an input image trained by a skin segmentation model to generate a video sequence frame; deriving a pixel-level displacement flow field of an input image converted to a video sequence frame according to the affine transformation process; in a skin segmentation model training process, predicting a corresponding first skin segmentation mask and a corresponding second skin segmentation mask according to the input image and the video sequence frame; acquiring a mask time sequence consistency constraint loss function of skin segmentation according to the displacement flow field, a first skin segmentation mask and a second skin segmentation mask; performing supervised training on the skin segmentation model by adopting the mask time sequence consistency constraint loss function of the skin segmentation; according to the technical scheme, the inter-frame consistency of the skin segmentation model in a video segmentation scene and the robustness of coping with images with different input resolutions are improved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and in particular to a skin image segmentation method, apparatus, live streaming system, electronic device and computer-readable storage medium. Background Technology

[0002] With the development of internet technology, live streaming has been adopted by most internet users. Live streaming, with its intuitive, real-time, and interactive content and format, has played an important role in promoting flexible employment, economic and social development, and enriching the spiritual and cultural life of the people. Streamers can better showcase their talents and abilities in live streaming, thus realizing self-worth for more streamers.

[0003] In live video streaming scenarios, computer vision and image processing technologies are extensively used. Skin image segmentation is a crucial fundamental task, widely applied in skin whitening, smoothing, and other effects. Traditional skin segmentation methods, based on single-frame image segmentation, suffer from jitter and flickering due to the lack of inter-frame modeling and consistency constraints, thus affecting the stability of skin whitening, smoothing, and other special effects. Summary of the Invention

[0004] Therefore, it is necessary to provide a skin image segmentation method, apparatus, live streaming system, electronic device, and computer-readable storage medium, which can improve the inter-frame consistency of the skin segmentation model in video segmentation scenarios and its robustness in dealing with images of different input resolutions.

[0005] A skin image segmentation method, comprising: An affine transformation is applied to the input image used for training the skin segmentation model to simulate the generation of video sequence frames; The pixel-level displacement flow field of the input image transformed into a video sequence frame is derived based on the affine transformation process. During the training of the skin segmentation model, the corresponding first skin segmentation mask and second skin segmentation mask are predicted based on the input image and video sequence frames. The mask temporal consistency constraint loss function for skin segmentation is obtained based on the displacement flow field, the first skin segmentation mask, and the second skin segmentation mask. The skin segmentation model is trained under supervision using the mask temporal consistency constraint loss function for skin segmentation.

[0006] In some embodiments, applying an affine transformation to the input image used for training the skin segmentation model simulates the generation of a video sequence frame, including:

[0007] in, For the input image, For the applied affine transformation, For video sequence frames, These are the pixel coordinates; The input image is a pixel-level affine transformation. A video sequence of frames that have undergone affine transformation of pixels; where For the parameters of the affine transformation, Used to simulate image rotation, scaling, and shearing transformations. Used to simulate translation transformations of images.

[0008] In some embodiments, deriving the pixel-level displacement flow field of the input image transformed into a video sequence frame based on the affine transformation process includes:

[0009] in, Represents the displacement flow field. This represents the displacement flow field at the pixel level.

[0010] In some embodiments, the mask temporal consistency constraint loss function for skin segmentation is obtained based on the displacement flow field and the first and second skin segmentation masks, including:

[0011] in, For the input image The corresponding first skin segmentation mask, The second skin segmentation mask is used to predict the corresponding frames in the video sequence. Let be the mask temporal consistency constraint loss function for skin segmentation. This is a differentiable warping operation based on the flow field.

[0012] In some embodiments, the skin image segmentation method further includes: Construct flow fields at different scale levels, and generate multi-scale flow fields by Gaussian pyramid downsampling of the displacement flow fields; Based on the multi-scale flow field, the skin segmentation model is trained under supervision at each scale using segmentation mask temporal consistency constraints, and the mask temporal consistency loss function is calculated for each scale. The temporal consistency loss functions of each mask are fused to obtain the total loss function of the temporal consistency constraint of the multi-scale pyramid, which is used for supervised training of the skin segmentation model.

[0013] In some embodiments, constructing flow fields at different scale levels and performing Gaussian pyramid downsampling on the displacement flow field to generate a multi-scale flow field includes:

[0014] in, It is a pyramid-scale hierarchy. For Gaussian kernel, For the first The Gaussian kernel standard deviation of the hierarchy is used to control the degree of fuzziness; This indicates that the actual pixel distance is calculated based on the image.

[0015] In some embodiments, the skin segmentation model is trained under supervised training at each scale using segmentation mask temporal consistency constraints based on the multi-scale flow field, and a mask temporal consistency loss function is calculated for each scale, including:

[0016] in, This indicates that the mask temporal consistency loss function is calculated at each scale. Indicates hierarchy.

[0017] In some embodiments, the temporal consistency loss functions of each mask are fused to obtain the total loss function for the temporal consistency constraints of the multi-scale pyramid, including:

[0018]

[0019]

[0020] in, This represents the loss function based on the temporal consistency constraint of a multi-scale pyramid. This represents the edge gradient loss function at the maximum resolution scale. The total loss function represents the temporal consistency constraint of the multi-scale pyramid.

[0021] A skin image segmentation apparatus, comprising: The affine transformation module is used to apply affine transformations to the input images for training the skin segmentation model to simulate the generation of video sequence frames; The flow field derivation module is used to derive the pixel-level displacement flow field of the input image transformed into a video sequence frame based on the affine transformation process. The mask prediction module is used to predict the corresponding first skin segmentation mask and second skin segmentation mask based on the input image and video sequence frames during the skin segmentation model training process. The loss function module is used to obtain the mask temporal consistency constraint loss function for skin segmentation based on the displacement flow field and the first and second skin segmentation masks. The supervised training module is used to supervise the training of the skin segmentation model using the mask temporal consistency constraint loss function of the skin segmentation.

[0022] A live streaming system includes: a broadcaster's terminal, a viewer's terminal, and a live streaming server; wherein the broadcaster's terminal and the viewer's terminal are respectively connected to the live streaming server via a communication network; The broadcaster's client is used to connect to the broadcaster in the live broadcast room and to collect the broadcaster's live video stream and upload it to the live broadcast server. The live streaming server is used to forward live streams between the broadcaster and the viewer and to send live video to the viewer; it obtains the broadcaster's face image from the broadcaster's live video stream and performs skin segmentation on the broadcaster's face image using the skin image segmentation method described in any one of claims 1-8. The viewer terminal is used to connect to the live broadcast room and to receive and play the live video.

[0023] An electronic device comprising: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the steps of the skin image segmentation method.

[0024] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded by a processor and the steps of the skin image segmentation method are executed.

[0025] The technical solution of this application applies an affine transformation to the input image used for training the skin segmentation model to simulate the generation of video sequence frames, derives the pixel-level displacement flow field of the input image transformed into the video sequence frames, and predicts the first and second skin segmentation masks corresponding to the input image and video sequence frames during the skin segmentation model training process. Then, based on the displacement flow field and the first and second skin segmentation masks, the temporal consistency constraint loss function of the skin segmentation mask is obtained, and the skin segmentation model is trained under supervision. This technical solution, for beauty-related effects in live streaming scenarios that rely on skin segmentation results for rendering, is based on geometric transformation-driven temporal consistency constraints for skin segmentation masks. By introducing multi-scale temporal consistency constraints, the inter-frame segmentation results of the skin segmentation model are made stable and consistent, alleviating the problem of jitter in inter-frame segmentation results, improving the inter-frame consistency of the skin segmentation model in video segmentation scenarios and its robustness to different input resolution images, avoiding problems such as jitter in beauty effects such as skin whitening and skin smoothing, and improving the user experience of related functions. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating an example of a live streaming application scenario. Figure 2 This is a flowchart of a skin image segmentation method according to one embodiment; Figure 3 This is a schematic diagram of the processing flow during the training phase of an example skin segmentation model. Figure 4 This is a flowchart illustrating the inference phase processing of an example skin segmentation model. Figure 5 This is a schematic diagram of a skin image segmentation device according to one embodiment; Figure 6 This is a schematic diagram of an example live streaming system structure; Figure 7 This is a block diagram of an example electronic device. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. "At least one" refers to one or more, and "multiple" means two or more; for example, multiple objects refer to two or more objects. Words such as "including" or "containing" indicate that the information preceding "including" or "containing" covers the information listed after "including" or "containing" and its equivalents, but does not exclude other information. In the embodiments of this application, "and / or" indicates that three relationships can exist; the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. "At least one" refers to one or more, and "multiple" means two or more; for example, multiple objects refer to two or more objects. Words such as "including" or "containing" indicate that the information preceding "including" or "containing" covers the information listed after "including" or "containing" and its equivalents, but does not exclude other information. In the embodiments of this application, "and / or" indicates that three relationships can exist; the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0031] The technical solutions provided in the embodiments of this application can be applied to, for example... Figure 1 The application scenarios of the methods related to this application shown can be applied to, for example... Figure 1 In the application scenarios shown, Figure 1 This is a schematic diagram illustrating an example of a live streaming application scenario. The live streaming system can include a live streaming server, a broadcaster's client, and a viewer's client. The broadcaster's client and the viewer's client communicate with the live streaming server via a communication network, enabling real-time online streaming for both. The terminal devices for the broadcaster's client and the viewer's client can be, but are not limited to, various personal computers, laptops, smartphones, and tablets. The live streaming server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0032] The technical solution of this application proposes a temporal loss function that applies consistency constraints to the inter-frame segmentation results of video based on geometric transformation. In order to further improve the robustness of the model for video segmentation of images with different input scales, the temporal loss function is extended to a multi-scale pyramid consistency constraint. This improves the inter-frame consistency of the skin segmentation model in video segmentation scenarios and its robustness to images with different input resolutions.

[0033] refer to Figure 2 As shown, Figure 2 This is a flowchart of a skin image segmentation method according to an embodiment, including the following steps: S10 applies an affine transformation to the input image used for training the skin segmentation model to simulate the generation of video sequence frames.

[0034] This step belongs to the training phase of the skin segmentation model, refer to... Figure 3 As shown, Figure 3 This is a schematic diagram of the training phase of an example skin segmentation model, based on geometric transformation-driven temporal consistency constraints for skin segmentation masks, for the input image. Affine transformation can be applied Simulate the generation of video sequence frames .

[0035] In some embodiments, an affine transformation is applied to the input image used for training the skin segmentation model to simulate the generation of video sequence frames. The affine transformation formula can be as follows:

[0036] in, For the input image, For the applied affine transformation, For video sequence frames, These are the pixel coordinates; The input image is a pixel-level affine transformation. A video sequence of frames that have undergone affine transformation of pixels; where For the parameters of the affine transformation, Used to simulate image rotation, scaling, and shearing transformations. Used to simulate translation transformations of images.

[0037] S20, based on the affine transformation process, derive the pixel-level displacement flow field of the input image transformed into a video sequence frame.

[0038] In this step, the image can be derived based on the above affine transformation process. Transform to pixel-level displacement flow field .

[0039] In some embodiments, the pixel-level displacement flow field of the input image transformed into a video sequence frame is derived based on the affine transformation process, and the calculation formula can be as follows:

[0040] in, Represents the displacement flow field. This represents the displacement flow field at the pixel level.

[0041] S30, during the skin segmentation model training process, predict the corresponding first skin segmentation mask and second skin segmentation mask based on the input image and video sequence frames.

[0042] In this step, during the training of the skin segmentation model, the input image... and The corresponding first skin segmentation mask is predicted. Second skin segmentation mask .

[0043] S40, obtain the mask temporal consistency constraint loss function for skin segmentation based on the displacement flow field and the first and second skin segmentation masks.

[0044] In this step, based on the aforementioned pixel-level displacement field This allows us to obtain the skin segmentation mask temporal consistency constraint loss function for the model during training, which can then be used for supervised training.

[0045] In some embodiments, the temporal consistency constraint loss function for skin segmentation is obtained based on the displacement flow field and the first and second skin segmentation masks. The Dice distance can be used to measure the distribution difference between the two masks. Therefore, the temporal consistency constraint loss function for skin segmentation may include the following:

[0046] in, For the input image The corresponding first skin segmentation mask, The second skin segmentation mask is used to predict the corresponding frames in the video sequence. Let be the mask temporal consistency constraint loss function for skin segmentation. This is a differentiable warping operation based on the flow field.

[0047] S50, the skin segmentation model is trained under supervision using the mask temporal consistency constraint loss function of the skin segmentation.

[0048] The technical solutions described in the above embodiments can be applied to the training phase of a skin segmentation model to improve the consistency between skin segmentation frames. Figure 4 As shown, Figure 4 This is an example of a processing flowchart for the inference stage of a skin segmentation model. By using geometric transformation-driven temporal consistency constraints on skin segmentation masks during the model training stage, the inference stage can use the skin segmentation model optimized by this method to predict stable and consistent skin region information, which can then be applied to downstream beautification operations such as skin smoothing and whitening.

[0049] To make the technical solution of this application clearer, more embodiments are described below.

[0050] To further improve the robustness of the skin segmentation model to images with different input resolution scales and to further improve the aforementioned temporal consistency constraints, it can be extended to a multi-scale pyramid temporal consistency constraint.

[0051] In some embodiments, the skin image segmentation method of this application further includes: (1) Construct flow fields of different scales and perform Gaussian pyramid downsampling on the displacement flow field to generate multi-scale flow fields.

[0052] Specifically, firstly, flow fields at different scale levels are constructed, and the original flow field is compared with the flow field. Gaussian pyramid downsampling is performed to generate a multi-scale flow field. For example, the calculation process may include the following formula:

[0053] in, For pyramid-scale levels, for example, This is the original image. (For 1 / 2 resolution, and so on). For Gaussian kernel, For the first The Gaussian kernel standard deviation of the hierarchy is used to control the degree of fuzziness; This indicates the calculation of actual pixel distances based on the image. Typically, for low-resolution scales (such as...), A larger standard deviation is needed. To cover pixels in the same physical area.

[0054] (2) Based on the multi-scale flow field, the skin segmentation model is trained under supervision at each scale using segmentation mask temporal consistency constraints, and the mask temporal consistency loss function is calculated for each scale.

[0055] In this step, based on the multi-scale flow field obtained above, the model can be trained under the supervision of segmented mask temporal consistency constraints at each scale, and the mask temporal consistency loss function is calculated for each scale.

[0056] In some embodiments, the formula for calculating the mask timing consistency loss function may include the following:

[0057] in, This indicates that the mask temporal consistency loss function is calculated at each scale. Indicates hierarchy.

[0058] (3) The temporal consistency loss functions of each mask are fused to obtain the total loss function of the temporal consistency constraint of the multi-scale pyramid, which is used to supervise the training of the skin segmentation model.

[0059] In this step, the mask temporal consistency loss function calculated at each scale is fused to obtain the following temporal consistency constraint based on a multi-scale pyramid.

[0060] For example, the loss function for the temporal consistency constraint of a multi-scale pyramid can be expressed as follows:

[0061] For example, to improve the consistency of edge segmentation in a skin segmentation model, the following edge gradient loss function can be used at the maximum scale resolution, which can be expressed as follows:

[0062] For example, the total loss function based on the temporal consistency constraint of the multi-scale pyramid can be expressed as follows:

[0063] in, and The weights corresponding to different loss function terms, This represents the loss function based on the temporal consistency constraint of a multi-scale pyramid. This represents the edge gradient loss function at the maximum resolution scale. The total loss function represents the temporal consistency constraint of the multi-scale pyramid.

[0064] The skin image segmentation method described in the above embodiments can be applied to the training phase of the skin segmentation model to improve the consistency between skin segmentation frames. In the inference phase, the skin segmentation model optimized by this method can predict stable and consistent skin region information and apply it to downstream beautification operations such as skin smoothing and whitening. Since the multi-scale temporal consistency constraint method improves the stability of the skin segmentation model's segmentation results between different frames of the video, the introduction of multi-scale temporal consistency constraints makes the inter-frame segmentation results of the skin segmentation model stable and consistent, alleviates the jitter problem of inter-frame segmentation results, and avoids the problem of beauty effect jitter in beautification effects such as skin whitening and skin smoothing. This improves the accuracy and stability of related beauty effects and greatly enhances the user experience of using related beauty effects.

[0065] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0066] Based on the same inventive concept, this application also provides an apparatus for implementing the related methods described above. The solution provided by this apparatus is similar to the solution described in the above methods. Therefore, the specific limitations in one or more related apparatus embodiments provided below can be found in the limitations of the related methods above, and will not be repeated here.

[0067] refer to Figure 5 As shown, Figure 5 This is a schematic diagram of a skin image segmentation device according to an embodiment. The device includes: Affine transformation module 10 is used to apply affine transformation to the input image for training the skin segmentation model to simulate the generation of video sequence frames; The flow field derivation module 20 is used to derive the pixel-level displacement flow field of the input image transformed into a video sequence frame based on the affine transformation process. The mask prediction module 30 is used to predict the corresponding first skin segmentation mask and second skin segmentation mask based on the input image and video sequence frames during the skin segmentation model training process. Loss function module 40 is used to obtain the mask temporal consistency constraint loss function for skin segmentation based on the displacement flow field and the first skin segmentation mask and the second skin segmentation mask; The supervised training module 50 is used to supervise the training of the skin segmentation model using the mask temporal consistency constraint loss function of the skin segmentation.

[0068] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0069] The skin image segmentation device of this embodiment can execute a skin image segmentation method provided in the embodiments of this application. The implementation principle is similar. The actions performed by each module in the skin image segmentation device in each embodiment of this application correspond to the steps in the skin image segmentation method in each embodiment of this application. For detailed functional descriptions of each module of the skin image segmentation device, please refer to the descriptions in the corresponding skin image segmentation methods shown above. They will not be repeated here.

[0070] The following describes an implementation example of a live streaming system.

[0071] The live streaming system provided in this embodiment is based on... Figure 6 As shown, Figure 6 This is a schematic diagram of an example live streaming system structure. The system includes: a broadcaster's client, a viewer's client, and a live streaming server. The broadcaster's client and the viewer's client are connected to the live streaming server via a communication network. The broadcaster's client is used to connect to the live streaming room and collect the broadcaster's live video stream, uploading it to the live streaming server. The live streaming server is used to forward the live stream between the broadcaster's client and the viewer's client and to send the live video to the viewer's client. The system also includes obtaining the broadcaster's facial image from the live video stream and performing skin segmentation on the broadcaster's facial image using the skin image segmentation method provided in this application embodiment. The viewer's client is used to connect to the live streaming room and receive and play the live video.

[0072] For example, the broadcaster's client and the viewer's client are connected to the live streaming server via a communication network. The viewer's client may include multiple viewer users' clients, such as viewer user 1, viewer user 2, ..., viewer user n, etc. The broadcaster's client can access the broadcaster's client in the live streaming room and capture the broadcaster's live video stream and upload it to the live streaming server. The live streaming server can forward the live stream between the broadcaster's client and the viewer's client and send the live video to the viewer's client. It can also obtain the broadcaster's live video from the broadcaster's live video stream for live streaming. The viewer's client can access the viewer's client in the live streaming room and receive the live video for playback.

[0073] As described in the above embodiment of the live streaming system, for beauty-related effects in live streaming scenarios that rely on skin segmentation results for rendering, based on geometric transformation-driven skin segmentation mask temporal consistency constraints, multi-scale temporal consistency constraints are introduced to make the inter-frame segmentation results of the skin segmentation model stable and consistent, alleviate the problem of inter-frame segmentation result jitter, improve the inter-frame consistency of the skin segmentation model in video segmentation scenarios and the robustness to different input resolution images, avoid the problem of beauty effect jitter in beauty effects such as skin whitening and skin smoothing, and improve the user experience of using related functions.

[0074] The live streaming system of this embodiment can execute a skin image segmentation method provided in the embodiments of this application. For more detailed functional descriptions of the live streaming system, please refer to the description of the corresponding skin image segmentation method shown above, which will not be repeated here.

[0075] The following describes embodiments of the electronic device and computer-readable storage medium of this application.

[0076] refer to Figure 7 As shown, Figure 7 This is a schematic diagram of an example electronic device, which may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. The computer device 100 may include one or more of the following components: processing component 102, memory 104, power supply component 106, multimedia component 108, audio component 110, input / output (I / O) interface 112, sensor component 114, and communication component 116.

[0077] Processing component 102 typically controls the overall operation of computer device 100, such as operations associated with display, telephone calls, data communication, camera operation, and recording operation.

[0078] The memory 104 is configured to store various types of data to support the operation of the computer device 100. Such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0079] The power supply unit 106 provides power to the various components of the computer device 100.

[0080] Multimedia component 108 includes a screen that provides an output interface between computer device 100 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). In some embodiments, multimedia component 108 includes a front-facing camera and / or a rear-facing camera.

[0081] The audio component 110 is configured to output and / or input audio signals.

[0082] I / O interface 112 provides an interface between processing component 102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0083] Sensor assembly 114 includes one or more sensors for providing various aspects of state assessment for computer device 100. Sensor assembly 114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact.

[0084] The communication component 116 is configured to facilitate wired or wireless communication between the computer device 100 and other devices. The computer device 100 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof.

[0085] Those skilled in the art will understand that the electronic device structure provided in the above embodiments is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the electronic device to which the solution of this application is applied. A specific electronic device may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0086] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in the above embodiments. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases, etc. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., etc., and are not limited to these.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0088] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A skin image segmentation method, characterized in that, include: An affine transformation is applied to the input image used for training the skin segmentation model to simulate the generation of video sequence frames; The pixel-level displacement flow field of the input image transformed into a video sequence frame is derived based on the affine transformation process. During the training of the skin segmentation model, the corresponding first skin segmentation mask and second skin segmentation mask are predicted based on the input image and video sequence frames. The mask temporal consistency constraint loss function for skin segmentation is obtained based on the displacement flow field, the first skin segmentation mask, and the second skin segmentation mask. The skin segmentation model is trained under supervision using the mask temporal consistency constraint loss function for skin segmentation.

2. The skin image segmentation method according to claim 1, characterized in that, Applying an affine transformation to the input image used for training the skin segmentation model to simulate the generation of a video sequence of frames includes: ； in, For the input image, For the applied affine transformation, For video sequence frames, These are the pixel coordinates; The input image is a pixel-level affine transformation. A video sequence of frames that are affine transformations of pixels; where For the parameters of the affine transformation, Used to simulate image rotation, scaling, and shearing transformations. Used to simulate translation transformations of images.

3. The skin image segmentation method according to claim 2, characterized in that, The pixel-level displacement flow field of the input image transformed into a video sequence frame is derived based on the affine transformation process, including: ； in, Represents the displacement flow field. This represents the displacement flow field at the pixel level.

4. The skin image segmentation method according to claim 3, characterized in that, The temporal consistency constraint loss function for skin segmentation is obtained based on the displacement flow field, the first skin segmentation mask, and the second skin segmentation mask, including: ； in, For the input image The corresponding first skin segmentation mask, The second skin segmentation mask is used to predict the corresponding frames in the video sequence. Let be the mask temporal consistency constraint loss function for skin segmentation. This is a differentiable warping operation based on the flow field.

5. The skin image segmentation method according to any one of claims 1 to 4, characterized in that, Also includes: Construct flow fields at different scale levels, and generate multi-scale flow fields by Gaussian pyramid downsampling of the displacement flow fields; Based on the multi-scale flow field, the skin segmentation model is trained under supervision at each scale using segmentation mask temporal consistency constraints, and the mask temporal consistency loss function is calculated for each scale. The temporal consistency loss functions of each mask are fused to obtain the total loss function of the temporal consistency constraint of the multi-scale pyramid, which is used for supervised training of the skin segmentation model.

6. The skin image segmentation method according to claim 5, characterized in that, Constructing flow fields at different scale levels, and generating multi-scale flow fields by Gaussian pyramid downsampling of the displacement flow field, including: ； in, It is a pyramid-scale hierarchy. For Gaussian kernel, For the first The Gaussian kernel standard deviation of the hierarchy is used to control the degree of fuzziness; This indicates that the actual pixel distance is calculated based on the image.

7. The skin image segmentation method according to claim 6, characterized in that, Based on the multi-scale flow field, the skin segmentation model is trained under supervised training at each scale using segmentation mask temporal consistency constraints, and the mask temporal consistency loss function is calculated for each scale, including: ； in, This indicates that the mask temporal consistency loss function is calculated at each scale. Indicates hierarchy.

8. The skin image segmentation method according to claim 6, characterized in that, The total loss function for the temporal consistency constraints of the multi-scale pyramid is obtained by fusing the temporal consistency loss functions of each mask, including: ； ； ； in, This represents the loss function based on the temporal consistency constraint of a multi-scale pyramid. This represents the edge gradient loss function at the maximum resolution scale. The total loss function represents the temporal consistency constraint of the multi-scale pyramid.

9. A skin image segmentation device, characterized in that, include: The affine transformation module is used to apply affine transformations to the input images for training the skin segmentation model to simulate the generation of video sequence frames; The flow field derivation module is used to derive the pixel-level displacement flow field of the input image transformed into a video sequence frame based on the affine transformation process. The mask prediction module is used to predict the corresponding first skin segmentation mask and second skin segmentation mask based on the input image and video sequence frames during the skin segmentation model training process. The loss function module is used to obtain the mask temporal consistency constraint loss function for skin segmentation based on the displacement flow field and the first and second skin segmentation masks. The supervised training module is used to supervise the training of the skin segmentation model using the mask temporal consistency constraint loss function of the skin segmentation.

10. A live streaming system, characterized in that, include: The system includes a broadcaster's client, a viewer's client, and a live streaming server; wherein the broadcaster's client and the viewer's client are respectively connected to the live streaming server via a communication network. The broadcaster's client is used to connect to the broadcaster in the live broadcast room and to collect the broadcaster's live video stream and upload it to the live broadcast server. The live streaming server is used to forward live streams between the broadcaster and the viewer and to send live video to the viewer; it obtains the broadcaster's face image from the broadcaster's live video stream and performs skin segmentation on the broadcaster's face image using the skin image segmentation method described in any one of claims 1-8. The viewer terminal is used to connect to the live broadcast room and to receive and play the live video.

11. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the steps of the skin image segmentation method according to any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded by a processor and the steps of the skin image segmentation method according to any one of claims 1-8 are executed.

Images