Human body image segmentation method and device, computer device and readable storage medium

Abstract

The application discloses a human body image segmentation method and device, computer equipment and a readable storage medium, relates to the technical field of image processing, and simultaneously predicts human body key points and human body mask images by using a model, improves the prediction effect and accuracy by means of the connection of tasks, simultaneously reduces the cost, and saves the computing power. The method comprises the following steps: acquiring a sample video, labeling human body key points and human body mask image sequences by using the sample video, and training an image segmentation model; in response to an image segmentation request, determining a video to be processed, collecting a video image frame of the video to be processed at a current time; calculating a background mask and a background image of the video image frame, splicing the video image frame, the background mask and the background image in a channel dimension to obtain an image splicing result; inputting the image splicing result into the image segmentation model for segmentation processing, obtaining a current frame human body mask image and current frame human body key point information of the video image frame, and outputting the human body image information obtained by segmentation as human body image information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, computer device, and readable storage medium for human image segmentation. Background Technology

[0002] In recent years, image processing technology has developed rapidly, and camera-based applications such as video conferencing and video games have become increasingly widespread. In these application scenarios, users are generally in relatively static environments, such as indoors. Therefore, some users, for privacy reasons or to better reflect their personal interests, may want to replace the background or even simply transfer their body movements to a virtual character within a virtual scene. Consequently, many video-related functions in applications automatically identify and locate key parts of the human body in video images, segment the human image, remove the original background, and replace it with a user-selected background of a fixed style, color, or special effects to meet their needs.

[0003] In related technologies, to achieve human image segmentation, two models need to be established beforehand: a human keypoint localization model and a human image segmentation model. The human keypoint localization model is trained based on a keypoint localization algorithm, which has three main input dimensions: single-frame input, fixed multi-frame input, and full-sequence input. In terms of output dimensions, it has two main output dimensions: two-dimensional keypoints and three-dimensional keypoints. The human image segmentation model is trained based on a human image segmentation algorithm.

[0004] In the process of developing this application, the applicant discovered that the relevant technology has at least the following problems:

[0005] The human keypoint localization model and the human image segmentation model perform different sub-tasks, which have a certain degree of similarity. However, the two different models decompose the entire human image segmentation process into two parts, and the similarity between the sub-tasks is not utilized in the entire image segmentation process. This not only leads to poor model performance and low accuracy, but also causes a certain degree of waste of computing power. Summary of the Invention

[0006] In view of this, this application provides a human image segmentation method, apparatus, computer device and readable storage medium, the main purpose of which is to solve the problems of poor performance of current models, low accuracy of execution results and a certain degree of waste of computing power.

[0007] According to a first aspect of this application, a method for human body image segmentation is provided, the method comprising:

[0008] Acquire sample videos, and use the human body key points annotated on each sample video image frame in the sample video and the human body masking image sequence corresponding to the sample video to train an image segmentation model;

[0009] In response to an image segmentation request, determine the video to be processed indicated by the image segmentation request, and acquire video image frames of the video to be processed at the current moment;

[0010] Calculate the background mask and background image of the video image frame, and then stitch the video image frame, the background mask, and the background image together in the channel dimension to obtain the image stitching result;

[0011] The image stitching result is input into the image segmentation model for segmentation processing to obtain the current frame human body mask image and the current frame human body key point information of the video image frame. The current frame human body mask image and the current frame human body key point information are then output as the segmented human body image information.

[0012] Optionally, the step of acquiring sample videos, using the human body key points annotated on each sample video image frame in the sample video and the corresponding human body masking image sequence, to train an image segmentation model, includes:

[0013] The sample video is acquired, and the sample video is subjected to video frame extraction processing to obtain a sample video image sequence of the sample video consisting of multiple sample video image frames;

[0014] Each sample video image frame in the sample video image sequence is labeled, and human body key points are labeled to each sample video image frame to obtain the human body masking image sequence.

[0015] Using the human body masking image sequence, calculate the sample background mask and sample background image of the sample video;

[0016] The sample video image sequence, the sample background image, and the sample background mask are stitched together in the channel dimension to obtain the sample stitching result;

[0017] The image segmentation model is constructed by encoding and decoding the sample splicing results based on a preset neural network algorithm.

[0018] Optionally, the step of annotating each sample video image frame included in the sample video image sequence, annotating human key points to each sample video image frame, and obtaining the human mask image sequence includes:

[0019] Human body key point annotation is performed on each of the multiple sample video image frames;

[0020] Binary human body mask annotation is performed on each sample video image frame to obtain the sample mask image corresponding to each sample video image frame;

[0021] Multiple sample mask images corresponding to the multiple sample video image frames are obtained, and the multiple sample mask images are arranged in chronological order to obtain the human body mask image sequence corresponding to the sample video.

[0022] Optionally, calculating the sample background mask and sample background image of the sample video using the human body mask image sequence includes:

[0023] For each of the multiple sample masking images, a preset coefficient is determined, and the difference between the preset coefficient and the value of each pixel position in the sample masking image is calculated to obtain the set of masking pixel differences corresponding to the sample masking image.

[0024] Obtain multiple sets of masking pixel differences corresponding to the multiple sample masking images, and superimpose the differences at the same pixel position in the multiple sets of masking pixel differences to obtain an initial masking image;

[0025] A preset quality threshold is determined, and the initial mask image is subjected to threshold processing using the preset quality threshold to obtain the sample background mask. The preset quality threshold is a threshold set according to the annotation quality.

[0026] The sample background image is calculated based on the multiple sample video images, the multiple mask pixel difference sets, the initial mask image, and the sample background mask.

[0027] Optionally, calculating the sample background image based on the plurality of sample video images, the plurality of mask pixel difference sets, the initial mask image, and the sample background mask includes:

[0028] For each of the plurality of mask pixel difference sets, the following processing is performed: a target sample video image is determined in the plurality of sample video images, and the mask pixel difference set is multiplied with the value of the same pixel position in the frame of the target sample video image to obtain the multiplication result corresponding to the mask pixel difference set, wherein the time point corresponding to the target sample video image in the sample video is the same as the time point corresponding to the sample mask image that generated the mask pixel difference set in the sample video;

[0029] Obtain multiple multiplication results corresponding to the multiple mask pixel difference sets, and superimpose the values ​​at the same pixel position in the multiple multiplication results to obtain the initial background image;

[0030] The initial masking image is adjusted using preset empirical values, and the ratio of the value at the same pixel position in the initial background image and the adjusted initial masking image is calculated to obtain a first ratio result.

[0031] The first ratio result is multiplied by the value of the same pixel position in the sample background mask to obtain the sample background image.

[0032] Optionally, the method further includes:

[0033] A preset first loss function is determined. Based on the first loss function and the human key points annotated on each sample video image frame, loss calculation is performed on the model parameters related to human key point processing in the image segmentation model to obtain a first loss result. The first loss result is then used to adjust the model parameters related to human key point processing in the image segmentation model. The first loss function is either a regression loss function or a heatmap-based loss function; and / or...

[0034] A preset second loss function is determined. Based on the second loss function and the human body masking image sequence, the parameters used to calculate the human body masking pixel value corresponding to each pixel position in the image segmentation model are subjected to loss calculation to obtain a second loss result. The parameters used to calculate the human body masking pixel value corresponding to each pixel position in the image segmentation model are adjusted using the second loss result. The second loss function is a regression loss function or a loss function used to calculate cross-entropy.

[0035] Optionally, calculating the background mask and background image of the video image frame includes:

[0036] Create a cumulative background image and a count image with the same size as the video image frame, wherein the value of each pixel position in the cumulative background image and the count image is 0;

[0037] A preset quality threshold is determined, and the count image is subjected to threshold processing using the preset quality threshold to obtain the background mask;

[0038] The counting image is adjusted using a preset empirical value, and the ratio of the value at the same pixel position in the accumulated background image to the value in the adjusted counting image is calculated to obtain a second ratio result.

[0039] The second ratio result is multiplied by the value of the same pixel position in the background mask to obtain the background image.

[0040] Optionally, the method further includes:

[0041] A preset coefficient is determined, the difference between the preset coefficient and the value of each pixel in the current frame human body mask image is calculated, and the counting image is updated using the difference corresponding to each pixel. When updating the counting image, the difference corresponding to each pixel is superimposed on the value of the pixel mapped to each pixel in the counting image.

[0042] The difference corresponding to each pixel is multiplied by the value of the same pixel position in the video image frame to obtain the intermediate pixel value corresponding to each pixel, and the accumulated background image is updated using the intermediate pixel value corresponding to each pixel. When updating the accumulated background image, the intermediate pixel value corresponding to each pixel is superimposed on the value of the pixel mapped by each pixel on the accumulated background image.

[0043] Optionally, after outputting the current frame human body mask image and the current frame human body key point information as the segmented human body image information, the method further includes:

[0044] Continue acquiring the next video image frame of the video to be processed at the next time step after the current time step;

[0045] Using the updated cumulative background image and the updated count image, calculate the next background mask and the next background image of the next video image frame, and stitch the next video image frame, the next background mask and the next background image in the channel dimension. Input the stitching result into the image segmentation model for segmentation processing to obtain the human body mask image and human body key point information of the next video image frame and output them.

[0046] Based on the obtained human body mask image of the next video image frame, the accumulated background image and count image are updated until the video to be processed ends.

[0047] According to a second aspect of this application, a human body image segmentation apparatus is provided, the apparatus comprising:

[0048] The training module is used to acquire sample videos and train an image segmentation model using the human body key points marked on each sample video image frame and the human body masking image sequence corresponding to the sample video.

[0049] The acquisition module is used to respond to an image segmentation request, determine the video to be processed indicated by the image segmentation request, and acquire video image frames of the video to be processed at the current moment;

[0050] The stitching module is used to calculate the background mask and background image of the video image frame, and stitch the video image frame, the background mask and the background image in the channel dimension to obtain the image stitching result;

[0051] The processing module is used to input the image stitching result into the image segmentation model for segmentation processing, to obtain the current frame human body mask image and the current frame human body key point information of the video image frame, and to output the current frame human body mask image and the current frame human body key point information as the segmented human body image information.

[0052] Optionally, the training module is configured to acquire the sample video, perform video frame extraction on the sample video to obtain a sample video image sequence consisting of multiple sample video image frames; perform annotation processing on each sample video image frame included in the sample video image sequence, annotating human body key points on each sample video image frame, and obtaining the human body mask image sequence; use the human body mask image sequence to calculate the sample background mask and sample background image of the sample video; concatenate the sample video image sequence, the sample background image, and the sample background mask in the channel dimension to obtain a sample concatenation result; and encode and decode the sample concatenation result based on a preset neural network algorithm to construct the image segmentation model.

[0053] Optionally, the training module is used to annotate human key points in each of the plurality of sample video image frames; to perform binary human mask annotation on each sample video image frame to obtain a sample mask image corresponding to each sample video image frame; to obtain a plurality of sample mask images corresponding to the plurality of sample video image frames, and to organize the plurality of sample mask images in chronological order to obtain a sequence of human mask images corresponding to the sample video.

[0054] Optionally, the training module is configured to: determine a preset coefficient for each of the plurality of sample masking images; calculate the difference between the preset coefficient and the value at each pixel position in the sample masking image to obtain a set of masking pixel differences corresponding to the sample masking image; obtain multiple sets of masking pixel differences corresponding to the plurality of sample masking images; superimpose the differences at the same pixel position in the multiple sets of masking pixel differences to obtain an initial masking image; determine a preset quality threshold; apply the preset quality threshold to the initial masking image to obtain the sample background mask, wherein the preset quality threshold is a threshold set according to the labeled quality; and calculate the sample background image based on the plurality of sample video images, the multiple sets of masking pixel differences, the initial masking image, and the sample background mask.

[0055] Optionally, the training module is configured to perform the following processing on each of the plurality of mask pixel difference sets: determine a target sample video image in the plurality of sample video images; multiply the mask pixel difference set with the value of the same pixel position in the frame of the target sample video image to obtain a multiplication result corresponding to the mask pixel difference set, wherein the time point corresponding to the target sample video image in the sample video is the same as the time point corresponding to the sample mask image that generated the mask pixel difference set in the sample video; obtain multiple multiplication results corresponding to the plurality of mask pixel difference sets; superimpose the values ​​of the same pixel position in the multiple multiplication results to obtain an initial background image; adjust the initial mask image using a preset empirical value, and calculate the ratio of the initial background image to the value of the same pixel position in the adjusted initial mask image to obtain a first ratio result; multiply the first ratio result with the value of the same pixel position in the sample background mask to obtain the sample background image.

[0056] Optionally, the training module is further configured to determine a preset first loss function, calculate the loss of model parameters related to human keypoint processing in the image segmentation model based on the first loss function and the human keypoints annotated on each sample video image frame, obtain a first loss result, and adjust the model parameters related to human keypoint processing in the image segmentation model using the first loss result, wherein the first loss function is a regression loss function or a heatmap-based loss function; and / or, determine a preset second loss function, calculate the loss of parameters used to calculate the human mask pixel value corresponding to each pixel position in the image segmentation model based on the second loss function and the human mask image sequence, obtain a second loss result, and adjust the parameters used to calculate the human mask pixel value corresponding to each pixel position in the image segmentation model using the second loss result, wherein the second loss function is a regression loss function or a loss function used to calculate cross-entropy.

[0057] Optionally, the stitching module is used to create a cumulative background image and a counting image with the same frame size as the video image, wherein the value of each pixel position in the cumulative background image and the counting image is 0; determine a preset quality threshold, apply the preset quality threshold to the counting image for thresholding to obtain the background mask; adjust the counting image using a preset empirical value, and calculate the ratio of the value of the same pixel position in the cumulative background image and the adjusted counting image to obtain a second ratio result; multiply the second ratio result by the value of the same pixel position in the background mask to obtain the background image.

[0058] Optionally, the stitching module is further configured to determine a preset coefficient, calculate the difference between the preset coefficient and the value of each pixel in the current frame human body mask image, and update the counting image using the difference corresponding to each pixel, wherein, when updating the counting image, the difference corresponding to each pixel is superimposed on the value of the pixel mapped by each pixel on the counting image; multiply the difference corresponding to each pixel by the value of the same pixel position in the video image frame to obtain the intermediate pixel value corresponding to each pixel, and update the accumulated background image using the intermediate pixel value corresponding to each pixel, wherein, when updating the accumulated background image, the intermediate pixel value corresponding to each pixel is superimposed on the value of the pixel mapped by each pixel on the accumulated background image.

[0059] Optionally, the acquisition module is further configured to continue acquiring the next video image frame of the video to be processed at the next time step of the current time step; using the updated cumulative background image and the updated count image, calculate the next background mask and the next background image of the next video image frame; and stitch the next video image frame, the next background mask, and the next background image in the channel dimension, input the obtained stitching result into the image segmentation model for segmentation processing, obtain the human body mask image and human body key point information of the next video image frame and output them; based on the obtained human body mask image of the next video image frame, continue to update the cumulative background image and the count image until the video to be processed ends.

[0060] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0061] According to a fourth aspect of this application, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0062] By employing the above technical solutions, this application provides a human image segmentation method, apparatus, computer device, and readable storage medium. This application acquires sample videos, uses the human key points annotated on each sample video image frame and the corresponding human mask image sequence in the sample video to train an image segmentation model. In response to an image segmentation request, it determines the video to be processed indicated by the image segmentation request, acquires video image frames of the video to be processed at the current moment, calculates the background mask and background image of the video image frame, and stitches the video image frame, background mask, and background image along the channel dimension to obtain an image stitching result. This image stitching result is then input into the image segmentation model for segmentation processing, obtaining the current frame human mask image and the current frame human key point information of the video image frame. The current frame human mask image and the current frame human key point information are then output as the segmented human image information. This application utilizes an independent image segmentation model to simultaneously predict human key points and human mask images, leveraging the inherent connection between the two tasks to improve the model's prediction performance and the accuracy of the model's output results, while simultaneously reducing the resource overhead required for prediction and saving significant computing power.

[0063] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0064] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0065] Figure 1 This illustration shows a flowchart of a human image segmentation method provided in an embodiment of this application.

[0066] Figure 2A This paper illustrates a flowchart of another human image segmentation method provided in an embodiment of this application.

[0067] Figure 2B A schematic diagram of a human image segmentation method provided in an embodiment of this application is shown;

[0068] Figure 2C This illustration shows a schematic diagram of an image segmentation model training method provided in an embodiment of this application;

[0069] Figure 2D A flowchart illustrating a human image segmentation method provided in an embodiment of this application is shown.

[0070] Figure 3 This paper shows a schematic diagram of the structure of a human image segmentation device provided in an embodiment of this application;

[0071] Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0072] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0073] This application provides a method for human body image segmentation, such as... Figure 1 As shown, the method includes:

[0074] 101. Obtain sample videos, and use the human body key points marked on each sample video image frame and the corresponding human body mask image sequence in the sample video to train the image segmentation model.

[0075] The applicant recognizes that, firstly, existing solutions decompose the human image segmentation process into two parts and predict them separately. While this provides some decoupling, it fails to consider the similarity between the two sub-tasks, resulting in a waste of computational resources. Secondly, existing solutions simply copy relevant algorithms from academia without fully considering the specific characteristics of the scene, namely, the relatively stable environment in the short term. This leads to solutions that are generally generalizable but lack specificity. Therefore, this application proposes a human image segmentation method that uses an image segmentation model to simultaneously predict human keypoints and the human image segmentation result (i.e., human masking). Then, the environmental background is reconstructed using the human image segmentation result, and the image captured by the camera is simultaneously input into the model, achieving a scene-specific solution with faster inference speed.

[0076] To simultaneously predict human keypoints and human occlusions, sample videos are required. The image segmentation model is trained using the human keypoints annotated on each frame of the sample videos and the corresponding human occlusion image sequence. The sample videos are offline video data used to train the unified model. A sample video can be a single offline video or multiple offline video segments; this application does not specify a particular number of video segments included in the sample video. Each frame of the sample videos is annotated with human keypoints, and each sample video corresponds to a human occlusion image sequence. These annotated human keypoints and human occlusion image sequences form the basis for subsequently training the unified image segmentation model.

[0077] 102. In response to the image segmentation request, determine the video to be processed indicated by the image segmentation request, and acquire video image frames of the video to be processed at the current moment.

[0078] In this embodiment, after creating an image segmentation model that simultaneously predicts human keypoints and human occlusion, the image segmentation model can be used to process video images. Specifically, the image segmentation model can be mounted on a video platform; that is, the technical solution of this application can be applied to a video platform or a server providing services to a video platform. The video platform can provide a video processing entry point on the user-facing front end. When the triggering of this video processing entry point is detected, it indicates that the user wishes to perform an image segmentation operation, confirming that the user's image segmentation request has been received.

[0079] In response to an image segmentation request, the video platform determines the video to be processed as indicated in the image segmentation request and captures video image frames of the video to be processed at the current moment. The video to be processed can be a live stream of the user or a real-time video of a voice call with other users, or it can be a pre-recorded and uploaded video; this application does not impose specific limitations on this. It should be noted that if it is real-time video, the current frame of the real-time video can be captured as a video image frame after receiving the image segmentation request; while if it is a recorded video, the user-selected time point can be determined as the current time within the recorded video and its frame captured as a video image frame.

[0080] 103. Calculate the background mask and background image of the video image frame, and stitch the video image frame, background mask and background image in the channel dimension to obtain the image stitching result.

[0081] In this embodiment, after acquiring the video image frame at the current moment, the background mask and background image of the video image frame are calculated. The video image frame, background mask, and background image are then stitched together along the channel dimension to obtain an image stitching result. This facilitates subsequent prediction of human key points and human masking using the image stitching result simultaneously. In practical applications, the background mask and background image can be single-channel grayscale images, while the video image frame can be a five-channel CMYK (Cyan Magenta Yellow blacK, a printing color mode) image. Thus, by stitching along the channel dimension, the five-channel image is stitched into a seven-channel image. It should be noted that the video image frame can also be a single-channel, three-channel, or four-channel image; this application does not specifically limit this.

[0082] 104. Input the image stitching result into the image segmentation model for segmentation processing to obtain the current frame human body mask image and the current frame human body key point information of the video image frame, and output the current frame human body mask image and the current frame human body key point information as the human body image information obtained by segmentation.

[0083] In this embodiment of the application, after obtaining the image stitching result, the image stitching result is input into the image segmentation model for segmentation processing, so as to obtain the current frame human body mask image and the current frame human body key point information of the video image frame, and output the current frame human body mask image and the current frame human body key point information as the human body image information obtained by segmentation.

[0084] It should be noted that after the current video image frame is segmented, the system continues to wait for the acquisition of the next frame, and then repeats steps 103 and 104 above for the acquired next video image frame. Furthermore, in practical applications, the processing of each video image frame can be performed sequentially, or multiple video image frames can be processed in parallel, such as real-time parallel processing of 30 or 40 frames, etc. This application does not specifically limit whether video image frames are processed sequentially or in parallel, or how many video image frames are processed in parallel.

[0085] The method provided in this application acquires sample videos, trains an image segmentation model using human keypoints annotated on each sample video image frame and the corresponding human mask image sequence, responds to an image segmentation request, determines the video to be processed indicated by the image segmentation request, acquires video image frames of the video to be processed at the current time, calculates the background mask and background image of the video image frame, stitches the video image frame, background mask, and background image in the channel dimension to obtain the image stitching result, and inputs the image stitching result into the image segmentation model for segmentation processing to obtain the current frame human mask image and the current frame human keypoint information of the video image frame, and outputs the current frame human mask image and the current frame human keypoint information as the segmented human image information. It uses an independent image segmentation model to simultaneously predict human keypoints and human mask images, improves the prediction effect of the model by leveraging the inherent connection between the two tasks, improves the accuracy of the model output results, and reduces the resource overhead required for prediction, saving a lot of computing power.

[0086] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, this application provides another human image segmentation method, such as... Figure 2A As shown, the method includes:

[0087] 201. Obtain sample videos, and use the human body key points marked on each sample video image frame and the corresponding human body mask image sequence to train the image segmentation model.

[0088] In this embodiment, to simultaneously predict human keypoints and human masks, sample videos need to be acquired. The image segmentation model is trained using the human keypoints annotated on each sample video frame and the corresponding human mask image sequence. The sample videos are offline video data used to train the unified model. A sample video can be a single offline video or multiple offline video segments; this application does not specify the number of video segments included in the sample video. After training the image segmentation model, when using it for subsequent image processing, the model input includes the currently acquired video image frame, the background mask constructed using the video image frame, and the background image. The model output is the human mask image and human keypoint information of the currently acquired video image frame. The process of training the image segmentation model is described below, specifically including steps one through four:

[0089] Step 1: Obtain sample videos and perform frame extraction on the sample videos to obtain a sample video image sequence consisting of multiple sample video image frames.

[0090] In this case, assuming a sample video V1, performing video frame extraction on V1 can yield multiple video image frames of V1 (i.e., multiple sample video image frames). These video image frames can then be used to construct the sample video image sequence I1 of V1. t The value of t ranges from 0 to T1-1, indicating the time point corresponding to each sample video image frame in the sample video.

[0091] It should be noted that there can be one or more sample videos. If there is only one, it can be represented directly by V. If there are multiple, it can be represented by V1. i This means that the value of i can be from 0 to n-1, where n is a positive integer. That is, multiple sample videos are numbered using V1, V2, etc., so the sample video image sequence obtained by frame extraction can also be represented according to the corresponding number, such as I1. t This application does not specify the number of sample videos used to train the model, such as It2.

[0092] Step 2: Annotate each sample video image frame in the sample video image sequence, annotate human body key points in each sample video image frame, and obtain the human body masking image sequence.

[0093] After generating a sample video image sequence for the sample video, it is necessary to annotate each sample video image frame in the sample video image sequence, annotating human key points for each sample video image frame, and obtaining a human body masking image sequence. The specific process is as follows:

[0094] First, human keypoint annotation is performed on each of the multiple sample video image frames, that is, key parts such as important joints like the knees and elbows, or important parts like the head and feet are annotated. Then, binary human mask annotation is performed on each sample video image frame to obtain the sample mask image corresponding to each sample video image frame. Multiple sample mask images corresponding to multiple sample video image frames are then obtained and arranged in chronological order to obtain the human mask image sequence corresponding to the sample video.

[0095] In this process, human body masking involves highlighting the area containing the human body in the image, while filling the area outside the human body with black. Specifically, as shown below... Figure 2B As shown, the generated sample mask image can be as follows: Figure 2B The white highlighted areas represent the human body, while the areas outside the human body are black.

[0096] It should be noted that, considering the large number of sample videos at times, in order to distinguish different human key points and human masking image sequences from different sample videos, P can be used. i t Represents a sequence of human body masking images, using Here, j represents the key points of the human body in the current image, and t and i are consistent with the description in step one above. t represents the corresponding time point in the sample video, and i represents the video number of the sample video. Since i only needs to be represented in the case of multiple sample videos, for ease of description, the following process uses one sample video as an example and omits i. In actual application, the number of sample videos is not limited. In the case of multiple sample videos, i can be used to number each sample video and the sequence of human body key points and human body masking images taken from that sample video.

[0097] Step 3: Using the human body masking image sequence, calculate the sample background mask and sample background image of the sample video.

[0098] After obtaining the human body masking image sequence, the sample background mask and sample background image of the sample video are calculated using the human body masking image sequence. The process of calculating the sample background mask is described first:

[0099] For each sample masking image in multiple sample masking images, a preset coefficient is determined, and the difference between the preset coefficient and the value at each pixel position in the sample masking image is calculated to obtain the set of masking pixel differences corresponding to the sample masking image. Subsequently, multiple sets of masking pixel differences corresponding to multiple sample masking images are obtained, and the differences at the same pixel position in the multiple sets of masking pixel differences are superimposed to obtain the initial masking image. Finally, a preset quality threshold is determined, and the initial masking image is thresholded using the preset quality threshold to obtain the sample background mask. The preset quality threshold is a threshold set based on the annotation quality, and can be represented by S. The value of S ranges from 1 to T-1, and can be set within this range according to the annotation quality; the higher the annotation quality, the lower S can be set. In practical applications, the initial masking image can be calculated using the following formula 1, and the sample background mask can be calculated using the following formula 2:

[0100] Formula 1:

[0101] Formula 2:

[0102] In Formula 1, This represents the initial masking image, 1 represents the preset coefficient, and P t Let M represent the sample masking image at time t, where t ranges from 0 to T-1, and the value of T is determined by the duration of the sample video; in Formula 2, M represents the sample background masking. This represents the initial masked image, and S represents the preset quality threshold.

[0103] After obtaining the initial mask image and the sample background mask, the sample background image is calculated based on multiple sample video images, multiple sets of mask pixel differences, the initial mask image, and the sample background mask. The process of calculating the sample background image is described below:

[0104] For each of the multiple mask pixel difference sets, the following processing is performed: A target sample video image is determined from multiple sample video images. The mask pixel difference set is multiplied by the value at the same pixel position in the frame of the target sample video image to obtain the multiplication result corresponding to the mask pixel difference set. The time point corresponding to the target sample video image in the sample video is the same as the time point corresponding to the sample mask image that generated the mask pixel difference set in the sample video. Subsequently, multiple multiplication results corresponding to the multiple mask pixel difference sets are obtained. The values ​​at the same pixel position in the multiple multiplication results are superimposed to obtain an initial background image. The initial mask image is then adjusted using a preset empirical value, and the ratio of the value at the same pixel position in the initial background image to the value at the same pixel position in the adjusted initial mask image is calculated to obtain a first ratio result. Finally, the first ratio result is multiplied by the value at the same pixel position in the sample background mask to obtain the sample background image. In practical applications, the initial background image can be calculated using the following formula 3, and the sample background image can be calculated using the following formula 4.

[0105] Formula 3:

[0106] Formula 4:

[0107] In formula 3, This represents the initial background image, 1 represents the preset coefficient, and P t Let I represent the sample masking image at time t, where t ranges from 0 to T-1, and the value of T is determined by the duration of the sample video. t This represents the sample video image (i.e., the target sample video image) at time t. This indicates that the values ​​at the same pixel location are multiplied; in Formula 4, B represents the sample background image. M represents the initial background image, and M represents the sample background mask. This represents the initial masking image, and esp represents the preset empirical value. This represents the multiplication of values ​​at the same pixel location.

[0108] Step 4: The sample video image sequence, sample background image, and sample background mask are stitched together along the channel dimension to obtain the sample stitching result. The sample stitching result is then encoded and decoded based on a preset neural network algorithm to construct an image segmentation model.

[0109] In this embodiment, after calculating the sample background image and sample background mask, the sample video image sequence, sample background image, and sample background mask are stitched together along the channel dimension to obtain the sample stitching result. The sample stitching result is then encoded and decoded based on a preset neural network algorithm to construct an image segmentation model. The sample background image and sample background mask can be single-channel grayscale images, while each sample video image in the sample video image sequence can be a five-channel CMYK image. Thus, by stitching along the channel dimension, the five-channel image is stitched together into a seven-channel image. It should be noted that the sample video image can also be a single-channel, three-channel, or four-channel image; this application does not specifically limit this.

[0110] Next, a unified image segmentation model is constructed using the sample concatenation results. This image segmentation model can be a convolutional neural network, a Transformer (a type of neural network architecture), or similar structure, comprising both an encoder and a decoder, thus enabling the encoding and decoding of the sample concatenation results. The specific training process is as follows: Figure 2C As shown, the input during model training is I. t B and M, which are the concatenated results of the target sample video image, sample background image, and sample background mask in the sample video image sequence along the channel dimension, are encoded using encoder Enc and then decoded using decoder Dec to create an image segmentation model. This image segmentation model can then be used to output... Figure 2C The human body key point information K shown t and human body masking image P t .

[0111] It should be noted that, in order to improve the accuracy of the model, supervised training can be performed on the model using human keypoint annotations and human mask image sequences on each sample video image frame during training. Specifically, for the human keypoint branch in the image segmentation model, a preset first loss function can be determined. Based on the first loss function and the human keypoint annotations on each sample video image frame, the model parameters related to human keypoint processing in the image segmentation model are loss-calculated to obtain the first loss result. The model parameters related to human keypoint processing in the image segmentation model are then adjusted using the first loss result. The first loss function is a regression loss function such as L1, L2, or WingLoss (progressive loss) or a loss function based on a heatmap. For the human body masking branch in the image segmentation model, a pre-defined second loss function can be determined. Based on the second loss function and the human body masking image sequence, the parameters used in the image segmentation model to calculate the human body masking pixel value corresponding to each pixel position are subjected to loss calculation to obtain the second loss result. The parameters used in the image segmentation model to calculate the human body masking pixel value corresponding to each pixel position are then adjusted using the second loss result. The second loss function is a regression loss function such as L1, L2, or WingLoss, or a loss function used to calculate cross-entropy. In this way, the image segmentation model can be optimized through the process of calculating loss, gradient backpropagation, and updating parameters.

[0112] 202. In response to the image segmentation request, determine the video to be processed indicated by the image segmentation request, and acquire video image frames of the video to be processed at the current moment.

[0113] In this embodiment, after creating an image segmentation model that simultaneously predicts human keypoints and human occlusion, the image segmentation model can be used to process video images. Specifically, the image segmentation model can be mounted on a video platform; that is, the technical solution of this application can be applied to a video platform or a server providing services to a video platform. The video platform can provide a video processing entry point on the user-facing front end. When the triggering of this video processing entry point is detected, it indicates that the user wishes to perform an image segmentation operation, confirming that the user's image segmentation request has been received.

[0114] In response to an image segmentation request, the video platform determines the video to be processed as indicated in the image segmentation request and captures video image frames of the video to be processed at the current moment. The video to be processed can be a live stream of the user or a real-time video of a voice call with other users, or it can be a pre-recorded and uploaded video; this application does not impose specific limitations on this. It should be noted that if it is real-time video, the current frame of the real-time video can be captured as a video image frame after receiving the image segmentation request; while if it is a recorded video, the user-selected time point can be determined as the current time within the recorded video and its frame captured as a video image frame.

[0115] 203. Create a cumulative background image and a count image with the same dimensions as the video image frames.

[0116] Since the image segmentation model will subsequently output predicted results, these results need to be recorded and used for subsequent image output. Therefore, in this embodiment, a cumulative background image and a counting image of the same size as the video image frame are created to fill the content calculated later. In both the cumulative background image and the counting image, the value of each pixel is 0. For ease of explanation, the cumulative background image is denoted as B0 and the counting image as M0, both filled with 0.

[0117] 204. Determine the preset quality threshold, perform thresholding on the counting image using the preset quality threshold to obtain the background mask, adjust the counting image using the preset empirical value, and calculate the ratio of the value of the same pixel position in the accumulated background image and the adjusted counting image to obtain the second ratio result. Multiply the second ratio result by the value of the same pixel position in the background mask to obtain the background image.

[0118] In this embodiment, it is necessary to calculate the background mask and background image of the video image frame, which will then be used for stitching in the channel dimension. Specifically, when calculating the background mask, a preset quality threshold needs to be determined. The counted image is then subjected to thresholding using this preset quality threshold to obtain the background mask. The specific process is consistent with the process of calculating the sample background mask in step 201 above, and can also be implemented using Formula 2. In Formula 2, the... That is, the input counting image M0. For the specific calculation process, please refer to the description in step 201, which will not be repeated here.

[0119] Next, the counting image is adjusted using preset empirical values, and the ratio of the value at the same pixel position in the accumulated background image to that in the adjusted counting image is calculated to obtain a second ratio result. This second ratio result is then multiplied by the value at the same pixel position in the background mask to obtain the background image. The process of calculating the background image is consistent with the process of calculating the sample background image in step 201 above, and can also be implemented using Formula 4. Formula 4... That is, the cumulative background image B0 that is brought in. That is, the input counting image M0. For the specific calculation process, please refer to the description in step 201, which will not be repeated here.

[0120] 205. The video image frame, background mask, and background image are stitched together along the channel dimension to obtain the image stitching result. The image stitching result is then input into the image segmentation model for segmentation processing to obtain the current frame human body mask image and the current frame human body key point information of the video image frame. The current frame human body mask image and the current frame human body key point information are then output as the human body image information obtained from the segmentation.

[0121] In this embodiment, after calculating the background mask and background image, the video image frame, background mask, and background image are stitched together along the channel dimension to obtain an image stitching result. This image stitching result is then input into an image segmentation model for segmentation processing, yielding the current frame human body mask image and the current frame human body key point information of the video image frame. The current frame human body mask image and the current frame human body key point information are then output as the segmented human body image information. The process of stitching along the channel dimension is consistent with the process described in step four of step 201 above, which involves stitching the sample video image sequence, sample background image, and sample background mask along the channel dimension; therefore, it will not be repeated here.

[0122] 206. Update the cumulative background image and count image using the current frame's human body mask image.

[0123] In this embodiment, since the cumulative background image B0 and the count image M0 are needed to record the output prediction result, the cumulative background image and the count image are updated using the current frame's human body mask image. The updating process of the cumulative background image and the count image is described below:

[0124] When updating the counting image, it is necessary to determine a preset coefficient, calculate the difference between the preset coefficient and the value of each pixel in the current frame's human body mask image, and update the counting image using the difference corresponding to each pixel. Specifically, when updating the counting image, the difference corresponding to each pixel is superimposed on the value of the pixel mapped to that pixel in the counting image. This can be achieved through the following formula 5:

[0125] Formula 5:

[0126] in, The updated counting image is shown below, where M0 is the original counting image and P is the updated image. t The current frame contains the human body masking image, and 1 is a preset coefficient.

[0127] When updating the cumulative background image, the difference corresponding to each pixel needs to be multiplied by the value of the same pixel position in the video image frame to obtain the intermediate pixel value corresponding to each pixel. Then, the cumulative background image is updated using the intermediate pixel value corresponding to each pixel. Specifically, when updating the cumulative background image, the intermediate pixel value corresponding to each pixel is superimposed on the value of the pixel mapped to each pixel in the cumulative background image. This can be achieved using the following formula 6:

[0128] Formula 6:

[0129] in, B0 is the updated cumulative background image, and P is the original cumulative background image. t The image shows the human body masking in the current frame, where 1 is a preset coefficient. t Represents a video image frame.

[0130] After updating the accumulated background image and the count image, the system continues to acquire the next video frame from the next moment in the video to be processed. Using the updated accumulated background image and the updated count image, it calculates the next background mask and the next background image for the next video frame. Then, it stitches the next video frame, the next background mask, and the next background image together along the channel dimension. The stitched result is input into an image segmentation model for segmentation, yielding the human body mask image and human keypoint information for the next video frame, which is then output. This process is repeated, updating the accumulated background image and the count image based on the obtained human body mask image for the next video frame, until the video to be processed ends.

[0131] In summary, the technical solution of this application is summarized as follows: Figure 2DAs shown, after acquiring the video image frame at the current moment, the background mask and background image of the video image frame are calculated. The video image frame, background mask, and background image are then stitched together along the channel dimension to obtain the image stitching result. This image stitching result is then input into the image segmentation model for segmentation processing. The image segmentation model outputs the human body mask image and human body key point information of the current frame for subsequent analysis, achieving human body segmentation and human body key point localization under stable background conditions. In this way, on the one hand, the technical solution of this application can simultaneously identify the background mask and background image, and realize the reconstruction of the environmental background through the stitching of the background mask and background image. The background mask is then combined with the real-time image and fed into the model for analysis, making full use of environmental information to reduce model errors. On the other hand, the technical solution of this application uses a unified model to simultaneously predict human body image matting and human body key points, improving the model effect by leveraging the inherent connection between the two tasks, while reducing the resource overhead required by existing technologies that use two sets of models.

[0132] The method provided in this application uses an independent image segmentation model to simultaneously predict human key points and human mask images. By leveraging the inherent connection between the two tasks, the prediction effect of the model is improved, the accuracy of the model output results is increased, and the resource overhead required for prediction is reduced, saving a lot of computing power.

[0133] Furthermore, as Figure 1 To specifically implement the method, this application provides a human image segmentation device, such as... Figure 3 As shown, the device includes: a training module 301, a data acquisition module 302, a stitching module 303, and a processing module 304.

[0134] The training module 301 is used to acquire sample videos and train an image segmentation model by using the human body key points marked on each sample video image frame in the sample video and the human body masking image sequence corresponding to the sample video.

[0135] The acquisition module 302 is used to respond to an image segmentation request, determine the video to be processed indicated by the image segmentation request, and acquire video image frames of the video to be processed at the current moment;

[0136] The stitching module 303 is used to calculate the background mask and background image of the video image frame, and stitch the video image frame, the background mask and the background image in the channel dimension to obtain the image stitching result;

[0137] The processing module 304 is used to input the image stitching result into the image segmentation model for segmentation processing, to obtain the current frame human body mask image and the current frame human body key point information of the video image frame, and to output the current frame human body mask image and the current frame human body key point information as the segmented human body image information.

[0138] In a specific application scenario, the training module 301 is used to acquire the sample video, perform video frame extraction on the sample video to obtain a sample video image sequence consisting of multiple sample video image frames; perform annotation processing on each sample video image frame included in the sample video image sequence to annotate human key points on each sample video image frame, and obtain the human body mask image sequence; use the human body mask image sequence to calculate the sample background mask and sample background image of the sample video; concatenate the sample video image sequence, the sample background image, and the sample background mask in the channel dimension to obtain the sample concatenation result; and encode and decode the sample concatenation result based on a preset neural network algorithm to construct the image segmentation model.

[0139] In a specific application scenario, the training module 301 is used to annotate human key points in each of the multiple sample video image frames; to perform binary human mask annotation on each sample video image frame to obtain a sample mask image corresponding to each sample video image frame; to obtain multiple sample mask images corresponding to the multiple sample video image frames, and to organize the multiple sample mask images in chronological order to obtain a sequence of human mask images corresponding to the sample video.

[0140] In a specific application scenario, the training module 301 is used to: determine a preset coefficient for each of the multiple sample masking images; calculate the difference between the preset coefficient and the value at each pixel position in the sample masking image to obtain a set of masking pixel differences corresponding to the sample masking image; obtain multiple sets of masking pixel differences corresponding to the multiple sample masking images; superimpose the differences at the same pixel position in the multiple sets of masking pixel differences to obtain an initial masking image; determine a preset quality threshold; apply the preset quality threshold to the initial masking image to obtain the sample background mask, wherein the preset quality threshold is a threshold set according to the labeled quality; and calculate the sample background image based on the multiple sample video images, the multiple sets of masking pixel differences, the initial masking image, and the sample background mask.

[0141] In a specific application scenario, the training module 301 is used to perform the following processing on each of the plurality of mask pixel difference sets: determining a target sample video image in the plurality of sample video images; multiplying the mask pixel difference set with the value at the same pixel position in the frame of the target sample video image to obtain the multiplication result corresponding to the mask pixel difference set, wherein the time point corresponding to the target sample video image in the sample video is the same as the time point corresponding to the sample mask image that generated the mask pixel difference set in the sample video; obtaining multiple multiplication results corresponding to the plurality of mask pixel difference sets; superimposing the values ​​at the same pixel position in the multiple multiplication results to obtain an initial background image; adjusting the initial mask image using a preset empirical value, and calculating the ratio of the initial background image to the value at the same pixel position in the adjusted initial mask image to obtain a first ratio result; multiplying the first ratio result with the value at the same pixel position in the sample background mask to obtain the sample background image.

[0142] In specific application scenarios, the training module 301 is further configured to determine a preset first loss function, calculate the loss of model parameters related to human keypoint processing in the image segmentation model based on the first loss function and the human keypoints annotated on each sample video image frame, obtain a first loss result, and adjust the model parameters related to human keypoint processing in the image segmentation model using the first loss result, wherein the first loss function is a regression loss function or a heatmap-based loss function; and / or, determine a preset second loss function, calculate the loss of parameters used to calculate the human mask pixel value corresponding to each pixel position in the image segmentation model based on the second loss function and the human mask image sequence, obtain a second loss result, and adjust the parameters used to calculate the human mask pixel value corresponding to each pixel position in the image segmentation model using the second loss result, wherein the second loss function is a regression loss function or a loss function used to calculate cross-entropy.

[0143] In a specific application scenario, the stitching module 303 is used to create a cumulative background image and a counting image with the same size as the video image frame, wherein the value of each pixel position in the cumulative background image and the counting image is 0; determine a preset quality threshold, and use the preset quality threshold to perform threshold processing on the counting image to obtain the background mask; adjust the counting image using a preset empirical value, and calculate the ratio of the value of the same pixel position in the cumulative background image and the adjusted counting image to obtain a second ratio result; multiply the second ratio result by the value of the same pixel position in the background mask to obtain the background image.

[0144] In specific application scenarios, the stitching module 303 is also used to determine a preset coefficient, calculate the difference between the preset coefficient and the value of each pixel in the current frame human body mask image, and update the counting image using the difference corresponding to each pixel. When updating the counting image, the difference corresponding to each pixel is superimposed on the value of the pixel mapped to the counting image. The difference corresponding to each pixel is multiplied by the value at the same pixel position in the video image frame to obtain the intermediate pixel value corresponding to each pixel. The intermediate pixel value corresponding to each pixel is then used to update the accumulated background image. When updating the accumulated background image, the intermediate pixel value corresponding to each pixel is superimposed on the value of the pixel mapped to the accumulated background image.

[0145] In specific application scenarios, the acquisition module 302 is also used to continue acquiring the next video image frame of the video to be processed at the next moment of the current moment; using the updated cumulative background image and the updated count image, calculate the next background mask and the next background image of the next video image frame; and stitch the next video image frame, the next background mask, and the next background image in the channel dimension, input the obtained stitching result into the image segmentation model for segmentation processing, obtain the human body mask image and human body key point information of the next video image frame and output them; based on the obtained human body mask image of the next video image frame, continue to update the cumulative background image and the count image until the video to be processed ends.

[0146] The apparatus provided in this application acquires sample videos, trains an image segmentation model using human keypoints annotated on each sample video image frame and the corresponding human mask image sequence, responds to an image segmentation request, determines the video to be processed indicated by the image segmentation request, acquires video image frames of the video to be processed at the current time, calculates the background mask and background image of the video image frame, stitches the video image frame, background mask, and background image in the channel dimension to obtain the image stitching result, and inputs the image stitching result into the image segmentation model for segmentation processing to obtain the current frame human mask image and the current frame human keypoint information of the video image frame, and outputs the current frame human mask image and the current frame human keypoint information as the segmented human image information. It uses an independent image segmentation model to simultaneously predict human keypoints and human mask images, improves the prediction effect of the model by leveraging the inherent connection between the two tasks, improves the accuracy of the model output results, and reduces the resource overhead required for prediction, saving a lot of computing power.

[0147] It should be noted that other corresponding descriptions of the functional units involved in the human image segmentation device provided in this application embodiment can be found in the following references. Figure 1 and Figures 2A to 2D The corresponding descriptions in [the document] will not be repeated here.

[0148] In an exemplary embodiment, see Figure 4 The invention also provides a computer device including a bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device, wherein the various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the human image segmentation method described in the above embodiments.

[0149] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the human image segmentation method.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0151] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0152] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0153] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0154] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for human body image segmentation, characterized in that, include: A sample video is acquired, and an image segmentation model is trained using the human key points annotated on each sample video image frame and the corresponding human mask image sequence. Specifically, the sample video is acquired, and frame extraction is performed to obtain a sample video image sequence composed of multiple sample video image frames. Each sample video image frame in the sample video image sequence is annotated with human key points, and a human mask image sequence is obtained. Using the human mask image sequence, a sample background mask and a sample background image are calculated. The sample video image sequence, the sample background image, and the sample background mask are concatenated along the channel dimension to obtain a sample concatenation result. The sample concatenation result is encoded and decoded based on a preset neural network algorithm to construct the image segmentation model. The step of calculating the sample background mask and sample background image of the sample video using the human body masking image sequence includes: for each sample masking image in a plurality of sample masking images, determining a preset coefficient, calculating the difference between the preset coefficient and the value at each pixel position in the sample masking image to obtain a set of masking pixel differences corresponding to the sample masking image; obtaining a plurality of masking pixel difference sets corresponding to the plurality of sample masking images, superimposing the differences at the same pixel position in the plurality of masking pixel difference sets to obtain an initial masking image; determining a preset quality threshold, applying the preset quality threshold to the initial masking image for thresholding processing to obtain the sample background mask, wherein the preset quality threshold is a threshold set according to the labeled quality; and calculating the sample background image based on the plurality of sample video images, the plurality of masking pixel difference sets, the initial masking image, and the sample background mask. In response to an image segmentation request, determine the video to be processed indicated by the image segmentation request, and acquire video image frames of the video to be processed at the current moment; Calculate the background mask and background image of the video image frame, and then stitch the video image frame, the background mask, and the background image together along the channel dimension to obtain the image stitching result; The image stitching result is input into the image segmentation model for segmentation processing to obtain the current frame human body mask image and the current frame human body key point information of the video image frame. The current frame human body mask image and the current frame human body key point information are then output as the segmented human body image information.

2. The method according to claim 1, characterized in that, The step of annotating each sample video image frame in the sample video image sequence, annotating human body key points in each sample video image frame, and obtaining the human body masking image sequence includes: Human body key point annotation is performed on each of the plurality of sample video image frames; Binary human body mask annotation is performed on each sample video image frame to obtain the sample mask image corresponding to each sample video image frame; Multiple sample mask images corresponding to the multiple sample video image frames are obtained, and the multiple sample mask images are arranged in chronological order to obtain the human body mask image sequence corresponding to the sample video.

3. The method according to claim 1, characterized in that, The step of calculating the sample background image based on the plurality of sample video images, the plurality of mask pixel difference sets, the initial mask image, and the sample background mask includes: For each of the plurality of mask pixel difference sets, the following processing is performed: a target sample video image is determined in the plurality of sample video images, and the mask pixel difference set is multiplied with the value of the same pixel position in the target sample video image to obtain the multiplication result corresponding to the mask pixel difference set, wherein the time point corresponding to the target sample video image in the sample video is the same as the time point corresponding to the sample mask image that generated the mask pixel difference set in the sample video; Obtain multiple multiplication results corresponding to the multiple mask pixel difference sets, and superimpose the values ​​at the same pixel position in the multiple multiplication results to obtain the initial background image; The initial masking image is adjusted using preset empirical values, and the ratio of the value at the same pixel position in the initial background image and the adjusted initial masking image is calculated to obtain a first ratio result. The first ratio result is multiplied by the value of the same pixel position in the sample background mask to obtain the sample background image.

4. The method according to claim 1, characterized in that, The method further includes: A preset first loss function is determined. Based on the first loss function and the human key points annotated on each sample video image frame, loss calculation is performed on the model parameters related to human key point processing in the image segmentation model to obtain a first loss result. The first loss result is then used to adjust the model parameters related to human key point processing in the image segmentation model. The first loss function is either a regression loss function or a heatmap-based loss function; and / or... A preset second loss function is determined. Based on the second loss function and the human body masking image sequence, the parameters used to calculate the human body masking pixel value corresponding to each pixel position in the image segmentation model are subjected to loss calculation to obtain a second loss result. The parameters used to calculate the human body masking pixel value corresponding to each pixel position in the image segmentation model are adjusted using the second loss result. The second loss function is a regression loss function or a loss function used to calculate cross-entropy.

5. The method according to claim 1, characterized in that, The calculation of the background mask and background image of the video image frame includes: Create a cumulative background image and a count image with the same size as the video image frame, wherein the value of each pixel position in the cumulative background image and the count image is 0; A preset quality threshold is determined, and the count image is subjected to threshold processing using the preset quality threshold to obtain the background mask; The counting image is adjusted using a preset empirical value, and the ratio of the value at the same pixel position in the accumulated background image to the value in the adjusted counting image is calculated to obtain a second ratio result. The second ratio result is multiplied by the value of the same pixel position in the background mask to obtain the background image.

6. The method according to claim 5, characterized in that, The method further includes: A preset coefficient is determined, the difference between the preset coefficient and the value of each pixel in the current frame human body mask image is calculated, and the counting image is updated using the difference corresponding to each pixel. When updating the counting image, the difference corresponding to each pixel is superimposed on the value of the pixel mapped to each pixel in the counting image. The difference corresponding to each pixel is multiplied by the value of the same pixel position in the video image frame to obtain the intermediate pixel value corresponding to each pixel, and the accumulated background image is updated using the intermediate pixel value corresponding to each pixel. When updating the accumulated background image, the intermediate pixel value corresponding to each pixel is superimposed on the value of the pixel mapped by each pixel on the accumulated background image.

7. The method according to claim 1, characterized in that, After outputting the current frame human body mask image and the current frame human body key point information as the segmented human body image information, the method further includes: Continue acquiring the next video image frame of the video to be processed after the current moment; Using the updated cumulative background image and the updated count image, calculate the next background mask and the next background image of the next video image frame, and stitch the next video image frame, the next background mask and the next background image in the channel dimension. Input the stitching result into the image segmentation model for segmentation processing to obtain the human body mask image and human body key point information of the next video image frame and output them. Based on the obtained human body mask image of the next video image frame, the accumulated background image and count image are updated until the video to be processed ends.

8. A human body image segmentation device, characterized in that, include: The training module is used to acquire sample videos and train an image segmentation model using the human body key points marked on each sample video image frame in the sample video and the human body mask image sequence corresponding to the sample video; the training module is used to acquire the sample videos and perform video frame extraction processing on the sample videos to obtain a sample video image sequence of the sample videos composed of multiple sample video image frames. Each sample video image frame in the sample video image sequence is labeled, and human key points are labeled to each sample video image frame to obtain the human body masking image sequence; using the human body masking image sequence, the sample background mask and sample background image of the sample video are calculated. The sample video image sequence, the sample background image, and the sample background mask are concatenated along the channel dimension to obtain a sample concatenation result. The sample concatenation result is then encoded and decoded using a preset neural network algorithm to construct the image segmentation model. The training module is used to determine a preset coefficient for each sample mask image in the multiple sample mask images, calculate the difference between the preset coefficient and the value at each pixel position in the sample mask image, and obtain a set of mask pixel differences corresponding to the sample mask image. Multiple sets of mask pixel differences corresponding to the multiple sample mask images are obtained, and the differences at the same pixel position in the multiple sets of mask pixel differences are superimposed to obtain an initial mask image. A preset quality threshold is determined, and the initial mask image is thresholded using the preset quality threshold to obtain the sample background mask. The preset quality threshold is a threshold set based on the labeled quality. The sample background image is calculated based on the multiple sample video images, the multiple sets of mask pixel differences, the initial mask image, and the sample background mask. The acquisition module is used to respond to an image segmentation request, determine the video to be processed indicated by the image segmentation request, and acquire video image frames of the video to be processed at the current moment; The stitching module is used to calculate the background mask and background image of the video image frame, and stitch the video image frame, the background mask and the background image in the channel dimension to obtain the image stitching result; The processing module is used to input the image stitching result into the image segmentation model for segmentation processing, to obtain the current frame human body mask image and the current frame human body key point information of the video image frame, and to output the current frame human body mask image and the current frame human body key point information as the segmented human body image information.

Images