Video image segmentation method and its device, equipment, and medium

Abstract

The present application relates to a video image segmentation method and its apparatus, device, and medium in the field of live network broadcast technology. The method comprises: obtaining a first image mask corresponding to a current image frame in a live video stream; determining a difference change map based on the confidence of corresponding pixels of the first image mask and a second image mask corresponding to a previous image frame, wherein the difference change map indicates whether each pixel belongs to the foreground in both the current image frame and the previous image frame; determining a low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguishing whether a pixel belongs to the low-confidence image region and selecting a confidence from the first image mask and the second image mask to construct an effective image mask of the current image frame; and separating a foreground image and a background image from the current image frame based on the effective image mask. The present application can accurately segment the foreground and background in an image frame.

Description

Technical Field

[0001] The present application relates to the field of network live broadcast technology, and in particular to a video image segmentation method and its corresponding device, computer equipment, and computer-readable storage medium. Background Art

[0002] In the online live broadcast scenario, the anchor user pushes the video stream to the live broadcast room to achieve application purposes such as talent display, information sharing, and knowledge education, allowing the anchor user to participate in social labor and obtain income through these activities, thereby promoting overall social benefits.

[0003] Currently, traditional video image segmentation technology usually uses a large number of image frames of the video stream with labeled foreground and background, background information between previous and next image frames in the video stream, and timing information between image frames in the video stream during the training phase of the deep learning model. After the model is trained to convergence, the deep learning model is used to segment the foreground and background in the image frames of the video stream. However, there are problems such as high labeling and training costs, and excessive model computational complexity.

[0004] In view of the shortcomings of traditional technology, the applicant has been engaged in research in related fields for a long time and has taken a different approach to solve the problems in the industry. Summary of the Invention

[0005] The primary purpose of the present application is to solve at least one of the above problems and to provide a video image segmentation method and its corresponding device, computer equipment, and computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A video image segmentation method provided to meet one of the purposes of this application includes the following steps:

[0008] Obtaining a first image mask corresponding to a current image frame in a live video stream, wherein the image mask includes a confidence level corresponding to each pixel in the image frame;

[0009] determining a difference change map based on the confidence levels of corresponding pixels of the first image mask and the second image mask corresponding to the previous image frame, wherein the difference change map indicates whether the confidence level of each pixel changes between the current image frame and the previous image frame;

[0010] determining a low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguishing whether a pixel point belongs to the low-confidence image region, and selecting a confidence from the first image mask and the second image mask to construct a valid image mask of the current image frame;

[0011] A foreground image and a background image are separated from the current image frame according to the valid image mask.

[0012] In a further embodiment, the step of determining the difference change map based on the confidence levels of corresponding pixels of the first image mask and the second image mask corresponding to the previous image frame includes the following steps:

[0013] Binarizing the first image mask and the second image mask respectively to obtain a first distinction map and a second distinction map;

[0014] An exclusive-OR operation result is obtained between corresponding pixels of the first distinction map and the second distinction map to obtain a difference change map.

[0015] In a further embodiment, the step of determining the low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguishing whether a pixel point belongs to the low-confidence image region, and selecting a confidence from the first image mask and the second image mask to construct a valid image mask for the current image frame includes the following steps:

[0016] Quantitatively determining whether the pixel belongs to the foreground based on the difference change map, and determining whether the confidence of the pixel in the first image mask belongs to a preset low-confidence interval, if both of the above determinations are true, determining that the pixel belongs to the low-confidence image region, otherwise, determining that the pixel does not belong to the low-confidence image region;

[0017] Detect whether the pixel point belongs to the low-confidence image area. When it belongs to the low-confidence image area, determine the confidence of the pixel point from the second image mask; otherwise, determine the confidence of the pixel point from the first image mask as the confidence of the pixel point in the effective image mask.

[0018] In a further embodiment, the step of separating the foreground image and the background image from the current image frame according to the effective image mask comprises the following steps:

[0019] Applying an optimization algorithm to optimize the effective image mask to obtain an optimized image mask, wherein the optimization algorithm includes any one or more of an opening operation algorithm, a closing operation algorithm, and a Gaussian blur algorithm;

[0020] performing binarization processing on the optimized image mask to obtain a binarized mask;

[0021] An image extraction operation is performed on the current image frame according to the binary mask to obtain a foreground image and a background image in the current image frame.

[0022] In a further embodiment, after the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included:

[0023] Get special effect image frame;

[0024] Inserting the special effect image frame between the foreground image and the background image to synthesize an updated image frame;

[0025] The updated image frame replaces the current image frame in the live video stream and is pushed to the live broadcast room.

[0026] In a further embodiment, after the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included:

[0027] Blurring the background image or replacing it with a virtual image to obtain an updated background image;

[0028] Combining the foreground image and the updated background image into an updated image frame;

[0029] The updated image frame replaces the current image frame in the live video stream and is pushed to the live broadcast room.

[0030] In a further embodiment, after the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included:

[0031] Detecting whether there is a face image in the foreground image, and if there is a face image, obtaining a digital human image corresponding to the foreground image;

[0032] replacing the foreground image with the digital human image and synthesizing it with the background image to form an updated image frame;

[0033] The updated image frame replaces the current image frame in the live video stream and is pushed to the live broadcast room.

[0034] On the other hand, a video image segmentation device provided to meet one of the purposes of the present application includes a mask acquisition module, a confidence comparison module, a mask reconstruction module and an image segmentation module, wherein the mask acquisition module is used to acquire a first image mask corresponding to a current image frame in a live video stream, the image mask including the confidence corresponding to each pixel in the image frame; the confidence comparison module is used to determine a difference change map based on the confidence of the corresponding pixel points of the first image mask and the second image mask corresponding to the previous image frame, the difference change map indicating whether the confidence of each pixel point changes between the current image frame and the previous image frame; the mask reconstruction module is used to determine a low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguish whether the pixel point belongs to the low-confidence image region, and select a confidence from the first image mask and the second image mask to construct an effective image mask of the current image frame; the image segmentation module is used to separate the foreground image and the background image from the current image frame based on the effective image mask.

[0035] In a further embodiment, the confidence comparison module includes: a first mask binarization submodule, used to binarize the first image mask and the second image mask respectively to obtain a first distinction map and a second distinction map; an element product submodule, used to obtain the XOR operation result between the corresponding pixel points of the first distinction map and the second distinction map to obtain a difference change map.

[0036] In a further embodiment, the mask reconstruction module includes: a region judgment submodule, which is used to quantitatively judge whether the pixel point belongs to the foreground based on the difference change map, and judge whether the confidence of the pixel point in the first image mask belongs to a preset low-confidence interval. When the above judgments are all true, it is determined that the pixel point belongs to the low-confidence image area, otherwise it does not belong to the low-confidence image area; a confidence determination submodule, which is used to detect whether the pixel point belongs to the low-confidence image area. When it belongs to the low-confidence image area, the confidence of the pixel point is determined from the second image mask, otherwise the confidence of the pixel point is determined from the first image mask as the confidence of the pixel point in the effective image mask.

[0037] In a further embodiment, the image segmentation module includes: a mask optimization submodule, which is used to apply an optimization algorithm to optimize the effective image mask to obtain an optimized image mask, and the optimization algorithm includes any one or any multiple of an opening operation algorithm, a closing operation algorithm, and a Gaussian blur algorithm; a second mask binarization submodule, which is used to binarize the optimized image mask to obtain a binarized mask; and an image extraction submodule, which is used to perform an image extraction operation on the current image frame according to the binarized mask to obtain the foreground image and background image in the current image frame.

[0038] In a further embodiment, the image segmentation module includes: a special effect acquisition submodule for acquiring a special effect image frame; a first image synthesis submodule for inserting the special effect image frame between the foreground image and the background image to synthesize an updated image frame; and a first image replacement submodule for replacing the current image frame in the live video stream with the updated image frame and pushing it to the live broadcast room.

[0039] In a further embodiment, the image segmentation module includes: a background processing submodule, which is used to blur the background image or replace it with a virtual image to obtain an updated background image; a second image synthesis submodule, which is used to synthesize the foreground image and the updated background image into an updated image frame; and a second image replacement submodule, which is used to replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0040] In a further embodiment, the image segmentation module includes: a face detection submodule, which is used to detect whether there is a face image in the foreground image, and when a face image exists, obtain a digital human image corresponding to the foreground image; a third image synthesis submodule, which is used to replace the foreground image with the digital human image and synthesize it with the background image into an updated image frame; a third image replacement submodule, which is used to replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0041] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the video image segmentation method described in the present application.

[0042] On the other hand, a computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the video image segmentation method in the form of computer-readable instructions. When the computer program is called and run by a computer, it executes the steps included in the method.

[0043] The technical solution of this application has many advantages, including but not limited to the following:

[0044] According to the first image mask corresponding to the current image frame in the live video stream and the confidence of the corresponding pixel points of the second image mask corresponding to the previous image frame, a difference change map is determined, and the low-confidence image area of ​​the current image frame is determined according to the difference change map and the first image mask. The confidence level is selected from the first image mask and the second image mask to distinguish whether the pixel point belongs to the low-confidence image area to construct the effective image mask of the current image frame. Based on this, the foreground image and the background image are separated from the current image frame. The present application can quickly and accurately determine the effective image mask for segmenting the image frame in the live video stream, thereby realizing the separation of the foreground and background in the image frame, and achieving efficient and accurate image segmentation effects with lightweight operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 This is an exemplary network architecture used for the live streaming service in the live streaming scenario of this application;

[0047] Figure 2 A flowchart of a typical embodiment of the video image segmentation method of the present application is shown;

[0048] Figure 3 A schematic diagram of an original image, a background image, and an effective image mask exemplified in the embodiment;

[0049] Figure 4 A schematic diagram of a process for constructing a confidence quantization table in an embodiment of the present application;

[0050] Figure 5 Schematic diagram of a process for constructing an effective image mask based on a first image mask and a second image mask in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a process for segmenting the foreground and background of a current image frame in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of a process for adding special effects to a live video stream in an embodiment of the present application;

[0053] Figure 8 This is a schematic diagram of a process for blurring the background of a live video stream in an embodiment of the present application;

[0054] Figure 9This is a flow chart of replacing a human face image in a live video stream with a digital human image in an embodiment of the present application;

[0055] Figure 10 This is a principle block diagram of the video image segmentation device of the present application;

[0056] Figure 11 This is a schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0057] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0058] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0060] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0061] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0062] It should be noted that the concept of "server" referred to in this application can similarly be expanded to apply to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0063] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0064] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0065] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0066] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0067] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0068] See also Figure 1The network architecture adopted in an exemplary application scenario of the present application includes a terminal device 80, a media server 81 and an application server 82. The application server 82 can be used to deploy a live broadcast service. The media server 81 or the terminal device 80 can run a computer program product programmed according to the live video gesture detection method of the present application, and the various steps of the method are implemented through the operation of the product to realize gesture recognition of characters in the live video stream. The terminal device 80 can be used by anchor users or audience users to log in to the live broadcast room supported by the live broadcast service. The anchor user can obtain the video recording through the camera unit in its terminal device 80 and submit it to the media server as a live video stream. The audience user can receive the live video stream pushed by the media server through its terminal device 80 and play it for display.

[0069] A video image segmentation method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the exemplary application scenario of the present application, it can be deployed and implemented in the server of an e-commerce platform. The method can be executed by accessing the interface opened after the computer program product is run and performing human-computer interaction with the process of the computer program product through a graphical user interface.

[0070] See also Figure 2 In one embodiment, the video image segmentation method of the present application includes the following steps:

[0071] Step S1100: Obtain a first image mask corresponding to a current image frame in a live video stream, wherein the image mask includes a confidence level corresponding to each pixel in the image frame;

[0072] When a live broadcast user is broadcasting live, a video stream is continuously uploaded to the terminal device in real time as the live video stream. For example, the camera device installed on the terminal device can directly shoot and generate the corresponding video stream, or the terminal device can receive the video stream generated by the camera device, or the terminal device can receive the video stream generated by an electronic product through a video capture card. In this way, the video stream can be obtained as the live video stream. The terminal device can be a laptop computer, desktop computer, tablet computer, smart phone, etc. The camera device can be a video camera, camera, etc. The computer device can be a laptop computer, desktop computer, tablet computer, smart phone, game console, TV, etc.

[0073] Each image frame in a live video stream can be used as the current image frame for image segmentation. To this end, the live video stream is first frame-segmented to obtain multiple image frames corresponding to the frame rate of the live video stream. For example, a live video stream with a frame rate of 60fps can be frame-segmented to obtain 60 image frames per second. Then, instance segmentation is performed on each image frame, and the currently processed image frame is used as the current image frame. The frame segmentation process can be flexibly implemented by those skilled in the art, such as open source interfaces, algorithms, plug-ins, programs, etc. suitable for video framing.

[0074] In one embodiment, a preset image segmentation model can be used to perform instance segmentation on each image frame, determine an image mask corresponding to the foreground image in the corresponding image frame, and use the determined image mask corresponding to the current image frame as the first image mask. Specifically, the image mask can be represented as a two-dimensional matrix array.

[0075] The image segmentation model is pre-trained to a convergence state and learns the ability to determine the image mask for segmenting the foreground image in the image frame. A deep learning model suitable for instance segmentation tasks, such as Mask RCNN, FCIS, MaskLab, PANet, etc., can be used as the image segmentation model. Since the training processes of Mask RCNN, FCIS, MaskLab, and PANet models are known to those skilled in the art, the training process will not be described in detail.

[0076] In one embodiment, a MaskRCNN pre-trained to convergence is used as the image segmentation model, and the current image frame is used as input. The CNN layer is used to extract the deep semantic information of the current image frame to obtain the corresponding semantic feature map. The semantic feature map is input into the RPN layer (region proposal network) to obtain the ROI (Region of Interest) belonging to the foreground of the current image frame. ROI pooling is used to convert all ROIs into a fixed size, and all ROIs are passed to the fully connected layer for regression and classification prediction to obtain the class (category) and box (Bounding box). Finally, the confidence of each pixel point in the current image frame is determined through the FCN branch to form the first image mask. The confidence level represents the reliability that the corresponding pixel point belongs to the foreground in the image frame. Specifically, the confidence value range is [0, 1]. It can be understood that the closer the confidence is to 1, the higher the reliability of the corresponding pixel belonging to the foreground in the image frame, and the lower the reliability of belonging to the background in the image frame. The closer the confidence is to 0, the lower the reliability of the corresponding pixel belonging to the foreground in the image frame, and the higher the reliability of belonging to the background in the image frame. Accordingly, a first threshold value can be preset. When the confidence value exceeds the first threshold value, it is determined that the pixel point corresponding to the confidence value belongs to the foreground in the image frame. Otherwise, it is determined that the pixel point corresponding to the confidence value belongs to the background in the image frame. The first threshold value can be set as needed by those skilled in the art based on the disclosure herein.

[0077] In a further embodiment, the preset image segmentation model can be optimized. Specifically, an adversarial network can be generated based on the image segmentation model to perform adversarial training, thereby optimizing the performance of the image segmentation model and improving the accuracy of the model after the adversarial training is completed. At this time, the image segmentation model has many parameters and a large amount of computation. Therefore, the image segmentation model can be used as a teacher model. At the same time, a student model with the same network structure but relatively fewer parameters is constructed based on the teacher model. Then, knowledge distillation training is performed on the student model and the teacher model, so that the prediction results output by the student model are as close as possible to the prediction results output by the teacher model, and the deviation is optimized. After the knowledge training is completed, a student model with model performance equivalent to that of the teacher model is obtained, so that the student model has better performance while having fewer parameters. At this point, the optimization is completed and the student model is used as the optimized image segmentation model. It can be understood that after optimization, the image segmentation model requires fewer computing resources, has high execution efficiency, and can accurately segment images. Given that the generation of adversarial networks, the implementation of adversarial training, the construction of student models, and the implementation of knowledge distillation training are all known to those skilled in the art, this part will not be described in detail.

[0078] Step S1200: Determine a difference change map based on the confidence levels of corresponding pixels of the first image mask and the second image mask corresponding to the previous image frame, wherein the difference change map indicates whether the confidence level of each pixel changes between the current image frame and the previous image frame.

[0079] Since each image frame in the live video stream will be processed as the current image frame in succession, the preset image segmentation model can be used to perform instance segmentation on the previous image frame of the current image frame, and the corresponding image mask determined can be used as the second image mask.

[0080] It can be understood that when the frame rate of the live video stream is higher, the change between frames in the live video stream is smaller, that is, the correlation between frames is stronger. In order to determine whether the confidence of each pixel corresponding to the position between the current image frame and the previous image frame has changed, in one embodiment, the difference in the confidence of each pixel corresponding to the same position between the current image frame and the previous image frame is calculated, and its absolute value is taken to obtain a difference change map. It is not difficult to understand that the smaller the value of the element in the difference change map, the smaller the change in the confidence of the pixel, and the larger the value, the larger the change in the confidence of the pixel. Based on this, a second threshold can be preset to measure the change. When the value corresponding to each element in the difference change map exceeds the second threshold, it is determined that the confidence of the pixel at the corresponding position has changed between the current image frame and the previous image frame. Otherwise, it is determined that the confidence of the pixel at the corresponding position has not changed between the current image frame and the previous image frame. The second threshold can be set as needed by those skilled in the art based on the disclosure herein.

[0081] In another embodiment, a binarization operation is performed based on whether each confidence level in the first image mask and the second image mask exceeds a first threshold value, so that each confidence level is set to a binary value of 1 or 0, and then the confidence levels of the pixels corresponding to the same position between the binarized first image mask and the second image mask are obtained and a bitwise XOR operation is performed to obtain a difference change map. It is not difficult to understand that a value of 1 for an element in the difference change map indicates that the confidence level of the pixel at the corresponding position has changed between the current image frame and the previous image frame, and a value of 0 indicates that the confidence level of the pixel at the corresponding position has not changed between the current image frame and the previous image frame.

[0082] Step S1300: determining a low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguishing whether a pixel belongs to the low-confidence image region, and selecting a confidence from the first image mask and the second image mask to construct a valid image mask of the current image frame;

[0083] On the one hand, based on the value of each element in the difference change map, it can be determined whether the confidence of each corresponding pixel has changed between the current image frame and the previous image frame. On the other hand, it can be understood that the confidence value range is [0,1]. When the confidence of a pixel is closer to 0 or 1, the higher the reliability of the pixel belonging to the background or foreground in the image frame, that is, the higher the accuracy of the confidence in the image mask determined by the image segmentation model, the higher the confidence. On the contrary, the more the confidence of a pixel deviates from 0 or 1, the lower the accuracy of the confidence in the image mask determined by the image segmentation model, the lower the confidence. Based on this, a low confidence interval can be preset. When the confidence of a pixel belongs to the low confidence interval, it means that the confidence of the pixel deviates from 0 or 1. The low confidence interval is [θ1, θ2], where 0θ1<θ2<1. Those skilled in the art can set it as needed according to the disclosure herein. For example, the low confidence interval is [0.15, 0.85].

[0084] According to the two aspects, when it is determined based on the value of the element in the difference change map that the confidence of the pixel point in the current image frame changes between the current image frame and the previous image frame, and when the confidence of the pixel point in the first image mask belongs to the low confidence interval, it is determined that the pixel point belongs to the low confidence image area, otherwise it does not belong to the low confidence area.

[0085] When a pixel point in the first image mask belongs to the low-confidence image area, it means that the confidence of the pixel point changes between the current image frame and the previous image frame. Moreover, the confidence of the pixel point of the current image frame determined by the image segmentation model has low accuracy. This confidence point should not be used to segment the current image frame. Instead, the confidence point of the pixel point in the second image mask can be used.

[0086] When a pixel point in the first image mask does not belong to the low-confidence image area, it means that the confidence of the pixel point has not changed between the current image frame and the previous image frame, or, it has changed but the confidence of the pixel point of the current image frame determined by the image segmentation model is highly accurate, and this confidence level can be selected to segment the current image frame.

[0087] According to the above, for each pixel in the current image frame, the corresponding confidence level is selected as the confidence level of the effective image mask.

[0088] Step S1400 : Separate a foreground image and a background image from the current image frame according to the valid image mask.

[0089] The confidence of each pixel in the effective image mask is bitwise multiplied with the eigenvalue of each pixel in the current image frame, so that the eigenvalue of the pixel in the foreground image in the current image frame remains unchanged or the eigenvalue of the edge contour pixel becomes smaller, that is, the display effect of the pixel is retained to a certain extent, and the eigenvalues ​​of the pixel in the background image are all 0, and the foreground image in the current image frame is obtained. The eigenvalue is obtained by extracting the image features corresponding to the pixel by the image segmentation model.

[0090] By setting the characteristic values ​​of the pixels belonging to the foreground image in the current image frame to 0, the background image in the current image frame can be obtained.

[0091] like Figure 3 As shown, 300 is the effective image mask, 301 is the original image, and 302 is the foreground image

[0092] It can be seen from the typical embodiments of the present application that the technical solution of the present application has many advantages, including but not limited to the following aspects:

[0093] According to the first image mask corresponding to the current image frame in the live video stream and the confidence of the corresponding pixel points of the second image mask corresponding to the previous image frame, a difference change map is determined, and the low-confidence image area of ​​the current image frame is determined according to the difference change map and the first image mask. The confidence level is selected from the first image mask and the second image mask to distinguish whether the pixel point belongs to the low-confidence image area to construct the effective image mask of the current image frame. Based on this, the foreground image and the background image are separated from the current image frame. The present application can quickly and accurately determine the effective image mask for segmenting the image frame in the live video stream, thereby realizing the separation of the foreground and background in the image frame, and achieving efficient and accurate image segmentation effects with lightweight operations.

[0094] See also Figure 4 In a further embodiment, step S1200, the step of determining a difference change map based on the confidence levels of corresponding pixels of the first image mask and the second image mask corresponding to the previous image frame, includes the following steps:

[0095] Step S1210: binarize the first image mask and the second image mask respectively to obtain a first distinction map and a second distinction map;

[0096] In order to facilitate the comparison of the difference between the confidence levels of corresponding pixels at the same position in the first image mask and the second image mask, the confidence levels are reasonably binarized, and the confidence levels in the first image mask and the second image mask that exceed the first threshold are set to 1, and the confidence levels that do not exceed the first threshold are set to 0, thereby obtaining the corresponding first and second distinction maps.

[0097] Step S1220 : Obtain an XOR operation result between corresponding pixels of the first distinction map and the second distinction map to obtain a difference change map.

[0098] A bitwise XOR operation is performed on the confidence levels of pixels corresponding to the same position in the first and second differentiation maps. Identical confidence levels are calculated as 0, while different confidence levels are calculated as 1, thereby obtaining a difference change map. It is understood that a value of 1 for an element in the difference change map indicates that the confidence level of the pixel at the corresponding position has changed between the current image frame and the previous image frame, while a value of 0 indicates that the confidence level of the pixel at the corresponding position has not changed between the current image frame and the previous image frame.

[0099] In this embodiment, an XOR operation is performed on the confidences of the corresponding pixels of the binarized first image mask and the second image mask to obtain a difference change map of the corresponding operation results. The execution is efficient and the effect is significant.

[0100] See also Figure 5 In a further embodiment, step S1300, the step of determining the low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguishing whether a pixel point belongs to the low-confidence image region, and selecting a confidence from the first image mask and the second image mask to construct a valid image mask of the current image frame, includes the following steps:

[0101] Step S1310: quantifying and determining whether the pixel belongs to the foreground based on the difference change map, and determining whether the confidence of the pixel in the first image mask belongs to a preset low-confidence interval. If both of the above determinations are true, the pixel is determined to belong to the low-confidence image region; otherwise, the pixel does not belong to the low-confidence image region.

[0102] On the one hand, based on the value of each element in the difference change map, it can be determined whether the confidence of each corresponding pixel has changed between the current image frame and the previous image frame. On the other hand, it can be understood that the confidence value range is [0,1]. When the confidence of a pixel is closer to 0 or 1, the higher the reliability of the pixel belonging to the background or foreground in the image frame, that is, the higher the accuracy of the confidence in the image mask determined by the image segmentation model, the higher the confidence. On the contrary, the more the confidence of a pixel deviates from 0 or 1, the lower the accuracy of the confidence in the image mask determined by the image segmentation model, the lower the confidence. Based on this, a low confidence interval can be preset. When the confidence of a pixel belongs to the low confidence interval, it means that the confidence of the pixel deviates from 0 or 1. The low confidence interval is [θ1, θ2], where 0θ1<θ2<1. Those skilled in the art can set it as needed according to the disclosure herein. For example, the low confidence interval is [0.15, 0.85].

[0103] According to the two aspects, when it is determined based on the value of the element in the difference change map that the confidence of the pixel point in the current image frame changes between the current image frame and the previous image frame, and when the confidence of the pixel point in the first image mask belongs to the low confidence interval, it is determined that the pixel point belongs to the low confidence image area, otherwise it does not belong to the low confidence image area.

[0104] Step S1320: Detect whether the pixel point belongs to the low-confidence image area. When it belongs to the low-confidence image area, determine the confidence of the pixel point from the second image mask; otherwise, determine the confidence of the pixel point from the first image mask as the confidence of the pixel point in the effective image mask.

[0105] When a pixel point in the first image mask belongs to the low-confidence image area, it means that the confidence of the pixel point changes between the current image frame and the previous image frame. Moreover, the confidence of the pixel point of the current image frame determined by the image segmentation model has low accuracy. This confidence point should not be used to segment the current image frame. Instead, the confidence point of the pixel point in the second image mask can be used to avoid flickering in the display of the segmented area.

[0106] When a pixel point in the first image mask does not belong to the low-confidence image area, it means that the confidence of the pixel point has not changed between the current image frame and the previous image frame, or, it has changed but the confidence of the pixel point of the current image frame determined by the image segmentation model is highly accurate, and this confidence level can be selected to segment the current image frame.

[0107] According to the above, for each pixel in the current image frame, the corresponding confidence level is selected as the confidence level of the effective image mask.

[0108] In this embodiment, a valid image mask is constructed by determining whether a pixel in the current image frame belongs to a low-confidence image region and selecting the corresponding confidence level of the pixel from the second image mask or the first image mask. This allows for rapid and accurate determination of whether the confidence level of a pixel in the current image frame is acceptable for selection, resulting in efficient execution.

[0109] See also Figure 6 In a further embodiment, step S1400, the step of separating the foreground image and the background image from the current image frame according to the effective image mask, includes the following steps:

[0110] Step S1410: Applying an optimization algorithm to optimize the effective image mask to obtain an optimized image mask, wherein the optimization algorithm includes any one or more of an opening operation algorithm, a closing operation algorithm, and a Gaussian blur algorithm;

[0111] The closing algorithm generally eliminates narrow discontinuities and long thin gaps in the target area, ie, the foreground portion of the effective image mask, eliminates small holes, and fills breaks in edge contours.

[0112] The opening operation algorithm is opposite to the closing operation algorithm, so it will not be discussed in detail.

[0113] The Gaussian blur algorithm uses Gaussian blur to smooth the effective image mask and remove the influence of interfering noise on subsequent image processing.

[0114] Step S1420: binarize the optimized image mask to obtain a binarized mask;

[0115] The confidence levels of the optimized image mask exceeding the first threshold are set to 1, and the confidence levels of the optimized image mask not exceeding the first threshold are set to 0, thereby obtaining a binary mask.

[0116] Step S1430 : performing an image extraction operation on the current image frame according to the binary mask to obtain a foreground image and a background image in the current image frame.

[0117] The confidence of each pixel in the binary mask is bitwise multiplied with the eigenvalue of each pixel in the current image frame, so that the eigenvalue of the pixel in the foreground image of the current image frame remains unchanged or the eigenvalue of the edge contour pixel becomes smaller, that is, the display effect of the pixel is retained to a certain extent, and the eigenvalue of the pixel in the background image is 0, and the foreground image in the current image frame is obtained. The eigenvalue is obtained by extracting the image features corresponding to the pixel by the image segmentation model.

[0118] By setting the characteristic values ​​of the pixels belonging to the foreground image in the current image frame to 0, the background image in the current image frame can be obtained.

[0119] In this embodiment, the optimized image mask obtained by optimizing the effective image mask is binarized, and the foreground image and the background image in the current image frame are separated according to the obtained binarized mask.

[0120] See also Figure 7 In a further embodiment, after step S1400, the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included:

[0121] Step S1500: obtaining a special effect image frame;

[0122] In live streaming scenarios, the live streaming platform can provide a special effects feature, allowing users to add special effects to their live video streams. When the user uses this feature, a special effects image frame corresponding to the special effect selected by the user is generated based on the size of the image frame of the live video stream. The image frame contains no pixels other than those displaying the special effect, and the non-pixel areas appear transparent.

[0123] Step S1510: inserting the special effect image frame between the foreground image and the background image to synthesize an updated image frame;

[0124] The pixel points of the display features in the special effect image frame in the updated image frame are covered on the pixel points at the same position of the background image and displayed. At the same time, the pixel points of the display features may be covered by the pixel points at the same position of the foreground image and cannot be displayed.

[0125] Step S1520: Replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0126] Usually, the live broadcast platform provides a live video stream push service. After receiving the replaced live video stream uploaded by the live broadcast user, the server of the live broadcast platform pushes the live video stream to the live broadcast user's live broadcast room, loads the live video stream in the live broadcast room and plays it for the audience users who enter the live broadcast room to watch.

[0127] In this embodiment, by inserting the special effect image frame between the foreground image and the background image of the current image frame to synthesize an updated image, which replaces the current image frame in the live video stream and is pushed to the live broadcast room, special effects can be added to the live video stream simply and efficiently, thereby improving the user experience.

[0128] See also Figure 8 In a further embodiment, after step S1400, the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included:

[0129] Step S1600: blurring the background image or replacing it with a virtual image to obtain an updated background image;

[0130] In live streaming scenarios, the live streaming platform can provide a background processing function, which allows users to process the background of their live video stream. When the background processing function is used by the live streaming user, in one embodiment, the background image can be blurred using Gaussian blur, so that the visual effect of the processed background image is as if the image is viewed through a semi-transparent screen.

[0131] In another implementation, the background image can be replaced with a virtual image. The virtual image can be any image provided by the live streaming platform, or a user can upload an image to the live streaming platform's server and receive it. The virtual image must be the same size as the background image; if they are different, the virtual image can be cropped accordingly.

[0132] Step S1610: synthesize the foreground image and the updated background image into an updated image frame;

[0133] The foreground image is placed at the back and the updated background image is placed at the front, and the two are superimposed to synthesize an updated image frame.

[0134] Step S1620: Replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0135] Usually, the live broadcast platform provides a live video stream push service. After receiving the replaced live video stream uploaded by the live broadcast user, the server of the live broadcast platform pushes the live video stream to the live broadcast user's live broadcast room, loads the live video stream in the live broadcast room and plays it for the audience users who enter the live broadcast room to watch.

[0136] In this embodiment, an updated image is obtained by blurring the background image or replacing the background image with a virtual image, which is used to replace the current image frame in the live video stream and pushed to the live broadcast room. This can make the background in the live video stream present a special visual effect, improve the user experience, and be efficient and convenient.

[0137] See also Figure 9 In a further embodiment, after step S1400, the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included:

[0138] Step S1700: Detect whether there is a face image in the foreground image, and if there is a face image, obtain a digital human image corresponding to the foreground image;

[0139] You can use an open-source face detection model that has been pre-trained to convergence for object detection tasks to learn the ability to detect faces in images. Options include SSD, YOLO series, R-CNN family, RetinaNet, CornerNet, and others. Since the training processes of these models are well known to those skilled in the art, we will not describe them in detail.

[0140] In live streaming scenarios, the live streaming platform can provide a face processing function, which allows live streaming users to process faces in their live video streams. When the face processing function is used by the live streaming user, the face detection model is used to perform target detection on the foreground image, using the face as the target object. The presence of the target object in the foreground image is detected. If the target object is present, the corresponding face image in the foreground image is replaced with a digital image to obtain a digital human image. The digital image can be provided by the live streaming platform and can be an animated human face, a pet face, etc.

[0141] Step S1710: replacing the foreground image with the digital human image and synthesizing it with the background image to form an updated image frame;

[0142] The replaced foreground image is placed at the back and the updated background image is placed at the front, and the two are superimposed to synthesize an updated image frame.

[0143] Step S1720: Replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0144] Usually, the live broadcast platform provides a live video stream push service. After receiving the replaced live video stream uploaded by the live broadcast user, the server of the live broadcast platform pushes the live video stream to the live broadcast user's live broadcast room, loads the live video stream in the live broadcast room and plays it for the audience users who enter the live broadcast room to watch.

[0145] In this embodiment, the facial image in the foreground image is detected to obtain the corresponding digital human image, and the foreground image and background image are replaced by the digital human image to synthesize and update the image frame, which replaces the current image frame in the live video stream and is pushed to the live broadcast room. This can make the faces in the live video stream present a special visual effect, improve the user experience, and be efficient and convenient.

[0146] See also Figure 10, a video image segmentation device provided to meet one of the purposes of the present application is a functional embodiment of the video image segmentation method of the present application, the device includes a mask acquisition module 1100, a confidence comparison module 1200, a mask reconstruction module 1300 and an image segmentation module 1400, wherein the mask acquisition module 1100 is used to obtain a first image mask corresponding to a current image frame in a live video stream, the image mask including the confidence corresponding to each pixel in the image frame; the confidence comparison module 1200 is used to compare the first image mask with the corresponding pixel of the second image mask corresponding to the previous image frame; The confidence level determines a difference change map, and the difference change map represents whether the confidence level of each pixel point changes between the current image frame and the previous image frame; a mask reconstruction module 1300 is used to determine the low-confidence image area of ​​the current image frame based on the difference change map and the first image mask, distinguish whether the pixel point belongs to the low-confidence image area, and select the confidence level from the first image mask and the second image mask to construct a valid image mask of the current image frame; an image segmentation module 1400 is used to separate the foreground image and the background image from the current image frame based on the valid image mask.

[0147] In a further embodiment, the confidence comparison module 1200 includes: a first mask binarization submodule, used to binarize the first image mask and the second image mask respectively to obtain a first distinction map and a second distinction map; an element product submodule, used to obtain the XOR operation result between the corresponding pixel points of the first distinction map and the second distinction map to obtain a difference change map.

[0148] In a further embodiment, the mask reconstruction module 1300 includes: a region judgment submodule, used to quantitatively judge whether the pixel point belongs to the foreground based on the difference change map, and judge whether the confidence of the pixel point in the first image mask belongs to a preset low confidence interval. When the above judgments are all true, it is determined that the pixel point belongs to the low confidence image area, otherwise it does not belong to the low confidence image area; a confidence determination submodule, used to detect whether the pixel point belongs to the low confidence image area. When it belongs to the low confidence image area, the confidence of the pixel point is determined from the second image mask, otherwise the confidence of the pixel point is determined from the first image mask as the confidence of the pixel point in the effective image mask.

[0149] In a further embodiment, the image segmentation module 1400 includes: a mask optimization submodule, which is used to apply an optimization algorithm to optimize the effective image mask to obtain an optimized image mask, and the optimization algorithm includes any one or any multiple of an opening operation algorithm, a closing operation algorithm, and a Gaussian blur algorithm; a second mask binarization submodule, which is used to binarize the optimized image mask to obtain a binarized mask; and an image extraction submodule, which is used to perform an image extraction operation on the current image frame according to the binarized mask to obtain the foreground image and background image in the current image frame.

[0150] In a further embodiment, after the image segmentation module 1400, it includes: a special effect acquisition submodule, used to obtain a special effect image frame; a first image synthesis submodule, used to insert the special effect image frame between the foreground image and the background image to synthesize an updated image frame; a first image replacement submodule, used to replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0151] In a further embodiment, after the image segmentation module 1400, it includes: a background processing submodule, used to blur the background image or replace it with a virtual image to obtain an updated background image; a second image synthesis submodule, used to synthesize the foreground image and the updated background image into an updated image frame; a second image replacement submodule, used to replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0152] In a further embodiment, the image segmentation module 1400 includes: a face detection submodule, which is used to detect whether there is a face image in the foreground image, and when a face image exists, obtain a digital human image corresponding to the foreground image; a third image synthesis submodule, which is used to replace the foreground image with the digital human image and synthesize it with the background image into an updated image frame; and a third image replacement submodule, which is used to replace the current image frame in the live video stream with the updated image frame and push it to the live broadcast room.

[0153] In order to solve the above technical problems, the embodiment of the present application also provides a computer device. Figure 11As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence, and when the computer-readable instructions are executed by the processor, the processor may implement a video image segmentation method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the video image segmentation method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0154] In this embodiment, the processor is used to execute Figure 10 The memory stores the program code and various data required to execute the specific functions of each module and its submodule in the video image segmentation device. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the video image segmentation device of the present application, and the server can call the server's program code and data to execute the functions of all submodules.

[0155] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the video image segmentation method of any embodiment of the present application.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0157] In summary, the present application can quickly and accurately determine the effective image mask for segmenting image frames in live video streams, thereby realizing the separation of foreground and background in image frames, and achieving efficient and accurate image segmentation effects with lightweight calculations.

[0158] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted.

[0159] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video image segmentation method, characterized in that: include: Obtaining a first image mask corresponding to a current image frame in a live video stream, wherein the image mask includes a confidence level corresponding to each pixel in the image frame; determining a difference change map based on the confidence levels of corresponding pixels of the first image mask and the second image mask corresponding to the previous image frame, wherein the difference change map indicates whether the confidence level of each pixel changes between the current image frame and the previous image frame; determining a low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguishing whether a pixel point belongs to the low-confidence image region, and selecting a confidence from the first image mask and the second image mask to construct a valid image mask of the current image frame; A foreground image and a background image are separated from the current image frame according to the valid image mask.

2. The video image segmentation method according to claim 1, characterized in that: The step of determining the difference change map based on the confidence of the corresponding pixel points of the first image mask and the second image mask corresponding to the previous image frame includes the following steps: Binarizing the first image mask and the second image mask respectively to obtain a first distinction map and a second distinction map; An exclusive-OR operation result is obtained between corresponding pixels of the first distinction map and the second distinction map to obtain a difference change map.

3. The video image segmentation method according to claim 1, wherein: The step of determining the low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguishing whether a pixel point belongs to the low-confidence image region, and selecting a confidence from the first image mask and the second image mask to construct a valid image mask of the current image frame includes the following steps: Quantitatively determining whether the pixel belongs to the foreground based on the difference change map, and determining whether the confidence of the pixel in the first image mask belongs to a preset low-confidence interval, if both of the above determinations are true, determining that the pixel belongs to the low-confidence image region, otherwise, determining that the pixel does not belong to the low-confidence image region; Detect whether the pixel point belongs to the low-confidence image area. When it belongs to the low-confidence image area, determine the confidence of the pixel point from the second image mask; otherwise, determine the confidence of the pixel point from the first image mask as the confidence of the pixel point in the effective image mask.

4. The video image segmentation method according to claim 1, wherein: The step of separating the foreground image and the background image from the current image frame according to the effective image mask includes the following steps: Applying an optimization algorithm to optimize the effective image mask to obtain an optimized image mask, wherein the optimization algorithm includes any one or more of an opening operation algorithm, a closing operation algorithm, and a Gaussian blur algorithm; performing binarization processing on the optimized image mask to obtain a binarized mask; An image extraction operation is performed on the current image frame according to the binary mask to obtain a foreground image and a background image in the current image frame.

5. The video image segmentation method according to any one of claims 1 to 4, characterized in that: After the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included: Get special effect image frame; Inserting the special effect image frame between the foreground image and the background image to synthesize an updated image frame; The updated image frame replaces the current image frame in the live video stream and is pushed to the live broadcast room.

6. The video image segmentation method according to any one of claims 1 to 4, characterized in that: After the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included: Blurring the background image or replacing it with a virtual image to obtain an updated background image; Combining the foreground image and the updated background image into an updated image frame; The updated image frame replaces the current image frame in the live video stream and is pushed to the live broadcast room.

7. The video image segmentation method according to any one of claims 1 to 4, characterized in that: After the step of separating the foreground image and the background image from the current image frame according to the effective image mask, the following steps are included: Detecting whether there is a face image in the foreground image, and if there is a face image, obtaining a digital human image corresponding to the foreground image; replacing the foreground image with the digital human image and synthesizing the digital human image with the background image to form an updated image frame; The updated image frame replaces the current image frame in the live video stream and is pushed to the live broadcast room.

8. A video image segmentation device, characterized in that: include: A mask acquisition module is used to acquire a first image mask corresponding to a current image frame in a live video stream, wherein the image mask includes a confidence level corresponding to each pixel in the image frame; a confidence comparison module, configured to determine a difference change map based on the confidences of corresponding pixels of the first image mask and the second image mask corresponding to the previous image frame, wherein the difference change map indicates whether the confidence of each pixel has changed between the current image frame and the previous image frame; a mask reconstruction module, configured to determine a low-confidence image region of the current image frame based on the difference change map and the first image mask, distinguish whether a pixel point belongs to the low-confidence image region, and select a confidence level from the first image mask and the second image mask to construct a valid image mask of the current image frame; An image segmentation module is configured to separate a foreground image and a background image from the current image frame according to the effective image mask.

9. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images