Live broadcast stream image segmentation method and device, equipment and medium

Abstract

The invention relates to the field of network live broadcast, and discloses a live broadcast stream image segmentation method and device, equipment and a medium, and the method comprises the steps: sequentially calling image frames from an image sequence of a live broadcast stream as current image frames; a depth estimation segmentation model is called, a segmented figure mask corresponding to the figure image is determined based on the current image frame, and the depth estimation segmentation model is supervised and trained by using scene depth information; performing inter-frame smoothing processing on the segmented figure mask of the current image frame by using the practical figure mask iteration of the historical image frame to obtain a practical figure mask of the current image frame; and determining a figure image in the current image frame according to the practical figure mask of the current image frame. By fusing the depth information and the inter-frame smoothing technology, real-time, accurate and stable live broadcast stream image segmentation is realized, the live broadcast watching experience is improved, and the industry technology progress is promoted.

Description

Technical Field

[0001] The present application relates to network live broadcast technology, and in particular to a live stream image segmentation method and its device, equipment, and medium. Background Art

[0002] When image segmentation technology is applied in the field of live webcasting, it can segment human images from video streams in real time for subsequent image processing tasks such as background replacement or human decoration. However, the real-time, stability and accuracy requirements of live broadcasting scenes pose challenges to image segmentation technology.

[0003] Image segmentation for live streaming usually needs to be performed quickly and accurately on the client device. To achieve this goal, a variety of methods have been developed in the industry. Early methods relied on traditional image processing techniques such as threshold segmentation, edge detection, or region growing algorithms. Although these methods are effective in some cases, they are often computationally intensive and cannot meet the strict real-time requirements of live streaming scenarios.

[0004] With the development of deep learning technology, methods based on convolutional neural networks (CNNs) have made revolutionary progress in the field of image segmentation. These methods are able to learn complex features of images and achieve high-precision segmentation. However, these methods usually require a lot of computing resources and are not suitable for running on resource-constrained devices. In addition, they mainly focus on the processing of single-frame images and do not make full use of the temporal information between video frames, resulting in incoherent segmentation results in the video stream.

[0005] To solve these problems, the industry has tried to optimize deep learning models to meet the needs of real-time applications. For example, by simplifying the network structure or compressing the model to reduce the amount of calculation. Although these efforts have improved the segmentation speed to a certain extent, they often come at the expense of segmentation accuracy.

[0006] In addition, the dynamic nature of live video brings additional challenges to image segmentation. The rapid movement of people or changes in the background may cause the segmentation results to jitter or flicker, affecting the audience's viewing experience. In order to improve the stability of the segmentation results, some methods try to introduce timing information in the segmentation process. However, these methods often require complex algorithms and additional computational overhead.

[0007] In general, existing live stream image segmentation technologies have difficulty in achieving a balance between real-time performance, stability, and accuracy. A new technical solution is needed that can achieve image segmentation quickly and accurately on resource-constrained devices with low system overhead while maintaining the stability and consistency of the segmentation results. This is exactly the technical challenge and demand currently faced by the field of live stream image segmentation. Summary of the invention

[0008] The purpose of the present application is to solve the above-mentioned problems and to provide a live streaming image segmentation method and its corresponding device, equipment, and non-volatile readable storage medium.

[0009] According to one aspect of the present application, a live streaming image segmentation method is provided, comprising the following steps:

[0010] Call image frames from the image sequence of the live stream in sequence as the current image frame;

[0011] Calling a depth estimation segmentation model to determine a segmented person mask corresponding to the person image in the current image frame based on the current image frame, wherein the depth estimation segmentation model has been supervised and trained using scene depth information;

[0012] Iteratively smooth the segmented person mask of the current image frame using the practical person mask of the historical image frame to obtain the practical person mask of the current image frame;

[0013] According to the practical person mask of the current image frame, a person image in the current image frame is determined.

[0014] According to another aspect of the present application, a live stream image segmentation device is provided, comprising:

[0015] An image calling module is configured to call image frames in sequence from an image sequence of a live stream as current image frames;

[0016] An intelligent segmentation module is configured to call a depth estimation segmentation model to determine a segmented person mask corresponding to a person image in a current image frame, wherein the depth estimation segmentation model has been supervised and trained using scene depth information;

[0017] An inter-frame smoothing module is configured to perform inter-frame smoothing processing on the segmented person mask of the current image frame by using the practical person mask of the historical image frame to iteratively obtain the practical person mask of the current image frame;

[0018] The portrait determination module is configured to determine the person image in the current image frame according to the practical person mask of the current image frame.

[0019] According to another aspect of the present application, a live stream image segmentation device is provided, comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the live stream image segmentation method described in the present application.

[0020] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the live stream image segmentation method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0021] The technical solution of the present application shows significant technical advantages in the field of live streaming image segmentation. By incorporating scene depth information when training the depth estimation segmentation model, the model can not only more accurately identify and segment the character images in the live streaming image frames, but also reduce the demand for computing resources while maintaining efficient computing speed, and achieve real-time segmentation. In addition, the present application also adopts inter-frame smoothing technology, using practical character masks in historical image frames to smooth the segmentation results of the current frame, effectively reducing the jitter and flicker of the segmentation edge, enhancing the coherence between video frames, and further improving the segmentation quality through iterative processing. Importantly, while improving the segmentation accuracy and stability, the present application does not sacrifice real-time performance, because the computational complexity of inter-frame iterative smoothing is small, which will not affect the overall segmentation speed. It is suitable for live broadcast scenes with high real-time requirements and meets the needs of professional-level image processing applications. By combining deep learning and inter-frame smoothing technology, the present application significantly improves the stability and accuracy of segmentation, provides viewers with a smoother and higher-quality live viewing experience, and promotes technological innovation in the live broadcast industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an exemplary network architecture suitable for applying the live streaming image segmentation method of the present application;

[0023] Figure 2 A schematic diagram of a flow chart of an embodiment of a live stream image segmentation method of the present application;

[0024] Figure 3 This is a principle block diagram of the live stream image segmentation device of the present application;

[0025] Figure 4 This is a structural diagram of a live stream image segmentation device used in this application. DETAILED DESCRIPTION

[0026] To facilitate understanding of various embodiments of the live streaming image segmentation method of the present application, an exemplary network architecture is first introduced. Figure 1 As shown, the exemplary network architecture includes multiple computer devices, including multiple servers and multiple terminal devices. Through the interaction between these devices, the network live broadcast service can be ensured, and information transmission can be achieved by transmitting live streams, specifically including image streams and audio streams. For example, a live server 83 and a media server 85 are deployed in the network architecture, and at the same time, the terminal device 90 of the anchor user and the terminal device 92 of the audience user are allowed to access the network architecture to communicate with each server.

[0027] The media server 85 is responsible for processing the media data of the live broadcast room of the network live broadcast, mainly processing the live stream. In one embodiment, the media server 85 receives the live stream generated by the anchor user, decodes the live stream, and implements various technical processing in the decoding space, including mixing, editing, analyzing and processing the image stream in the live stream according to the live stream image segmentation method of the present application to determine the character image in each image frame, etc. Then, the media server 85 corresponds to each terminal device 90, 92 in the live broadcast room, encodes the processed live stream and transmits it to each audience user for playback.

[0028] It should be noted that, in addition to being applied in the media server 85, the live stream image segmentation method of the present application can also be applied in the terminal devices 90 and 92 of the anchor user and the audience user. Specifically, in one embodiment, the anchor user's terminal device 90 can analyze and process the image stream in the live stream generated in its terminal device according to the live stream image segmentation method of the present application to realize the recognition of the character image. In another embodiment, the terminal device 92 of the audience user can also execute the live stream image segmentation method of the present application, analyze and process the image stream in the live stream it receives, and obtain the corresponding character image.

[0029] The live broadcast server 83 is a bridge that maintains the operation of the live broadcast room to connect the anchor user and the audience user. To this end, the live broadcast server 83 is not only responsible for providing the users in the live broadcast room with the address for pulling the live broadcast stream from the media server 85, but also handles interactive functions related to the live broadcast, such as barrage, gifts and comments, thereby enhancing the interactivity of the live broadcast and the audience's sense of participation.

[0030] In such a network architecture, a computer program product that implements the live streaming image segmentation method of the present application can be flexibly deployed in different network nodes for operation, thereby providing a corresponding live streaming image segmentation system. For anchor users and audience users, the program product can be integrated into a network live broadcast program and deployed in their terminal devices 90, 92, such as smart phones, tablet computers, personal computers or game consoles. On the terminal devices 90, 92, the program can analyze and process the image stream in real time to obtain the character images therein. In addition, the computer program product can also be directly deployed in a media server 85 for centralized processing of the image stream in the live streaming. At the media server 85 end, the program can concurrently process a large number of live streamings from different anchors, and identify the character images for the image streams in the live streaming of each anchor user.

[0031] From the above description of the network architecture, it can be seen that the method of the present application can be implemented on the terminal devices 90, 92 or the media server 85, providing a high degree of flexibility and a wide range of application potentials.

[0032] In a live broadcast scenario, after identifying the character images in the image stream using the live broadcast stream image segmentation method of the present application, it can be applied to a variety of live broadcast application embodiments to enhance the live broadcast experience and interactivity.

[0033] In one embodiment, the background in the live video can be replaced in real time. When the anchor is broadcasting live, after the character image in the live stream is identified by this method, the original background and the character image in the corresponding image frame can be separated and processed, and then the original background can be automatically replaced with a virtual background. The virtual background can be an image such as a dynamic graphic, different scenes, or a background customized by the audience. Such a function can add fun to the live broadcast and enable the anchor to provide a richer visual experience without being restricted by the actual environment.

[0034] In another embodiment, the character image in each image frame of the live video can be beautified. After the character image is identified, a beauty filter or special effects, such as smooth skin and enlarged eyes, can be applied to beautify the character image part, thereby improving the appearance of the anchor in real time. This not only improves the viewing experience of the live broadcast, but also enables the anchor to appear in front of the audience with a more confident image.

[0035] In another embodiment, it can be used to realize the superposition of character images and interactive elements. For example, during the live broadcast, in response to the special effect adding instruction, the character image in the image frame can be identified and virtual items such as hats, glasses or decorations can be accurately superimposed on the anchor to obtain an optimized image frame, adding interactivity and entertainment to the live broadcast. The audience can trigger these special effect adding instructions by sending gifts or participating in voting, thereby interacting more closely with the anchor.

[0036] In another embodiment, character images can also be used for safety and content supervision in live broadcasts. By real-time monitoring of the behavior patterns of character images in continuous image frames in the live stream, inappropriate content such as violence, nudity or sensitive symbols can be identified and filtered. When the corresponding behavior pattern is an illegal pattern, permission control can be implemented on the live broadcast activities of the anchor user, such as stopping the live broadcast, thereby ensuring the compliance of the live broadcast content.

[0037] Through these embodiments, it can be seen that the live stream image segmentation method of the present application brings wide application potential to the live broadcast field, which not only improves the technical content of live broadcast, but also provides viewers with a richer and more interactive viewing experience.

[0038] See also Figure 2 According to a live streaming image segmentation method provided by the present application, the method can be implemented as a computer program product and installed and run in a computer device such as a media server or a terminal device. In some embodiments thereof, the method includes the following steps:

[0039] Step S3100: sequentially calling image frames from the image sequence of the live stream as the current image frame;

[0040] In this application, it is first necessary to obtain image frames from the image sequence of the live stream. Live stream refers to the video data stream transmitted in real time during the live broadcast on the Internet. It is usually composed of continuous image frames and is transmitted together with the audio stream. The image sequence refers to this series of continuous image frames, which are arranged in chronological order and constitute the dynamic picture of the live video.

[0041] An image frame refers to each static image in the live stream, which is a snapshot of a moment in the video stream. The current image frame refers to the image frame currently being processed in the image sequence. During the live broadcast process, image frames are generated in real time and transmitted continuously. In order to achieve image segmentation, these image frames need to be called in sequence.

[0042] Depending on the device where the computer program product of the present application is installed and runs, the source and processing of the live stream may be slightly different.

[0043] When the computer program product is deployed on a media server, the live stream usually comes from the anchor's terminal device, is encoded and transmitted to the server, and then pushed to the viewer by the server. In this process, the media server is responsible for processing the received live stream and sequentially calling image frames from it for segmentation processing.

[0044] If the computer program product is deployed on the anchor user's terminal device, the live stream comes directly from the camera or image acquisition module on the device. The terminal device will process the acquired image frames and also needs to call each frame in turn for segmentation.

[0045] For the terminal device of the audience user, the live stream is transmitted from the media server or the anchor terminal device. The computer program product at the audience end will receive these live streams and call the image frames therefrom in sequence for segmentation processing, so as to perform character image segmentation on the received video stream in real time.

[0046] In the process of implementing live stream image segmentation, sequential calls mean that the image frames in the live stream are processed continuously. Each time a frame of image is received, it is segmented and processed, and then the next frame is processed immediately. This process is continuous to ensure the real-time and smoothness of the live video.

[0047] Step S3200: calling a depth estimation segmentation model to determine a segmented person mask corresponding to the person image in the current image frame based on the current image frame, wherein the depth estimation segmentation model has been supervised and trained using scene depth information;

[0048] The depth estimation segmentation model of the present application is deployed on a computer device where a computer program product is located. The model can identify a person mask in an image input therein as a segmentation person mask in an end-to-end manner. After a current image frame is determined, the depth estimation segmentation model can be called, and the current image frame can be passed to the model, and the model determines the segmentation person mask corresponding to the person image in the current image frame.

[0049] Since the image frames in the live stream need to be segmented continuously, the depth estimation segmentation model needs to continuously use each image frame as the current image frame to perform image segmentation to obtain the corresponding segmented person mask. In some embodiments, a multi-tasking mechanism can be enabled, using multiple threads, each thread uses a single image frame in the continuous image frames in the image sequence as the current image frame, and independently calls the depth estimation segmentation model to perform image segmentation processing on the current image frame, thereby accelerating the acquisition of the segmented person mask of each image frame in the image sequence through a concurrent mechanism, and improving the image segmentation response speed.

[0050] The depth estimation segmentation model of this application is based on a convolutional neural network (CNN) to achieve efficient and accurate segmentation of human images in live streaming image frames. The model is specially designed and trained to run quickly on a variety of devices while maintaining high performance segmentation results.

[0051] In this application, scene depth information is used as a supervisory signal when training a depth estimation segmentation model, which means that the model not only considers the visual features of the image, but also the depth information of the objects in the image during training. By combining the depth information, the model can better understand the three-dimensional structure of the scene, thereby more accurately identifying the outline and position of the person during the segmentation process.

[0052] In order to meet the requirements of real-time processing, in some embodiments, the depth estimation segmentation model selects a lightweight network architecture, such as MobileNet or ShuffleNet, which uses depth-separable convolution to reduce the amount of calculation and the number of parameters. For example, MobileNet uses a depth-separable convolution structure to effectively reduce the complexity of the model while maintaining segmentation performance. ShuffleNet optimizes information flow through channel shuffling operations, thereby improving network efficiency.

[0053] In one embodiment, after the model training is completed, it can be embedded in a computer program product implemented according to the live stream image segmentation method of the present application, so as to be directly deployed in a computer device on which the computer program product runs. It is not difficult to understand that in a live broadcast scenario, the computer device can be a media server, a terminal device of an anchor user, or a terminal device of an audience user. When the model receives the current image frame in the live stream, it will perform forward propagation and output a segmented character mask corresponding to the character image in the current image frame.

[0054] In another embodiment, when the computer program product is deployed in a media server, the depth estimation segmentation model can also be decoupled from the computer program product and deployed in other cluster servers that the media server can call in real time, which can also ensure the real-time performance of image segmentation.

[0055] The segmentation person mask is a binary image of the same size as the input image, in which the person area is marked with one value (such as 1) and the background area is marked with another value (such as 0). In this way, the segmentation person mask clearly distinguishes the person and background in the image, providing accurate person area information for subsequent image processing tasks.

[0056] In order to further optimize the segmentation effect, the depth estimation segmentation model can adopt a multi-task learning strategy to optimize the segmentation loss and depth estimation loss simultaneously during the training process. In addition, it can also be combined with the optimization of the character boundary loss. This strategy enables the model to take into account the depth information of the scene during reasoning, improving the accuracy of segmentation.

[0057] In practical applications, this depth estimation segmentation model can process high-resolution image frames and quickly generate relatively accurate segmented character masks, meeting the strict real-time requirements of live broadcast scenarios.

[0058] Step S3300, using the practical person mask of the historical image frame to iteratively perform inter-frame smoothing processing on the segmented person mask of the current image frame to obtain the practical person mask of the current image frame;

[0059] After completing the acquisition of the segmented person mask corresponding to the current image frame, the practical person mask of the historical image frame can be used to perform inter-frame smoothing on the segmented person mask of the current image frame, so as to reduce the jitter and flicker of the segmentation result between image frames, thereby improving the smoothness and stability of the segmented image.

[0060] Inter-frame smoothing is performed in an iterative manner, where the segmented person mask of each image frame is affected by the mask of the previous image frame. Specifically, the system maintains one or more historical practical person masks that represent the segmentation results of the person image in the previous image frame. When processing the current frame, the system merges the current segmented person mask with these historical masks to achieve smoothing.

[0061] It should be pointed out that, since the segmented character masks corresponding to the historical image frames have been smoothed based on the principle of iteration, when the segmented character masks of the current image frame are smoothed using the historical image frames, the historical mask referenced is the practical character mask obtained after the segmented character masks of the historical image frames are smoothed. Also based on the principle of iteration, the historical image frame may refer to the previous image frame that is immediately before the current image frame in terms of time sequence, that is, the previous image frame of the current image frame. Based on this iterative mechanism, the practical character mask of the current image frame can also be regarded as a product obtained after smoothing the segmented character masks of all the image frames that are prior in terms of time sequence (also corresponding to the practical character masks) based on its segmented character mask.

[0062] The smoothing process can be implemented in many ways. One common way is to use a filter, such as a Kalman filter, a Gaussian filter, a bilateral filter, a median filter, etc.

[0063] Gaussian filters are known for their simple implementation and stable effects. They can effectively remove image noise and achieve smoothing effects by averaging pixels based on Gaussian weight distribution. This method is suitable for real-time processing because it has low computational complexity and can be quickly applied to consecutive video frames.

[0064] Bilateral filtering is a technique that can maintain edge sharpness while smoothing an image. It performs weighted averaging on pixels in the neighborhood of a central pixel by considering the spatial proximity of pixels and the similarity of pixel values. Although bilateral filtering is relatively computationally intensive, its advantage in maintaining edge information makes it an ideal choice for live video processing that requires clear edges.

[0065] The median filter processes the image by selecting the median value in the neighborhood as the current pixel value, which is particularly suitable for removing salt and pepper noise. The median filter does not rely on the statistical characteristics of the neighborhood pixel values, so it can well preserve edge features while removing noise, which is suitable for real-time video processing scenarios.

[0066] These filters can balance the information of the current frame and the historical image frames according to the predetermined smoothing parameters, and can achieve high real-time performance, have efficiency advantages, and are more suitable for deployment in terminal devices. For example, the Kalman filter predicts the person mask of the current frame based on the prediction parameters that act as smoothing parameters, including the prediction error covariance, the measurement noise covariance, the Kalman gain, etc., and updates these parameters based on the difference between the current frame and the prediction result.

[0067] In actual applications, the system first generates a predicted person mask through a filter, and then compares it with the current segmentation mask generated by the depth estimation segmentation model. Based on the difference between the two, the system adjusts the gain parameter of the filter so that the useful information in the person mask of the historical image frame is considered while retaining the details of the segmented person mask of the current image frame. In this way, the segmentation results can remain smooth and coherent even when the person moves quickly or the live broadcast environment changes.

[0068] In addition, in scenarios where computer devices have sufficient resources, such as when a media server applies the live stream image segmentation method of the present application, the inter-frame smoothing process can also apply the optical flow method, which is further optimized by considering the motion information between video frames. The optical flow method predicts the motion trajectory of the object by analyzing the pixel motion information between consecutive video frames, thereby achieving inter-frame smoothing. For example, the system can analyze the optical flow information between adjacent frames to estimate the movement of a person in the scene, and use this information to improve the effect of inter-frame smoothing.

[0069] Through the disclosure of the above exemplary inter-frame smoothing techniques, it can be known that these inter-frame smoothing techniques have their own characteristics and are suitable for different live broadcast scenarios. The Kalman filter is very suitable for use in live broadcast scenarios due to its real-time performance, computational efficiency, and dynamic adjustment capabilities. Gaussian filters are suitable for fast smoothing; bilateral filters are suitable for maintaining edge clarity while smoothing; median filters are suitable for removing random noise, especially in noisy environments; and optical flow methods are suitable for scenarios that consider motion information for smoothing. In practical applications, the appropriate inter-frame smoothing technology can be selected based on the specific needs of the live broadcast scenario and the available computing resources.

[0070] The practical person mask obtained by inter-frame smoothing will be used to determine the person image in the current image frame. This not only provides accurate segmentation results, but also ensures that the person image obtained by segmentation in the image stream has a high degree of stability and naturalness, meeting the needs of live broadcast scenarios.

[0071] Step S3400: Determine the person image in the current image frame according to the practical person mask of the current image frame.

[0072] After inter-frame smoothing, the practical person mask of the current image frame is ready. As a binary image, the practical person mask clearly marks the person area and the background area in the image. This practical person mask can be used to separate the person image and the background image in the image.

[0073] Each pixel value in the practical person mask represents whether the pixel belongs to the person or the background. Generally speaking, the person area will be marked with one value (such as 1) and the background area will be marked with another value (such as 0). This binarization process provides a clear dividing line for image segmentation.

[0074] With this practical person mask, you can use simple image processing techniques to separate the person from the background. For example, you can use the practical person mask to index the original image frame and extract the person area. In specific operations, you can traverse each pixel of the image, and for the pixel point whose value in the practical person mask represents the person, copy the pixel value of the corresponding position in the original image frame to a new image area, thereby constructing a complete person image.

[0075] On the contrary, if a background image is needed, the pixels in the practical person mask can be traversed, and when a pixel whose value indicates that it belongs to the background is encountered, the pixel value of the corresponding position in the original image frame is copied to the background image area. In this way, an image containing only the background can be obtained.

[0076] In practical applications, this process can be implemented by programming. For example, using an image processing library such as OpenCV, you can easily read the actual person mask and the current image frame, and then extract the person and background images through array indexing and conditional statements. In addition, in order to improve efficiency, bit operations can be used to speed up the mask extraction process.

[0077] In a live broadcast scenario, this process is done in real time. This means that as the live stream continues, each frame is processed to generate a continuous sequence of characters and background images. This allows the host to change the background in real time or add various visual effects to the live broadcast.

[0078] In addition, the extracted character images and background images can be used in the various live broadcast application embodiments disclosed above. For example, the character images can be used for beauty, filter effects or virtual props superposition, while the background images can be used for background replacement, scene fusion, etc.

[0079] In summary, by using the practical person mask of the current image frame, the image frames in the live stream can be effectively segmented into person images and background images, providing rich visual content and interactive experience for the live broadcast.

[0080] It is not difficult to understand from the above description of the embodiments that the technical solution of the present application can exhibit the following significant technical advantages:

[0081] First, by incorporating scene depth information into the training process of the depth estimation segmentation model, the application enables the model to more accurately identify and segment the human images in the image frames of the live stream and obtain the corresponding segmented human masks, while also maintaining a high computing speed during reasoning. This end-to-end segmentation method reduces the demand for computing resources, allowing the model to achieve real-time segmentation even on ordinary client devices.

[0082] Secondly, based on the segmented character mask of the current image frame obtained by the model, the present application uses the existing practical character masks in the historical image frames to perform inter-frame smoothing on the segmentation result of the current image frame to obtain the practical character mask of the current image frame, which effectively reduces the jitter and flicker of the segmentation edge and enhances the consistency of the segmentation results between video frames. The practical character mask of the current image frame can also continue to be used for iterative processing of the segmented character mask of the next image frame. Through iterative smoothing, the segmentation quality is further improved, making the segmented character image more natural and stable.

[0083] Subsequently, although the present application has achieved significant improvements in segmentation accuracy and stability, it has not been at the expense of real-time performance. The amount of computation required to achieve smoothing of the character mask by inter-frame iteration is relatively small, and will not affect the overall segmentation speed. Therefore, the present application is not only suitable for live broadcasting scenes with extremely high real-time requirements, but also can meet the needs of professional-level image processing applications.

[0084] In addition, this application combines deep learning and inter-frame smoothing technology to significantly improve the stability and accuracy of segmentation while ensuring real-time performance, providing viewers in the live broadcast room with a smoother and higher-quality live viewing experience. This technological advancement helps promote technological innovation in the live broadcast industry and bring richer visual enjoyment to viewers.

[0085] Based on any embodiment of the method of the present application, before the image frame is sequentially called from the image sequence of the live stream as the current image frame, the method includes:

[0086] Step S2100, iteratively inputting image samples in the training data set into the depth estimation segmentation model for reasoning, corresponding to each prediction branch preset for the backbone network of the depth estimation segmentation model, respectively predicting scene depth information, character boundary information, and character mask information;

[0087] The depth estimation segmentation model of the present application needs to be trained before it is put into use to ensure accuracy and efficiency in practical applications. To this end, a corresponding training data set can be prepared, and the image samples in the training data set can be traversed and called to implement iterative training, and the image samples in the training data set can be used to input the depth estimation segmentation model for reasoning.

[0088] The depth estimation segmentation model includes a backbone network followed by multiple prediction branches, which are responsible for predicting different aspects of the scene and predicting the corresponding results of the image samples from different aspects. Specifically, in this embodiment, a prediction branch is set to predict the scene depth information of the image sample, which helps the model understand the front and back distances of objects in the image. Another prediction branch focuses on predicting character boundary information, which is to identify the outline and shape of the character in the image. The last prediction branch is used to predict character mask information, which provides accurate segmentation of the character area.

[0089] During the inference process, the backbone network of the model extracts deep semantics from image samples. Based on the deep semantics, each prediction branch outputs prediction results, including scene depth map, character boundary map and character mask, which provide scene depth information, character boundary information and character mask information respectively. These prediction results will be compared with the annotation information of the image samples to calculate the loss value. The loss value is an indicator to measure the accuracy of the model prediction, which reflects the difference between the model prediction and the actual annotation.

[0090] Step S2200, supervising the training of the depth estimation and segmentation model until convergence using the scene depth information, character boundary information, and character mask information annotated for the image sample;

[0091] In order to improve the prediction ability of the depth estimation segmentation model during training, the loss value is needed to guide the adjustment of the model parameters. This process is called backpropagation, which updates the model's weights by calculating the gradient of the loss function, so that the model can make more accurate predictions during the next inference.

[0092] In the training data set, corresponding annotation information of each prediction branch of the model is provided for each image sample, including scene depth information, character boundary information and character mask information. These annotation information can be manually annotated in advance, or generated with the help of mature deep learning models. For example, the scene depth information can be predicted and generated by the depth estimation model for the image sample, while the character mask can be generated by segmentation using a known image segmentation model, and the character boundary information can be calculated and determined based on the scene depth information using a preset algorithm.

[0093] Various annotation information is used to supervise the output results of the corresponding prediction branches according to the corresponding relationship between information content, and to calculate the corresponding loss values ​​so that the model can be updated according to the loss values. Various types of loss functions can be used to calculate the corresponding loss values ​​of each prediction branch, such as mean square error loss, cross entropy loss, or mean absolute error loss. Choosing a suitable loss function is crucial to the performance of the model because it directly affects the learning efficiency of the model and the final prediction accuracy.

[0094] Through continuous iterative training, the prediction accuracy of the model will gradually improve until the performance of the model on the training data set reaches a satisfactory level, such as the overall loss value of each prediction branch is lower than the preset threshold, which indicates that the model has converged. The converged model will be able to accurately predict the scene depth information, character boundary information and character mask in the image, providing strong support for live streaming image segmentation.

[0095] Step S2300: retain the backbone network in the depth estimation segmentation model and the prediction branch corresponding to the character mask information, so as to determine the segmented character mask in the image frame.

[0096] After the training process, the depth estimation segmentation model can accurately predict the scene depth information, character boundary information and character mask information in the image. However, in actual use, usually only the character mask information is needed. Therefore, it is necessary to retain the prediction branch corresponding to the character mask and delete the other prediction branches corresponding to the character boundary and scene depth.

[0097] Specifically, this step only retains the backbone network of the depth estimation segmentation model and the prediction branch corresponding to the person mask information, and finally obtains a practical depth estimation segmentation model. The backbone network in the model is its core, which is responsible for extracting the deep features and semantic information of the image, providing a basis for the subsequent prediction branch. The person mask prediction branch is directly responsible for generating the final segmented person mask, which is the most critical output in live stream image segmentation.

[0098] The benefit of pruning other prediction branches is that it can also reduce the computational burden of computer equipment, improve processing speed, and enable the model to meet real-time requirements. At the same time, this also ensures that the model can focus on generating accurate character masks.

[0099] In addition, the retained model part can be integrated into various devices, including media servers, anchor user's terminal devices or audience user's terminal devices. In this way, real-time image segmentation processing can be effectively performed at both the live broadcast production end and the audience end.

[0100] This embodiment implements the training and optimization of the depth estimation segmentation model. This process first iterates the image samples in the reasoning training data set, and uses multiple prediction branches of the model to predict the scene depth, character boundaries, and character mask information respectively, to ensure that the model can understand the image content from multiple dimensions. Subsequently, each prediction branch is carefully supervised and trained using the corresponding annotated information until the model converges, thereby ensuring the accuracy and robustness of the model in practical applications. Finally, by retaining the backbone network and key prediction branches, not only the reasoning efficiency of the final depth estimation segmentation model is improved, but also the accuracy of the segmentation results is ensured.

[0101] More specifically, on the basis of using person mask information for supervised training, person boundary information and scene depth information are also used for supervised training in order to enhance the depth estimation segmentation model's ability to understand three-dimensional spatial scenes, thereby improving the model's accuracy and robustness in image segmentation tasks. Person boundary information helps the model learn how to identify and locate the outline of people in the image, and can effectively segment people even when the color of the person and the background are similar or the lighting conditions are complex. The scene depth information provides the model with the relative distance information of each object in the image, especially the front and back distance information, helping the model to better understand the three-dimensional structure of the scene and distinguish between foreground people and background. This multi-dimensional supervisory signal enables the model to learn features from different angles during training and improve its generalization ability.

[0102] By combining the three types of information for supervised training, the model can not only identify the position and shape of the person, but also understand the relative position of the person in the scene, especially when the person is intertwined or occluded by the background. This training method can significantly improve the model's spatial perception ability, thereby improving the quality of image segmentation, especially in real-time and dynamically changing video streams such as live streaming, which can achieve smoother and more natural segmentation effects, reduce segmentation errors and edge jitter, and provide clearer and more accurate person masks, thereby optimizing the overall performance of live streaming image segmentation.

[0103] It can be seen that the unique technical advantage achieved by this embodiment is that it can simultaneously process the depth information and semantic information of the image, improve the model's ability to understand complex scenes through a multi-task learning strategy, and optimize the model structure to make it more suitable for real-time processing of live streaming images, thus meeting the dual needs of high efficiency and high-precision segmentation in live broadcast scenarios.

[0104] On the basis of any embodiment of the method of the present application, the depth estimation and segmentation model is trained to convergence by correspondingly supervising the scene depth information, character boundary information, and character mask information annotated for the image sample, including:

[0105] Step S2210, corresponding to the result outputted from each prediction branch of the image sample, the branch loss value of each prediction branch is calculated using the scene depth information, the character boundary information and the character mask information annotated for the image sample;

[0106] In this embodiment, the training process of the depth estimation segmentation model provides a loss function corresponding to each prediction branch to optimize the model so that the model can eventually segment the characters in the live stream image more accurately. As mentioned above, the depth estimation segmentation model contains three prediction branches: portrait segmentation prediction branch, boundary detection prediction branch and depth estimation prediction branch. Each branch corresponds to a specific loss function, which is finally merged into an overall loss value to update the model.

[0107] The loss function of the portrait segmentation prediction branch uses Dice loss, which is particularly suitable for processing data with imbalanced categories. The Dice loss function evaluates the segmentation quality by calculating the ratio of the intersection and union between the predicted mask and the true label, and obtains the corresponding branch loss value L h , the formula is:

[0108]

[0109] Among them, m pi Represents the predicted person mask information, m gti Represents the segmentation label, that is, the person mask information pre-annotated for the corresponding image sample.

[0110] The boundary detection prediction branch uses the binary cross entropy loss function, which is a commonly used loss function for classification tasks and is particularly suitable for predicting pixel-level boundary information. The binary cross entropy loss function calculates the difference between the predicted edge and the true edge label to obtain the corresponding branch loss value L b , the formula is:

[0111]

[0112] where b pi represents the predicted character boundary information, b gti Represents the true edge label, that is, the person boundary information pre-labeled in the corresponding image sample.

[0113] The depth estimation prediction branch uses the L1 loss function to calculate the absolute error between the predicted depth information and the true depth information, and obtains the corresponding branch loss value L d , the formula is:

[0114]

[0115] where d pi Represents the predicted depth information, dgti Represents the real depth information, that is, the scene depth information pre-annotated for the corresponding image sample.

[0116] Step S2220: merge the branch loss values ​​of each prediction branch into an overall loss value, and perform gradient update on the depth estimation segmentation model according to the overall loss value.

[0117] In order to implement the parameter update of the depth estimation segmentation model, the branch loss values ​​corresponding to each prediction branch are further fused, and the three branch loss values ​​are fused into an overall loss value L, the formula is as follows:

[0118] L=L h +αL b +βL d

[0119] It can be seen from the formula that the branch loss values ​​of each prediction branch can be merged into an overall loss value by weighted merging, where α and β are the weights of the corresponding loss function terms, which can be flexibly set and are usually set to smaller values. It can be considered to be between [0.05, 0.30]. For example, in this embodiment, α = β = 0.1 can be preset. The weights are determined according to this interval. This combination allows the model to moderately consider the segmentation quality, edge accuracy, and depth accuracy during the training process, thereby achieving more comprehensive and accurate character image segmentation, which can improve the convergence efficiency of the model while ensuring the character mask segmentation effect.

[0120] This embodiment optimizes the depth estimation segmentation model by selecting corresponding loss functions for each prediction branch, improves the spatial perception ability of the model, and realizes efficient and accurate segmentation of people in live streaming images. The Dice loss function is used to optimize the portrait segmentation prediction branch, which effectively solves the category imbalance problem and improves the segmentation quality corresponding to the predicted character mask information; the binary cross entropy loss function is used to train the boundary detection prediction branch, which enhances the recognition ability of the model to predict the character boundary information; the L1 loss function is used to train the depth estimation prediction branch, which improves the prediction accuracy of the predicted scene depth information. By fusing these loss functions into an overall loss value and assigning flexible and adjustable weights, the model can balance the importance of segmentation, edge detection and depth estimation during training, making the final segmentation result more refined and realistic. This multi-task learning strategy not only improves the accuracy of segmentation, but also enhances the model's adaptability to complex scenes. Especially in live streaming image segmentation applications with high real-time requirements, it can achieve smooth and natural segmentation effects, significantly improving the live viewing experience.

[0121] Based on any embodiment of the method of the present application, before iteratively inputting image samples in the training data set into the depth estimation segmentation model for reasoning, it includes:

[0122] Step S1100, obtaining a basic data set, which includes a plurality of image samples and their corresponding person masks, wherein the person masks mark person mask information corresponding to an image region where a person image in the corresponding image sample is located;

[0123] The basic dataset provides raw materials for the preparation of training samples for the model, which contains a large number of image samples and a character mask corresponding to each image sample. The collected image samples cover a variety of scenes, lighting conditions, character postures, and backgrounds to ensure that the model can show good generalization capabilities under various live streaming conditions.

[0124] The person mask corresponding to each image sample is a binary image in which the person area is marked with one value (such as 0) and the non-person area is marked with another value (such as 1). The pre-annotated person masks are generated by accurate manual annotation of the original image or using existing automated segmentation tools, and they define the precise outlines and boundaries of the people in the image.

[0125] In one embodiment, the image samples in the basic dataset are closely related to the expected live broadcast scene. For example, if the live broadcast mainly involves indoor interviews, the images in the dataset should mainly reflect indoor environments and common indoor backgrounds. By ensuring the relevance of the dataset, the practicality and accuracy of the model in actual live broadcasts can be improved.

[0126] Step S1200: Generate a corresponding scene depth map for each image sample in the basic data set using a scene depth estimation model, wherein the scene depth map marks scene depth information of the image content in the corresponding image sample;

[0127] The role of the scene depth estimation model is to analyze each image sample in the basic data set and predict the depth of each pixel relative to the camera, generating a corresponding scene depth map, which is used to provide the scene depth information pre-annotated with the image sample. The scene depth information provides the depth information of the objects in the image in the depth direction, which helps to distinguish the foreground characters from the background.

[0128] A mature depth estimation model can be used to generate the corresponding scene depth map for the image sample. For example, the DepthAnything model is specifically designed to predict scene depth information from a single image. DepthAnything uses convolutional neural networks (CNN) to learn depth clues in images, can handle a variety of scenes, and generate high-quality depth maps. In addition to DepthAnything, there are other models such as Make3D, Microsoft's Depth360 or MIT's SingleImage Depth Prediction (SID) model that can also be used for this purpose.

[0129] These mature models are able to receive a single image sample as input and output the corresponding scene depth map end-to-end. The output scene depth map is an image with the same resolution as the input image, in which each pixel value represents the depth information of the corresponding position in the scene. This depth information can then be used to assist in image segmentation tasks, not only providing the relative position information of the person and the background to distinguish the person from the background, but also helping to handle complex scenes such as occlusion and overlapping areas.

[0130] In this embodiment, the image samples in the basic data set are input one by one into the DepthAnything model or other similar depth estimation models. The model analyzes the image and outputs the corresponding scene depth map, which will be used in the subsequent training process to help generate more accurate character masks.

[0131] Step S1300: determining a character boundary map according to the character mask and the scene depth map of the image sample, wherein the character boundary map marks character boundary information of the character image in the corresponding image sample;

[0132] The character boundary map is an image that marks the boundary areas of characters in an image. These boundary areas usually include character contours and possible occlusion areas, and are used to provide character boundary information pre-annotated for corresponding image samples.

[0133] The person mask already indicates which areas of the image sample contain people, so combined with the scene depth map, it is possible to analyze which pixels in these areas are at the edge of the person. Scene depth information helps distinguish the boundary between the person and the background, especially when the person and the background are similar in color or the lighting conditions are complex. By combining the person mask and the scene depth map, the edge area where the person touches the background can be identified.

[0134] In order to generate a person boundary map, a variety of image processing techniques can be used, such as gradient calculation, edge detection algorithms (such as Sobel operator, Canny operator), etc. These techniques can help identify areas with drastic depth changes, which usually correspond to the boundaries of people. For example, if a pixel belongs to a person in the person mask, and the depth value in the depth map is significantly different from the depth value of the surrounding non-person area, then this pixel is likely to be the boundary of the person.

[0135] The generated character boundary map will be used to supervise the boundary detection branch in the training depth estimation segmentation model. In this way, the model can not only learn to segment characters, but also learn to identify and optimize the boundaries of characters, thereby improving the segmentation quality, especially in the edge areas of characters.

[0136] Step S1400: storing each image sample and its corresponding character mask, scene depth map, and character boundary map in a training data set for training the depth estimation and segmentation model.

[0137] According to the above process, each image sample already has the corresponding person mask, scene depth map and person boundary map, which completes its annotation information. In the annotation information of the image sample, the person mask provides the precise position information of the person in the image, the scene depth map provides the depth information of each pixel in the image relative to the camera, and the person boundary map annotates the pixel points at the edge of the person in detail. These three types of information complement each other and provide the model with a comprehensive perspective to understand the image content.

[0138] Based on this, each image sample can be stored in a training dataset together with its corresponding person mask, scene depth map, and person boundary map so as to be used for training the depth estimation segmentation model. With the help of the training dataset, the depth estimation segmentation model will learn how to predict accurate person mask information, scene depth information, and person boundary information based on the content of the input image sample.

[0139] This embodiment efficiently integrates and prepares the training data set, significantly improving the efficiency of preparing multi-dimensional information. Based on the image samples and character masks of the basic data set, a mature depth estimation model is used to generate a scene depth map, and then the character mask and scene depth map are combined to determine the character boundary map. Finally, all relevant information is integrated and stored to obtain a rich and high-quality training data set. This coherent process not only optimizes the data preparation process and reduces manpower requirements, but also improves the consistency and availability of the data, reduces the overall preparation cost, and provides a solid data foundation for the training of live stream image segmentation models.

[0140] Based on any embodiment of the method of the present application, determining a character boundary map according to a character mask and a scene depth map of an image sample includes:

[0141] Step S1310, converting the scene depth map of the image sample into a corresponding gradient map;

[0142] First, for each image sample, the scene depth map (depth i ) is converted into a gradient map The gradient map can be obtained by calculating the first-order derivative or gradient of the depth map, which can highlight the areas in the image where the depth changes significantly. These areas usually correspond to the edges of objects, including the outlines of people. Gradient calculation can be achieved by applying edge detection techniques such as Sobel operator, Prewitt operator or Canny operator in image processing.

[0143] Step S1320, multiplying the gradient map of the image sample and the character mask to obtain a character texture map;

[0144] The obtained gradient map and the corresponding character mask i ) performs pixel-by-pixel multiplication, that is, As a character texture map. The character mask is a binary image in which the character area is marked as 1 and the background area is marked as 0. Through this multiplication operation, only the part of the gradient map that overlaps with the character mask is retained, and the other areas are set to 0 to generate a character texture map. The character texture map effectively limits the depth change information near the character outline, preparing for the next step of edge refinement.

[0145] Step S1330: Filter the character texture map, and retain pixels whose values ​​exceed a preset threshold to obtain a character boundary map.

[0146] Finally, the character texture map is threshold filtered to extract the final character boundary map. Specifically, by setting a threshold (α), only when the pixel value in the character texture map exceeds this threshold, these pixels are retained in the final character boundary map. This threshold filtering process helps to further refine and clarify the edge of the character, exclude those areas with little gradient change, and obtain a more accurate character boundary.

[0147] Through the above process, the precise boundary information of the person can be effectively extracted from the image sample and a person boundary map can be generated. This person boundary map not only contains the position information of the person edge, but also provides the depth change information of the edge area by combining with the depth information, which provides rich feature information for the subsequent segmentation model training, so that the depth estimation segmentation model can pay more attention to the segmentation accuracy of the person edge area during the training process, thereby improving the quality of live stream image segmentation, especially in the person edge area.

[0148] Based on any embodiment of the method of the present application, after each image sample and its corresponding person mask, scene depth map and person boundary map are stored in a training data set, the method includes:

[0149] Step S1500, determining the depth value of the person image in the image sample according to the scene depth map and the person mask of each image sample in the training data set;

[0150] Extract the scene depth map and person mask of each image sample from the training dataset. The scene depth map provides the depth information of each pixel in the image, while the person mask indicates the position of the person in the image. By combining these two information, the depth value of the person image in the image can be calculated. Specifically, the formula ∑depth i *maski To determine the depth value, where depth i is the scene depth map, mask i This formula multiplies the value in the depth map by the value in the character mask and then sums them up to get the depth value corresponding to the character area.

[0151] Step S1600: According to the depth of the depth value, the person image in the image sample with a shallow depth is superimposed and merged into the image sample with a deep depth to obtain a newly added image sample;

[0152] According to the calculated depth value of the person image, the person image in the image sample with a shallower depth is superimposed and fused into the image sample with a deeper depth to expand a new image sample. For example, two image samples are selected, wherein the depth value of the person image in the first image sample is smaller, so the depth of the person is shallower, and the depth value of the person image in the second image sample is larger, so the depth of the person is deeper. Then, the person image in the first image sample with a shallower depth is superimposed on the second image sample with a deeper depth according to its person mask information to create a newly added image sample. This newly added image sample contains the person in the two original image samples and the background information of one of the image samples.

[0153] Step S1700: for each newly added image sample, fuse the character mask, scene depth map and character boundary map of each source image sample, associate them with the newly added image sample and store them in the training data set for training the depth estimation segmentation model.

[0154] For each newly added image sample, its corresponding person mask, scene depth map, and person boundary map are associated with the newly added image sample and stored in the training dataset. In this way, each newly added image sample contains complete annotation information and can be used to train the depth estimation segmentation model. This ensures the consistency and availability of the training dataset, provides the model with rich training samples, and helps improve the performance and accuracy of the model in practical applications.

[0155] This embodiment significantly improves the efficiency of preparing training samples and reduces the cost through innovative sample fusion technology, while enhancing the generalization characteristics of the samples.

[0156] First, by calculating the character depth value of each image sample, the depth hierarchy relationship between the character and the background can be automatically identified and extracted, providing a basis for subsequent sample fusion.

[0157] Then, the shallower character images are merged with the darker background, and the depth relationship between different character images is properly handled to create new training samples. This process does not require additional image acquisition or manual labeling, which greatly reduces the cost and time of data preparation.

[0158] In addition, the newly added samples naturally enrich the diversity of the training data set because they incorporate different characters and backgrounds, enhance the model's adaptability and generalization features for different scenarios, and especially help the model achieve end-to-end one-step segmentation of multiple character images in live image frames.

[0159] Finally, these new samples and their complete annotation information are integrated into the training dataset to ensure the consistency and availability of the data, providing a high-quality data foundation for the training of the depth estimation segmentation model, which helps to improve the accuracy and robustness of the model in actual live stream image segmentation.

[0160] On the basis of any embodiment of the method of the present application, the segmented person mask of the current image frame is iteratively smoothed between frames using the person mask of the historical image frame to obtain a practical person mask of the current image frame, including:

[0161] Step S3310, using the filter configuration parameters determined in the process of generating the practical person mask of the historical image frame and the error information between the segmented person mask of the current image frame and the practical person mask to update the parameters of the Kalman filter, and obtain the filter configuration parameters corresponding to the current image frame, the filter configuration parameters including the prediction error covariance, the measurement noise covariance and the Kalman gain;

[0162] The Kalman filter is used to iteratively filter each image frame in the image sequence to achieve inter-frame smoothing, so as to optimize the quality and stability of the segmented character mask, and can achieve better real-time performance on devices with limited computing resources, such as various terminal devices.

[0163] When applying the Kalman filter for inter-frame smoothing, it is first necessary to initialize its filter configuration parameters. In this embodiment, the filter configuration parameters include prediction error covariance P, measurement noise covariance R, process noise covariance Q and Kalman gain K.

[0164] The prediction error covariance P represents the uncertainty of the model's estimate of the current state. At initialization, P is set to a large value, reflecting the high uncertainty of the initial state estimate before any actual measurements are obtained. The measurement noise covariance R represents the noise level during the actual measurement process, reflecting the credibility of the output of the depth estimation segmentation model. The process noise covariance Q represents the uncertainty of the filter model's prediction, reflecting the size of the error that the model may make when predicting the next frame. The Kalman gain K is a weight value that weighs the importance of prediction and actual measurement.

[0165] According to the properties of each filtering configuration parameter, in one embodiment, each parameter may be initialized using the following combination:

[0166] P0=1; R0=1; K0=0.5; Q=300

[0167] The subscript 0 indicates the initialization state.

[0168] During the filtering process, some filter configuration parameters can be iteratively updated, including the prediction error covariance P, the measurement noise covariance R, and the Kalman gain K. Considering the trust in the Kalman filter model, the process noise covariance Q can be used as a constant and not be updated.

[0169] Since the Kalman filter is applied iteratively to each image frame in the sequence, it smoothes the segmented person mask based on the depth estimation segmentation model generating a corresponding segmented person mask for each image frame in the live stream to generate a smoothed person mask as a practical person mask. Therefore, in this embodiment, the image frame before the current image frame that generates the segmented person mask is used as a historical image frame, and the practical person mask M of the historical image frame is used. t―1 and the segmented person mask M of the current image frame t The error information between the historical image frames can be used to update some of the filter configuration parameters of the Kalman filter. t―1 and the segmented person mask M of the current image frame t Subtract to obtain, which indicates the change of the mask between the previous frame and the current frame. Using this error information, the prediction error covariance P can be updated t , the formula example is as follows:

[0170] P t =P t―1 +(M t―1 ―M t )⊙(M t―1 ―M t )Q

[0171] In this embodiment, t represents the parameter corresponding to the current image frame, and t-1 represents the parameter corresponding to the previous image frame, that is, the historical image frame.

[0172] At the same time, the measurement noise covariance R t Update according to the following formula:

[0173]

[0174] The update of the above two parameters is implemented in the prediction stage of the iterative process. In the update stage of the Kalman filter, the Kalman gain in the filter configuration parameters also needs to be updated. The Kalman gain K t Calculated according to the following formula:

[0175]

[0176] Kalman gain K t Used to determine the weights of the predicted and actual measured values ​​when updating the segmented person mask of the current image frame. In this way, the filter can dynamically adjust the trust in the predicted and actual measured values ​​to obtain smoother and more stable segmentation results.

[0177] At this point, the filter configuration parameters required for the current image frame to generate its final practical person mask have been configured according to the filter configuration parameters corresponding to the historical image frame, that is, the t-1 sequence number, including: prediction error covariance P t , measurement noise covariance R t , Kalman gain K t This set of parameters completes the current round of iteration and has an impact on the prediction error covariance P t After the update again, the final determination will be obtained, and then continue to be used as the filtering configuration parameters corresponding to the historical image frame for the next image frame to implement the parameter update of the Kalman filter.

[0178] In addition, since the practical person mask M of the historical image frame is passed in each round of iteration t―1 To facilitate understanding of the program processing, the following variables are used Practical people masks representing historical image frames:

[0179]

[0180] Step S3320, using the Kalman gain in the updated filter configuration parameters to weight the error information to extract correction information, and adding the correction information to the practical person mask of the historical image frame to obtain a predicted person mask corresponding to the current image frame;

[0181] The Kalman filter uses the filter configuration parameters updated in the previous step, and can further combine the Kalman gain and the error information mentioned above to generate a predicted person mask corresponding to the current image frame.

[0182] Specifically, the error information is obtained by comparing the practical person mask M of the historical image frames t―1 That is to say and the segmented person mask M of the current image frame t The calculation method is obtained as This error information reflects the changes in the mask between the two frames and is the key data for the Kalman filter to perform state estimation correction.

[0183] Kalman gain K t is a weight factor between 0 and 1, which is used to balance the weight between the predicted value and the actual measured value. In this step, the Kalman gain K t Used to weight the error information, the calculation formula is:

[0184]

[0185] This correction information indicates the difference between the actual segmented character mask of the current image frame and the predicted mask corresponding to the current image frame predicted based on the practical character mask of the historical image frame, and reflects the proportion of this difference in the prediction of the current frame through the adjustment of the Kalman gain.

[0186] Next, this correction information is combined with the practical person mask of the historical image frame Combined to generate the predicted person mask corresponding to the current image frame, reassigned to the variable M t :

[0187]

[0188] This process is actually adjusting the practical person mask of the historical image frame to make it closer to the actual segmentation result of the current image frame. In this way, the error caused by motion prediction or model uncertainty can be reduced, thereby improving the accuracy of the segmentation mask. The adjusted data is reassigned to M t , M t In this link, it has become the predicted person mask corresponding to the current image frame.

[0189] Step S3330, smoothly fusing the predicted person mask of the current image frame and the practical person mask of the historical image frame according to the Kalman gain in the updated filtering configuration parameters to obtain the practical person mask of the current image frame;

[0190] The predicted person mask M of the current image framet and the Kalman gain K t All have been updated in the previous step, so this gain can be used to adjust the weight between the predicted person mask and the practical person mask of the historical image frame to achieve smoothing. The formula is as follows:

[0191]

[0192] The meaning of this formula is that the practical character mask of the historical image frame and the predicted person mask M of the current image frame t Weighted smooth fusion of Kalman gain K t As the weight used in the smoothing weighting, the proportion of the predicted person mask in the actual person mask of the current image frame is determined, and (1-K t ) is the weight of the practical person mask of the historical image frame. The practical person mask of the current image frame is obtained by weighted smooth fusion and is reassigned to M t , here, M t will refer to the practical person mask of the current image frame. In the next iteration, M t―1 To refer to this practical character mask M in this iteration t In this way, the latest information of the current frame can be integrated while maintaining historical information, thus achieving a smooth effect.

[0193] This smoothing is important because it helps reduce jitter and inconsistency in the segmentation results due to frame-to-frame changes. For example, in a live broadcast scene, if the person is moving quickly or the background is complex, the information of the current frame alone may produce unstable segmentation results. By smoothing with the Kalman filter, we can ensure that the segmentation results remain smooth and accurate even in these complex situations.

[0194] Step S3340: Update the prediction error covariance applied to the current image frame according to the Kalman gain applied to the historical image frame to determine the final filter configuration parameters, and use the final determined filter configuration parameters to update the parameters of the Kalman filter required for the next image frame to achieve iterative smoothing processing.

[0195] After the inter-frame smoothing process above, the predicted person mask M of the current image frame is obtained. t Next, the filter configuration parameters need to be finally iteratively updated so that these parameters can be used to update the parameters of the Kalman filter of the next image frame to achieve iterative smoothing processing.

[0196] In this step, only the prediction error covariance needs to be updated, while the measurement noise covariance and Kalman gain do not need to be updated. t―1 To update the prediction error covariance of the current image frame, and pass the measurement noise covariance and Kalman gain to the relevant variables. According to the programming statement representation, the update formula is as follows:

[0197] P t―1 =(1-K t―1 )·P t

[0198] K t―1 =K t

[0199] R t―1 =R t

[0200] According to the above program assignment statements, it is not difficult to understand that the prediction error covariance P used by the current image frame to generate its actual character mask t Combine historical image frames to generate the Kalman gain K corresponding to the practical character mask t―1 To reduce uncertainty, the new prediction error covariance is reassigned to P t―1 For the next round of iterations. In this way, the filter can gradually adjust the prediction accuracy of future states.

[0201] For the current image frame, the Kalman gain K corresponding to the actual person mask is generated. t and the measurement noise covariance R t , so there is no need to update in this link, so the corresponding value is directly assigned to the variable K t―1 and R t―1 For direct call in the next iteration.

[0202] Through the above updates, the filter configuration parameters of the Kalman filter for the next image frame are finally determined: the updated prediction error covariance P t―1 , Kalman gain P t―1 and the measurement noise covariance R t―1 These parameters will be used as filtering configuration parameters for the iterative filtering process of the next frame of image.

[0203] It is worth noting that the process noise covariance Q is used as a constant in this embodiment and does not participate in iterative updates. This is because the process noise covariance Q represents the uncertainty of model prediction, and this embodiment defaults to it as a relatively stable system parameter and does not need to be updated frequently.

[0204] Finally, these filter configuration parameters will be used to update the parameters of the Kalman filter of the next image frame, realizing the iterative process of inter-frame smoothing. By cyclically executing the above iterative calculation process, the Kalman filter completes the smoothing of the temporal segmentation results of the video, ensuring that the segmentation results of consecutive frames in the live stream have better coherence and stability.

[0205] Through this iterative update method, the Kalman filter can dynamically adjust the trust in the predicted and actual measured values ​​to adapt to the dynamic changes of the character image in the live stream. This not only improves the accuracy of segmentation, but also makes the segmentation results smoother and more natural, meeting the high requirements of live broadcast scenarios for real-time and stability.

[0206] This embodiment achieves smoothing of segmentation results between live stream image frames by applying the Kalman filter principle to the segmented character mask generated by the depth estimation segmentation model, thus achieving significant technical advantages.

[0207] First, by iteratively updating the filtering configuration parameters, including the prediction error covariance, measurement noise covariance, and Kalman gain, the filter model’s trust in the prediction and actual measurement values ​​can be dynamically adjusted, thereby optimizing the quality of the segmentation mask. This is particularly suitable for live streaming processing with high real-time requirements, as it can achieve fast and accurate segmentation on a variety of terminal devices.

[0208] Secondly, the advantage of real-time processing is also reflected in computational efficiency. Although complex inter-frame smoothing is implemented, due to the high efficiency of the Kalman filter, no significant additional computational burden is introduced. This enables real-time segmentation and smoothing even on devices with limited computing resources, meeting the needs of live broadcast scenarios.

[0209] Furthermore, by utilizing the error information between the practical person mask of the historical image frame and the segmented person mask of the current image frame, the filter parameters can be accurately updated to reduce the uncertainty of the segmentation results. By fusing the information of the historical and current frames in a weighted manner, a smooth and coherent practical person mask is generated, which effectively reduces the segmentation jitter and flicker caused by inter-frame changes and improves the naturalness and stability of the segmented image.

[0210] In addition, by iteratively updating the filter configuration parameters, customized processing is provided for each frame of the image, so that the segmentation results can adapt to dynamic changes in the live stream, such as the rapid movement of people or changes in the background. This adaptability ensures that high-quality segmentation results can be maintained even in complex and changing live broadcast environments, so that upstream services that rely on practical human mask segmentation of human images can obtain excellent user viewing experience.

[0211] See also Figure 3According to one aspect of the present application, a live stream image segmentation device is provided, comprising an image calling module 3100, an intelligent segmentation module 3200, an inter-frame smoothing module 3300, and a portrait determination module 3400, wherein the image calling module 3100 is configured to sequentially call image frames from an image sequence of the live stream as the current image frame; the intelligent segmentation module 3200 is configured to call a depth estimation segmentation model to determine a segmented character mask corresponding to a character image therein based on the current image frame, wherein the depth estimation segmentation model has been supervised and trained using scene depth information; the inter-frame smoothing module 3300 is configured to iteratively perform inter-frame smoothing processing on the segmented character mask of the current image frame using practical character masks of historical image frames to obtain a practical character mask of the current image frame; the portrait determination module 3400 is configured to determine the character image in the current image frame according to the practical character mask of the current image frame.

[0212] On the basis of any embodiment of the device of the present application, prior to the image calling module 3100, the device of the present application also includes: a training prediction module, configured to iteratively input the image samples in the training data set into the depth estimation segmentation model for inference, corresponding to each prediction branch preset for the backbone network of the depth estimation segmentation model, and respectively predicting the scene depth information, character boundary information and character mask information; a training supervision module, configured to supervise the training of the depth estimation segmentation model to convergence using the scene depth information, character boundary information and character mask information annotated for the image samples; a pruning deployment module, configured to retain the backbone network in the depth estimation segmentation model and the prediction branches corresponding to the character mask information, for determining the segmented character mask in the image frame.

[0213] Based on any embodiment of the device of the present application, the training supervision module includes: a loss calculation module, which is configured to correspond to the results output by each prediction branch of the image sample, and respectively calculate the branch loss value of each prediction branch using the scene depth information, character boundary information and character mask information annotated for the image sample; a fusion update module, which is configured to fuse the branch loss values ​​of each prediction branch into an overall loss value, and perform gradient update on the depth estimation segmentation model according to the overall loss value.

[0214] On the basis of any embodiment of the device of the present application, prior to the training prediction module, the device of the present application also includes: a material acquisition module, configured to acquire a basic data set, which includes multiple image samples and their corresponding character masks, and the character masks mark the character mask information corresponding to the image area where the character image in the corresponding image sample is located; a depth recognition module, configured to use a scene depth estimation model to generate a corresponding scene depth map for each image sample in the basic data set, and the scene depth map marks the scene depth information of the image content in the corresponding image sample; a boundary determination module, configured to determine a character boundary map based on the character mask and scene depth map of the image sample, and the character boundary map marks the character boundary information of the character image in the corresponding image sample; a data storage module, configured to store each image sample and its corresponding character mask, scene depth map and character boundary map in the training data set for implementing training of the depth estimation segmentation model.

[0215] Based on any embodiment of the device of the present application, the boundary determination module includes: a gradient determination module, configured to convert the scene depth map of the image sample into a corresponding gradient map; a texture extraction module, configured to multiply the gradient map of the image sample and the character mask to obtain a character texture map; a boundary extraction module, configured to filter the character texture map and retain pixels whose values ​​exceed a preset threshold to obtain a character boundary map.

[0216] On the basis of any embodiment of the device of the present application, after the data storage module, the device of the present application also includes: a depth setting module, which is configured to determine the depth value of the character image in the image sample according to the scene depth map and character mask of each image sample in the training data set; a sample augmentation module, which is configured to superimpose and fuse the character image in the image sample with a shallower depth into the image sample with a deeper depth according to the depth of the depth value to obtain a newly added image sample; a fusion and appending module, which is configured to fuse the character mask, scene depth map and character boundary map of each source image sample for each newly added image sample, and associate them with the newly added image sample and store them in the training data set for implementing training of the depth estimation and segmentation model.

[0217] Based on any embodiment of the device of the present application, the inter-frame smoothing module 3300 includes: a parameter updating module, which is configured to use the filtering configuration parameters determined in the process of generating the practical character mask of the historical image frame, and the error information between the segmented character mask of the current image frame and the practical character mask to update the parameters of the Kalman filter to obtain the filtering configuration parameters corresponding to the current image frame, wherein the filtering configuration parameters include the prediction error covariance, the measurement noise covariance and the Kalman gain; a mask prediction module, which is configured to use the Kalman gain in the updated filtering configuration parameters to weightedly extract the correction information from the error information, and The correction information is added to the practical character mask of the historical image frame to obtain the predicted character mask corresponding to the current image frame; a smoothing fusion module is configured to smoothly fuse the predicted character mask of the current image frame and the practical character mask of the historical image frame according to the Kalman gain in the updated filtering configuration parameters to obtain the practical character mask of the current image frame; a transfer iteration module is configured to update the prediction error covariance applied to the current image frame according to the Kalman gain applied to the historical image frame to determine the final filtering configuration parameters, and the finally determined filtering configuration parameters are used for the parameter update of the Kalman filter required for the next image frame to achieve iterative smoothing processing.

[0218] Another embodiment of the present application also provides a live stream image segmentation device. Figure 4 As shown, a schematic diagram of the internal structure of a live stream image segmentation device. The live stream image segmentation device includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable non-volatile readable storage medium of the live stream image segmentation device stores an operating system, a database, and computer-readable instructions. The database may store an information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a live stream image segmentation method.

[0219] The processor of the live stream image segmentation device is used to provide computing and control capabilities to support the operation of the entire live stream image segmentation device. The memory of the live stream image segmentation device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the live stream image segmentation method of the present application. The network interface of the live stream image segmentation device is used to connect and communicate with a terminal.

[0220] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the live stream image segmentation device to which the scheme of the present application is applied. The specific live stream image segmentation device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0221] In this embodiment, the processor is used to execute Figure 3 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the live stream image segmentation device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0222] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live stream image segmentation method of any embodiment of the present application.

[0223] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0224] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0225] In summary, this application has significant advantages in improving the interactivity and viewing experience of live broadcasts, has broad application prospects in the fields of live broadcasting and related image processing, and is expected to become the mainstream direction of future live stream image segmentation technology.

Claims

1. A live stream image segmentation method, characterized in that: include: Call image frames from the image sequence of the live stream in sequence as the current image frame; Calling a depth estimation segmentation model to determine a segmented person mask corresponding to the person image in the current image frame based on the current image frame, wherein the depth estimation segmentation model has been supervised and trained using scene depth information; Iteratively smooth the segmented person mask of the current image frame using the practical person mask of the historical image frame to obtain the practical person mask of the current image frame; According to the practical person mask of the current image frame, a person image in the current image frame is determined.

2. The live streaming image segmentation method according to claim 1, characterized in that: Before calling an image frame from the image sequence of the live stream as the current image frame in order, including: Iteratively input the image samples in the training data set into the depth estimation segmentation model for inference, and respectively predict the scene depth information, character boundary information, and character mask information corresponding to the prediction branches preset for the backbone network of the depth estimation segmentation model; Supervising the training of the depth estimation and segmentation model until convergence using the scene depth information, character boundary information, and character mask information annotated for the image sample; The backbone network in the depth estimation segmentation model and the prediction branch corresponding to the character mask information are retained to determine the segmented character mask in the image frame.

3. The live streaming image segmentation method according to claim 2, characterized in that: Using the scene depth information, character boundary information, and character mask information annotated for the image sample to correspondingly supervise the depth estimation and segmentation model training until convergence, comprising: Corresponding to the result outputted by each prediction branch of the image sample, the branch loss value of each prediction branch is calculated using the scene depth information, the character boundary information and the character mask information annotated for the image sample; The branch loss values ​​of each prediction branch are merged into an overall loss value, and the depth estimation segmentation model is gradient updated according to the overall loss value.

4. The live streaming image segmentation method according to claim 2, characterized in that: Iteratively input image samples in the training dataset into the depth estimation segmentation model before reasoning, including: Obtaining a basic data set, which includes a plurality of image samples and their corresponding person masks, wherein the person masks mark person mask information corresponding to an image region where a person image in the corresponding image sample is located; Generate a corresponding scene depth map for each image sample in the basic data set using a scene depth estimation model, wherein the scene depth map marks scene depth information of the image content in the corresponding image sample; Determine a person boundary map according to the person mask of the image sample and the scene depth map, wherein the person boundary map marks the person boundary information of the person image in the corresponding image sample; Each image sample and its corresponding person mask, scene depth map, and person boundary map are stored in a training data set for training the depth estimation and segmentation model.

5. The live streaming image segmentation method according to claim 4, characterized in that: Determine a character boundary map according to the character mask and the scene depth map of the image sample, including: Convert the scene depth map of the image sample into a corresponding gradient map; Multiply the gradient map of the image sample and the character mask to obtain the character texture map; The character texture map is filtered, and pixels whose values ​​exceed a preset threshold are retained to obtain a character boundary map.

6. The live streaming image segmentation method according to claim 4, characterized in that: After storing each image sample and its corresponding person mask, scene depth map, and person boundary map in the training dataset, it includes: Determine the depth value of the person image in the image sample according to the scene depth map and the person mask of each image sample in the training data set; According to the depth of the depth value, the person image in the image sample with a shallower depth is superimposed and fused to the image sample with a deeper depth to obtain a newly added image sample; For each newly added image sample, the character mask, scene depth map and character boundary map of each source image sample are fused, associated with the newly added image sample and stored in the training data set for implementing training of the depth estimation segmentation model.

7. The live streaming image segmentation method according to any one of claims 1 to 6, characterized in that: The segmented person mask of the current image frame is iteratively smoothed between frames using the person mask of the historical image frame to obtain a practical person mask of the current image frame, including: The Kalman filter is updated by using the filter configuration parameters determined in the process of generating the practical person mask of the historical image frame and the error information between the segmented person mask of the current image frame and the practical person mask to obtain the filter configuration parameters corresponding to the current image frame, wherein the filter configuration parameters include the prediction error covariance, the measurement noise covariance and the Kalman gain; The error information is weighted by using the Kalman gain in the updated filter configuration parameters to extract correction information, and the correction information is added to the practical person mask of the historical image frame to obtain a predicted person mask corresponding to the current image frame; The predicted person mask of the current image frame and the practical person mask of the historical image frame are smoothly fused according to the Kalman gain in the updated filter configuration parameters to obtain the practical person mask of the current image frame; The prediction error covariance applied to the current image frame is updated according to the Kalman gain applied to the historical image frame to determine the final filter configuration parameters, and the final determined filter configuration parameters are used to update the parameters of the Kalman filter required for the next image frame to achieve iterative smoothing processing.

8. A live stream image segmentation device, characterized in that: include: An image calling module is configured to call image frames in sequence from an image sequence of a live stream as current image frames; An intelligent segmentation module is configured to call a depth estimation segmentation model to determine a segmented person mask corresponding to a person image in a current image frame, wherein the depth estimation segmentation model has been supervised and trained using scene depth information; An inter-frame smoothing module is configured to perform inter-frame smoothing processing on the segmented person mask of the current image frame by using the practical person mask of the historical image frame to iteratively obtain the practical person mask of the current image frame; The portrait determination module is configured to determine the person image in the current image frame according to the practical person mask of the current image frame.

9. A live streaming image segmentation device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images