Image segmentation mask correction method and its device, equipment and medium

Abstract

The present application relates to the field of network live broadcast technology, and discloses an image segmentation mask correction method and its device, equipment, and medium, the method comprising: calculating a differential sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, the matrix being a moving average result of a confidence difference matrix of image segmentation masks between two adjacent video frames in the current video frame and its multiple previous video frames; calculating a confidence sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, the matrix being determined according to a moving average result of confidences of each pixel in the image segmentation masks of the current video frame and its multiple previous video frames; determining a confidence correction matrix of the current video frame according to the confidence sliding average matrix and the differential sliding average matrix; and correcting the image segmentation mask of the current video frame according to the confidence correction matrix. The technical solution of the present application has the advantages of plug-and-play, high versatility, and low computational complexity.

Description

Technical Field

[0001] The present application relates to the field of network live broadcast technology, and in particular to an image segmentation mask correction method and its corresponding device, computer equipment and computer-readable storage medium. Background Art

[0002] In the online live broadcast scenario, the host user pushes his or her video stream to the live broadcast room to achieve application purposes such as talent display, information sharing, and knowledge education, allowing the host user to participate in social labor and gain income through these activities, thereby promoting overall social benefits.

[0003] In order to facilitate the recognition and independent processing of certain real objects in the video stream, such as people, commodities and other related images, so that the anchor user can modify the background in the video image or add corresponding information, the video images in the video stream are often segmented with the help of image segmentation technology to obtain the corresponding image segmentation masks of these objects, so as to use the image segmentation masks to perform various subsequent processing on the corresponding objects. Stable video segmentation results bring better user experience in mainstream applications such as live background change and video conference background blur.

[0004] Traditional video segmentation methods are mainly based on image segmentation of each frame of the video. Due to the accuracy limitation of image segmentation, the image segmentation masks of each video frame in the video are generally unstable in time sequence, which will cause the identified objects to have inaccurate edges, unclear edges, blurred edges, edge jumps, and other unsmooth phenomena. Specifically, the same object is recognized as different categories by the image segmentation model in different frames, which is mainly manifested as flickering of the video segmentation results. This flickering phenomenon mainly exists at the edges of foreground objects or objects that are difficult to segment by the image segmentation model.

[0005] Some methods explored in the industry mainly correct the image segmentation results of the current video frame based on the image segmentation results of two adjacent frames, attempting to overcome such problems. However, in practice, they are unable to effectively deal with the problem of continuous jittering of multi-frame results, or have problems such as high computational cost and low efficiency. Summary of the invention

[0006] The primary purpose of the present application is to solve at least one of the above problems and to provide an image segmentation mask correction method and its corresponding device, computer equipment and computer-readable storage medium.

[0007] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0008] An image segmentation mask correction method proposed to meet one of the purposes of this application includes the following steps:

[0009] Calculating a differential sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, wherein the differential sliding average matrix is ​​determined according to a moving average result of confidence difference matrices of image segmentation masks between two adjacent video frames in the current video frame and a plurality of previous video frames;

[0010] Calculating a confidence sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, where the confidence sliding average matrix is ​​determined based on a moving average result of confidences of each pixel in the image segmentation mask of the current video frame and a plurality of previous video frames;

[0011] Determining a confidence correction matrix of a current video frame according to the confidence sliding average matrix and the differential sliding average matrix;

[0012] The image segmentation mask of the current video frame is corrected according to the confidence correction matrix.

[0013] In a further embodiment, before the step of calculating the differential sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, the following steps are included:

[0014] The confidence matrices of the image segmentation masks of the current video frame and the previous video frame adjacent thereto are modulated and added to obtain a weight matching matrix for quantifying the uncertainty of whether each pixel of the image segmentation mask of the current video frame belongs to the foreground or the background;

[0015] A confidence difference matrix between the image segmentation masks of the current video frame and its adjacent previous video frame is calculated, where the confidence difference matrix is ​​the element product of the absolute difference matrix between the confidence matrices corresponding to the image segmentation masks and the weight matching matrix.

[0016] In a further embodiment, calculating a confidence sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence comprises the following steps:

[0017] According to the exponential moving average algorithm, calling the confidence difference matrix of the image segmentation mask between the current video frame and its adjacent previous video frame and the differential sliding average matrix of the adjacent previous video frame, wherein the confidence difference matrix is ​​used to store the difference in confidence between each corresponding pixel point of the image segmentation mask between the current video frame and its adjacent previous video frame;

[0018] An exponential moving average algorithm is applied to obtain a moving average result of the confidence difference matrix of the current video frame and its previous video frame with a preset smoothing weight.

[0019] In a further embodiment, calculating a confidence sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence comprises the following steps:

[0020] According to the exponential moving average algorithm, the image segmentation mask of the current video frame and the confidence sliding average matrix corresponding to the previous video frame are called, the image segmentation mask is represented by a confidence matrix, and the confidence matrix is ​​used to store the confidence of each pixel point of the image segmentation mask;

[0021] An exponential moving average algorithm is applied to obtain a moving average result of the confidence matrix of the current video frame and its previous video frame with a preset smoothing weight to obtain the confidence average matrix.

[0022] In some further embodiments, determining a confidence correction matrix of a current video frame according to the confidence sliding average matrix and the differential sliding average matrix comprises the following steps:

[0023] Performing confidence compensation on the confidence sliding average matrix to obtain a confidence compensation matrix;

[0024] Calculating an element-wise product matrix of the confidence compensation matrix and the differential sliding average matrix;

[0025] A constant suppression factor is applied to adjust the correction strength represented by the element product matrix to obtain a confidence correction matrix.

[0026] In some further embodiments, correcting the image segmentation mask of the current video frame according to the confidence correction matrix includes the following steps:

[0027] Get the confidence matrix corresponding to the image segmentation mask of the current video frame;

[0028] Calculating a sum matrix of the confidence matrix and the confidence correction matrix;

[0029] The value of each element in the sum matrix is ​​trimmed to a confidence interval to obtain a modified image segmentation mask.

[0030] In some extended embodiments, the image segmentation mask correction method of the present application includes the following initialization steps:

[0031] Constructing a differential sliding average matrix corresponding to the first video frame in the video frame sequence, and assigning values ​​to all elements of the differential sliding average matrix by arbitrarily determining initial values;

[0032] Using the confidence matrix corresponding to the image segmentation mask of the first video frame in the video frame sequence as the confidence sliding average matrix corresponding to the first video frame;

[0033] The first video frame is skipped, and starting from the next adjacent video frame, other steps of the method are executed frame by frame as the current video frame.

[0034] An image segmentation mask correction device provided to meet one of the purposes of the present application includes: a difference calculation module, a confidence calculation module, a suppression calculation module, and a mask correction module, wherein:

[0035] A differential calculation module, used to calculate a differential sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, wherein the differential sliding average matrix is ​​determined based on a moving average result of a confidence difference matrix of an image segmentation mask between two adjacent video frames in the current video frame and a plurality of previous video frames thereof;

[0036] A confidence calculation module, used to calculate a confidence sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, wherein the confidence sliding average matrix is ​​determined based on a moving average result of confidences of each pixel in the image segmentation mask of the current video frame and a plurality of previous video frames;

[0037] A suppression calculation module, used to determine a confidence correction matrix of a current video frame according to the confidence sliding average matrix and the differential sliding average matrix;

[0038] The mask correction module is used to correct the image segmentation mask of the current video frame according to the confidence correction matrix.

[0039] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the image segmentation mask correction method described in the present application.

[0040] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the image segmentation mask correction method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0041] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0042] Compared with the prior art, the advantages of this application include but are not limited to the following aspects:

[0043] First, when the present application corrects the image segmentation mask of each current video frame in the video frame sequence, the differential sliding average matrix and the confidence sliding average matrix of the current video frame are first determined respectively, wherein the differential sliding average matrix is ​​obtained by moving average of the confidence frame difference information between two adjacent video frames in the current video frame and its multiple previous video frames, so as to realize the statistics and effective representation of multiple frame difference information of the video frame sequence; the confidence sliding average matrix is ​​obtained by moving average of the confidence of each pixel point of the image segmentation mask in the current video frame and its multiple previous video frames, so as to realize the statistics and effective representation of the confidence of multiple image segmentation masks of the video frame sequence. On the basis of determining the differential sliding average matrix and the confidence sliding average matrix, for each current video frame, the two matrices are used to realize the effective integration of the frame difference information and the confidence information in the entire video frame sequence, and on this basis, according to the relationship between the two, the confidence correction matrix used to correct the image segmentation mask of the current video frame is determined, and the confidence correction matrix is ​​used to realize the effective correction of the image segmentation mask of the current video frame.

[0044] Secondly, since the differential sliding average matrix and the confidence sliding average matrix used by the present application to determine the confidence correction matrix are both the results of the synthesis of the temporal information of the video frame sequence, the temporal confidence correction matrix has stronger smoothness, and the transition between the image segmentation masks obtained by the correction is softer and more accurate, which can remove the sudden mis-segmentation phenomenon, overcome the flickering and instability of the image segmentation results, and especially suppress the continuous instability within a certain period of time. Therefore, according to the present application, the edge recognition effect of object segmentation of video images is clear, accurate and smooth.

[0045] Furthermore, the present application directly processes the image segmentation masks of the video frame sequence, so it can be plug-and-play, that is, it is independent of the image segmentation algorithm of the video frame. The image segmentation mask obtained by any image segmentation algorithm can be expressed as a confidence matrix corresponding to each pixel point, and the technical solution of the present application is used to optimize it, so that the technical solution of the present application obtains a general technical advantage.

[0046] In addition, during the implementation of this application, the differential sliding average matrix and the confidence sliding average matrix are used as the basis, and then the confidence correction matrix is ​​calculated. Then, the confidence matrix of the image segmentation mask of the current video frame is corrected according to the confidence correction matrix. The entire process is implemented using matrix calculations. The complexity of the matrix calculations is very low, and there is no need to use highly complex morphological processing methods such as convolution and inter-frame template matching. Therefore, the amount of calculation is low and the operating efficiency is higher. It can not only accurately identify the target objects in the video frames, but also the recognition speed is very fast. For application scenarios with high timeliness requirements such as live broadcasts, it has obvious scene application advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 A flowchart of a typical embodiment of the image segmentation mask correction method of the present application is shown;

[0049] Figure 2 A flowchart of a process for exemplarily determining a weight matching matrix of frame difference information between image segmentation masks of a current video frame and a previous video frame in this application;

[0050] Figure 3 A schematic diagram of an algorithm principle comprehensively exemplified according to various embodiments of the image segmentation mask correction method of the present application;

[0051] Figure 4 Based on Figure 3 The example of the inter-frame image segmentation mask change effect obtained by actual measurement due to the algorithm described in the example. The applicant blurred the human body in the picture for the sake of portrait rights protection and information health considerations.

[0052] Figure 5 This is a principle block diagram of the image segmentation mask correction device of the present application;

[0053] Figure 6 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0055] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0057] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0058] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0059] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0060] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0061] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0062] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0063] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0064] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0065] The video stream that needs to be segmented can be processed with the help of preset image segmentation models such as U-net, U 2net series of deep learning models or image segmentation algorithms perform image segmentation on each video frame to obtain the corresponding image segmentation mask. Image segmentation refers to the pixel-by-pixel classification of each frame in the video. The image segmentation mask obtained by classification is in the form of a confidence matrix. In this application, it is specifically expressed in three dimensions: width, height, and category: M∈R h×w×c , where h and w represent the height and width of the video frame respectively, c represents each category, and each category corresponds to an object in an image. Each element in the confidence matrix represents the confidence that the corresponding pixel point of the video frame belongs to a category of object. For ease of understanding, in the description of each embodiment of the present application, it is assumed that only one category of foreground object is used for explanation. It should be noted that this principle can also be directly extended to the application of multi-category video segmentation tasks.

[0066] The goal of implementing various embodiments of the present application is to obtain an image segmentation mask M obtained by image segmentation of a current video frame in a video frame sequence of a video stream. i ∈R h×w×c The correction is performed, which is extended to the correction of the image segmentation mask of the entire video frame sequence or even each video frame of the entire video stream.

[0067] When the present application implements the modification of the image segmentation mask corresponding to the current video frame in a video frame sequence, the image segmentation mask sequence of each video frame in the video frame sequence is represented as M0, M1, ..., M i ∈R h×w×c , with the image segmentation mask M of the current video frame i And its adjacent previous video frame, that is, the image segmentation mask M of the previous frame i-1 As input, it is processed frame by frame. At the same time, the sliding average results of two statistical time series information, namely the differential sliding average matrix Dif, are maintained during frame-by-frame processing. ema ∈R h×w×c and the confidence sliding average matrix M ema ∈R h×w×c .

[0068] In order to facilitate the use of timing information, in some embodiments of the present application, initialization can be performed according to the first video frame in the video frame sequence to prepare for two statistical timing information. Specifically, the confidence matrix corresponding to the image segmentation mask of the first video frame M0 can be directly used as the confidence sliding average matrix M ema The initial value of the differential sliding average matrix is ​​created at the same time, and a certain initial value such as 0 is selected as the differential sliding average matrix Dif emaThe various elements of are uniformly assigned values, and are used as the initial data of the two statistical timing information. Then, the image segmentation mask of the first video frame is not processed, and the image segmentation mask M0 is directly output as its correction result M′0, that is, M′0=M0. Then, starting from the next video frame, each frame is used as the current video frame, and the various relevant steps adopted in the various embodiments of the present application are applied for processing.

[0069] Accordingly, please refer to Figure 1 The image segmentation mask correction method of the present application can be programmed as a computer program product and deployed in a computer device for execution. In its typical embodiment, when processing each current video frame, the method includes the following steps:

[0070] Step S1100, calculating a differential sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, wherein the differential sliding average matrix is ​​determined based on a moving average result of confidence difference matrices of image segmentation masks between two adjacent video frames in the current video frame and a plurality of previous video frames thereof:

[0071] For the current video frame in the video frame sequence, obtain its image segmentation mask M i As mentioned above, each element of the image segmentation mask corresponds to a pixel point of the current video frame, and stores the confidence that the pixel point belongs to a certain category of objects. When the image segmentation mask is determined, the confidence is generally normalized to the confidence interval of [0, 1]. Based on this, the image segmentation mask of the current video frame and the image segmentation masks of multiple previous video frames can be used to count the frame difference information between two adjacent video frames to obtain the differential sliding average matrix. The previous video frame refers to the video frame that is prior to the current video frame in terms of time sequence.

[0072] The differential sliding average matrix is ​​determined by calculating each current video frame in the video frame sequence one by one, but during the calculation, each current video frame will refer to the differential sliding average matrix determined by other previous video frames to achieve statistical synthesis of the changing characteristics of the frame difference information in time sequence.

[0073] The frame difference information refers to the confidence difference matrix obtained by calculating the absolute value of the confidence difference for each pixel between the confidence matrices corresponding to the image segmentation masks of two adjacent video frames in the current video frame and its multiple previous video frames, that is, Dif i =|M i -M i-1|. A single confidence difference matrix represents the relative change characteristics of the confidence of each pixel between two corresponding adjacent video frames. Multiple confidence difference matrices in time series can further represent the continuous change of this relative change characteristic. Therefore, multiple confidence difference matrices in time series can be further statistically synthesized to obtain the differential sliding average matrix In other embodiments of the present application, the confidence difference matrix may be further modified before calculating the moving average result, which will not be discussed here.

[0074] When performing statistical synthesis on the variation characteristics of the frame difference information, a moving average method can be used for calculation. The moving average method is applied to predict the variation characteristics of the frame difference of the current video frame relative to a series of previous video frames using a group of recent frame difference information. The moving average method is suitable for immediate prediction and can effectively eliminate random fluctuations in the prediction, which is very useful. The moving average method used in the present application can include: any one of a simple moving average method, a weighted moving average method, and an exponential moving average method.

[0075] The moving average method is a smoothing prediction technology. Its basic idea is to calculate the average time sequence of a certain number of items one by one according to the time series data to reflect the long-term trend. Therefore, when the frame difference information of the time series fluctuates greatly due to the influence of periodic changes and random fluctuations, and it is difficult to show the development trend of the event, the moving average method can eliminate the influence of these factors, show the development direction and trend of the event, and then analyze and predict the long-term trend of the series based on the trend line.

[0076] The differential moving average matrix described in this application, in some embodiments including this embodiment, is a moving average result of the confidence difference matrix of the image segmentation mask between two adjacent video frames in the current video frame and multiple previous video frames. In other embodiments of this application, the moving average result can be further modified, which will not be discussed here.

[0077] Step S1200: Calculate a confidence sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence. The confidence sliding average matrix is ​​determined based on the moving average results of the confidences of each pixel in the image segmentation masks of the current video frame and multiple previous video frames:

[0078] This step can be calculated in parallel with the previous step S1100, or can be calculated one after the other, and those skilled in the art can implement it flexibly.

[0079] Similar to the previous step S1100, this step performs a confidence sliding average matrix on the image segmentation mask of the current video frame. The calculation of the above-mentioned moving average method can be similarly performed using any one of the simple moving average method, the weighted moving average method, and the exponential moving average method, thereby adapting to the current video frame to achieve statistical synthesis of the confidence of each pixel point in each video frame in time sequence.

[0080] Specifically, for the current video frame, not only the confidence matrix of its own image segmentation mask is considered, but also the confidence matrices of multiple previous video frames are called. According to the corresponding relationship of the pixels, the average confidence of each pixel is moved and averaged to obtain the confidence sliding average matrix corresponding to the current video frame. The confidence of multiple recent video frames in time sequence is integrated through the confidence average matrix.

[0081] The confidence sliding average matrix described in the present application, in some embodiments including the present embodiment, is a moving average result of the confidence of each pixel in the image segmentation mask of the current video frame and multiple previous video frames thereof.

[0082] Step S1300: determining a confidence correction matrix of the current video frame according to the confidence sliding average matrix and the differential sliding average matrix:

[0083] As mentioned above, for the image segmentation mask of the current video frame, the confidence sliding average matrix It represents the average confidence obtained by moving the confidence matrices of the current video frame and its multiple previous video frames to achieve statistical synthesis in time series. The differential sliding average matrix A moving average is used to describe the temporal variation characteristics of the frame difference information determined based on each other's image segmentation masks between two adjacent video frames in the current video frame and its multiple previous video frames. Accordingly, the confidence sliding average matrix and the differential sliding smoothing matrix are obtained according to the two-way moving average results corresponding to the time series, thereby realizing the integration of the temporal variation characteristics of the image segmentation masks of each video frame in the video frame sequence, wherein the confidence matrix of the image segmentation mask of the current video frame is used as the calculation basis. Therefore, the confidence sliding average matrix and the differential sliding average matrix are fused. In a specific implementation, the two matrices can be element-wise multiplied corresponding to each pixel point to obtain the confidence correction matrix Sup which integrates the two-way temporal statistical information based on the image segmentation mask of the current video frame. i Element-wise product, also known as Hadamard Product, is a type of matrix multiplication. When two matrices are multiplied element-wise, the two matrices are matrices of the same order, and the element-wise product process is to directly multiply the corresponding elements in the two matrices as the values ​​of the corresponding elements in the result matrix.

[0084] In an alternative embodiment, in the process of determining the confidence correction matrix using the confidence sliding average matrix and the differential sliding average matrix, appropriate confidence compensation can be performed on the confidence sliding average matrix to compensate for the confidence mutation in time series, which will be disclosed in detail later.

[0085] In another alternative embodiment, in the process of determining the confidence correction matrix using the confidence sliding average matrix and the differential sliding average matrix, the fusion result of the confidence sliding average matrix and the differential sliding average matrix can be appropriately controlled in terms of correction strength to ensure the effectiveness of the confidence correction matrix.

[0086] It is not difficult to understand that the confidence correction matrix Sup determined accordingly i , which describes the image segmentation mask M for the current video frame i degree of correction.

[0087] Step S1400: Correct the image segmentation mask of the current video frame according to the confidence correction matrix:

[0088] When the confidence correction matrix Sup is obtained i After that, the confidence matrix M corresponding to the image segmentation mask of the current video frame can be i Fusion is performed to correct the current image segmentation mask and obtain the corrected image segmentation mask M′ i In one embodiment, the two matrices can be simply added together, that is, the corresponding elements of the two matrices are summed for each pixel point, and then the summation result is appropriately pruned so that the confidence of each element of the pruned confidence matrix returns to the confidence interval, thereby obtaining the modified image segmentation mask M′. i .

[0089] Complete the image segmentation mask M′ of the current video frame i After the correction, the image segmentation of the current video frame is completed, and various downstream tasks are implemented based on this, such as blurring the background and marking the borders of specific objects in the live video stream, to obtain a more accurate and stable edge processing effect.

[0090] After completing the correction of the image segmentation mask of the current video frame, the next video frame in the video frame sequence can be started as the current video frame, and steps S1100 to S1400 of the present application can be looped to continue to perform the same processing on the new current video frame, and so on, to achieve the correction of the image segmentation mask for each video frame in the video frame sequence.

[0091] As for the confidence sliding average matrix generated in the process of correcting the image segmentation mask of the current video frame and the difference moving average matrix In an alternative embodiment, the confidence sliding average matrix can be cached in the corresponding confidence sliding average sequence and differential sliding average sequence respectively, so that when calculating the next current video frame, the confidence sliding average matrix of the previous current video frame can be directly called. and the difference moving average matrix Participate in the calculation of the moving average result of the next current video frame, thereby simplifying the calculation complexity and improving the calculation efficiency.

[0092] According to the typical embodiments of the present application and various alternative embodiments disclosed therein, it can be understood that the present application has many technical advantages, including but not limited to the following aspects:

[0093] First, when the present application corrects the image segmentation mask of each current video frame in the video frame sequence, the differential sliding average matrix and the confidence sliding average matrix of the current video frame are first determined respectively, wherein the differential sliding average matrix is ​​obtained by moving average of the confidence frame difference information between two adjacent video frames in the current video frame and its multiple previous video frames, so as to realize the statistics and effective representation of multiple frame difference information of the video frame sequence; the confidence sliding average matrix is ​​obtained by moving average of the confidence of each pixel point of the image segmentation mask in the current video frame and its multiple previous video frames, so as to realize the statistics and effective representation of the confidence of multiple image segmentation masks of the video frame sequence. On the basis of determining the differential sliding average matrix and the confidence sliding average matrix, for each current video frame, the two matrices are used to realize the effective integration of the frame difference information and the confidence information in the entire video frame sequence, and on this basis, according to the relationship between the two, the confidence correction matrix used to correct the image segmentation mask of the current video frame is determined, and the confidence correction matrix is ​​used to realize the effective correction of the image segmentation mask of the current video frame.

[0094] Secondly, since the differential sliding average matrix and the confidence sliding average matrix used by the present application to determine the confidence correction matrix are both the results of the synthesis of the temporal information of the video frame sequence, the temporal confidence correction matrix has stronger smoothness, and the transition between the image segmentation masks obtained by the correction is softer and more accurate, which can remove the sudden mis-segmentation phenomenon, overcome the flickering and instability of the image segmentation results, and especially suppress the continuous instability within a certain period of time. Therefore, according to the present application, the edge recognition effect of object segmentation of video images is clear, accurate and smooth.

[0095] Furthermore, the present application directly processes the image segmentation masks of the video frame sequence, so it can be plug-and-play, that is, it is independent of the image segmentation algorithm of the video frame. The image segmentation mask obtained by any image segmentation algorithm can be expressed as a confidence matrix corresponding to each pixel point, and the technical solution of the present application is used to optimize it, so that the technical solution of the present application obtains a general technical advantage.

[0096] In addition, during the implementation of this application, the differential sliding average matrix and the confidence sliding average matrix are used as the basis, and then the confidence correction matrix is ​​calculated. Then, the confidence matrix of the image segmentation mask of the current video frame is corrected according to the confidence correction matrix. The entire process is implemented using matrix calculations. The complexity of the matrix calculations is very low, and there is no need to use highly complex morphological processing methods such as convolution and inter-frame template matching. Therefore, the amount of calculation is low and the operating efficiency is higher. It can not only accurately identify the target objects in the video frames, but also the recognition speed is very fast. For application scenarios with high timeliness requirements such as live broadcasts, it has obvious scene application advantages.

[0097] See also Figure 2 In some of the further embodiments, based on any embodiment of the present application, in the process of determining the differential sliding average matrix corresponding to each current video frame, according to the uncertainty between the confidence matrices corresponding to the image segmentation masks of the current video frame and its adjacent previous video frames, the weight matching matrix required for calculating the differential sliding average matrix is ​​determined, so as to adjust the corresponding frame difference information according to this uncertainty to obtain a more accurate differential sliding average matrix. Accordingly, before the step S1100, the step of calculating the differential sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, includes the following steps:

[0098] Step S2100: modulate and add the confidence matrices of the image segmentation masks of the current video frame and the adjacent previous video frame to obtain a weight matching matrix for quantifying the uncertainty of whether each pixel of the image segmentation mask of the current video frame belongs to the foreground or the background:

[0099] For the image segmentation mask M of the current video frame i and its adjacent previous video frame, i.e., the image segmentation mask M of the previous video frame i-1 , respectively call the preset modulation function to modulate them, realize the linear mapping between each other, and then perform matrix addition and average of the modulation results between each other, thereby obtaining an uncertainty matrix, which can be used in the form of a weight matching matrix A for the confidence difference matrix Dif between the current video frame and its adjacent previous video frame iThe modulation function may be a cosine function or a Gaussian function, or other functions that can be flexibly applied by those skilled in the art.

[0100] By modulation, the confidence of each pixel between the current video frame and its adjacent video frames is modulated and averaged to form a harmonic confidence, which averages the uncertainty of each pixel belonging to the foreground or background. Therefore, the meaning of the weight matching matrix A is: the image segmentation mask M of the current video frame and its adjacent previous video frame i , M i-1 In the above, the closer the harmonic confidence of a pixel is to the two ends of the confidence interval (0 or 1), the higher the certainty that the pixel belongs to the foreground or background in the current video frame. Therefore, the value of the corresponding element in the weight matching matrix A is lower, that is, more certain. When the harmonic confidence of a pixel is around 0.5, it means that the certainty is lower, and the value of the corresponding element in the weight matching matrix A is larger, that is, more uncertain.

[0101] For example, when a human arm is shaking in a video image, the confidence of a specific pixel on the arm in the current video frame and the previous video frame will suddenly jump from 0 to 1. If the absolute difference between the two is directly calculated, the result is 1. However, in fact, this pixel is a foreground / background with a high degree of certainty and is not an unstable place. If the absolute difference is directly calculated to characterize the frame difference information of the pixel, it may cause misjudgment. Therefore, it is necessary to characterize it through the uncertainty matrix to achieve a better indication effect of the frame difference information.

[0102] Step S2200: Calculate the confidence difference matrix between the image segmentation masks of the current video frame and its adjacent previous video frame, where the confidence difference matrix is ​​the element product of the absolute difference matrix between the confidence matrices corresponding to the image segmentation masks and the weight matching matrix:

[0103] Since the weight matching matrix A represents the image segmentation mask M of the current video frame i Each pixel of the image segmentation mask M of the adjacent previous video frame i-1 The confidence uncertainty of the corresponding pixel in can be used to adjust the reliability of the frame difference value of the pixel in the confidence difference matrix corresponding to the frame difference information of two adjacent image segmentation masks. Based on this, the absolute difference between the image segmentation masks of the current video frame and its adjacent previous video frame is calculated to obtain the corresponding absolute difference matrix |M i -M i-1 |, and then perform element-wise multiplication operation with the weight matching matrix, so that the element-wise multiplication between the two is calculated based on each pixel point to obtain a new matrix as the confidence difference matrix Dif i. Based on this, this process can be expressed as the following formula:

[0104] Dif i =A×|M i -M i-1 |

[0105] According to this embodiment, when calculating the differential sliding movement matrix In addition to directly using the absolute difference matrix of the image segmentation masks of two adjacent video frames as the confidence difference matrix for calculating the differential sliding average matrix, the confidence difference matrix can be further associated with the weight matching matrix on the basis of the absolute difference matrix to obtain the confidence difference matrix for calculating the differential sliding average matrix. Since the latter is associated with the uncertainty of the same moving pixel between the previous and next video frames, a differential sliding movement matrix that more accurately represents the changing relationship of each pixel in time sequence can be obtained, thereby achieving accurate representation of the frame difference information between the previous and next video frames.

[0106] In some further embodiments, an exponential moving average algorithm may be applied to calculate the differential sliding average matrix based on the above embodiments. Accordingly, the step S1100, calculating the confidence sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, includes the following steps:

[0107] Step S3100: According to the exponential moving average algorithm, the confidence difference matrix of the image segmentation mask between the current video frame and its adjacent previous video frame and the differential sliding average matrix of the adjacent previous video frame are called, and the confidence difference matrix is ​​used to store the confidence difference between each corresponding pixel point of the image segmentation mask between the current video frame and its adjacent previous video frame:

[0108] According to various embodiments of the present application, for each current video frame, the image segmentation mask M i , both the image segmentation mask M of the previous video frame and its image segmentation mask M can be obtained. i-1 The confidence difference matrix Dif between i , the confidence difference matrix Dif i It can be the absolute difference information obtained without associating the weight matching matrix A as described above, or the difference information obtained by associating the weight matching matrix A, and the differential sliding average matrix corresponding to the previous video frame can also be obtained Therefore, in theory, when calculating the differential sliding average matrix corresponding to the current video frame When , the exponential moving average formula can be used for calculation:

[0109]

[0110] Among them, EMAd Is a hyperparameter, which is the preset smoothing weight.

[0111] According to this formula, it is only necessary to calculate the differential sliding average matrix corresponding to the previous video frame. The confidence difference matrix Dif corresponding to the current video frame i , associated with the smoothing weight ema d By smoothly fusing the two matrices, we can obtain the differential sliding average matrix corresponding to the current video frame. Similarly, the differential sliding average matrix of the current video frame is It can be cached for the next video frame to call, and so on.

[0112] It should be pointed out that the exponential moving average algorithm is essentially an algorithm that performs moving weighted average of similar information at each time point in a time-series manner. Therefore, according to this formula, the differential moving average matrix corresponding to each current video frame can be deduced: In essence, it is obtained by performing a moving weighted average calculation based on the confidence difference matrix corresponding to the current video frame and each previous video frame. This embodiment uses this formula for simplification, which can facilitate calculation and improve calculation efficiency.

[0113] Step S3200: Apply the exponential moving average algorithm to obtain the moving average result of the confidence difference matrix of the current video frame and its previous video frame with a preset smoothing weight:

[0114] The confidence difference matrix Dif obtained in the previous step i With the difference moving average matrix By inserting the formula of the above-mentioned exponential moving average algorithm for calculation, under the effect of the smoothing weight, the statistical synthesis of the frame difference information of the current video frame and its multiple previous video frames represented by the confidence difference matrix can be achieved, and the corresponding moving average result can be obtained, which can be used as the differential sliding average matrix corresponding to the current video frame.

[0115] According to this embodiment, the differential moving average matrix is ​​determined by applying an exponential moving average algorithm. By utilizing the characteristic that the closer the exponential moving average algorithm is to the current video frame, the greater its weight is, the differential moving average matrix of the current video frame can be effectively determined, and the dynamic information of each pixel point in the current video frame and its previous video frames in the video frame sequence can be effectively and comprehensively represented, so that the basis for correcting the image segmentation mask can be more accurate.

[0116] In some further embodiments, an exponential moving average algorithm may be applied to calculate the confidence sliding average matrix based on the above embodiments. Accordingly, the step S1200, calculating the confidence sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, includes the following steps:

[0117] Step S4100: according to the exponential moving average algorithm, call the image segmentation mask of the current video frame and the confidence sliding average matrix corresponding to the previous video frame, wherein the image segmentation mask is represented by a confidence matrix, and the confidence matrix is ​​used to store the confidence of each pixel point of the image segmentation mask;

[0118] According to various embodiments of the present application, for each image segmentation mask of the current video frame, its confidence matrix M can be obtained: i , and the confidence sliding average matrix corresponding to the previous video frame can also be obtained Therefore, in theory, when calculating the confidence sliding average matrix corresponding to the current video frame When , the exponential moving average formula can be used for calculation:

[0119]

[0120] Among them, EMA m Is a hyperparameter, which is the preset smoothing weight.

[0121] According to this formula, it is only necessary to calculate the confidence sliding average matrix corresponding to the previous video frame. The confidence matrix M corresponding to the current video frame i , associated with the smoothing weight ema m By smoothly fusing the two matrices, we can obtain the confidence sliding average matrix corresponding to the current video frame. Similarly, the confidence sliding average matrix of the current video frame It can be cached for the next video frame to call, and so on.

[0122] According to the principle of the exponential moving average algorithm, the confidence sliding average matrix corresponding to each current video frame can be deduced In essence, it is obtained by performing a moving weighted average calculation based on the confidence matrices corresponding to the current video frame and each of the previous video frames. This embodiment uses this formula for simplification, which can facilitate calculation and improve calculation efficiency.

[0123] Step S4200: Apply the exponential moving average algorithm to obtain the moving average result of the confidence matrix of the current video frame and its previous video frame with a preset smoothing weight to obtain the confidence average matrix:

[0124] The confidence matrix M obtained in the previous step i and the confidence sliding average matrix By inserting the formula of the above-mentioned exponential moving average algorithm for calculation, under the effect of the smoothing weight, the statistical synthesis of the confidence matrices of the current video frame and its multiple previous video frames can be achieved, and the corresponding moving average result can be obtained, which can be used as the confidence sliding average matrix corresponding to the current video frame.

[0125] According to this embodiment, the exponential moving average algorithm is applied to determine the confidence sliding average matrix. By utilizing the characteristic that the closer the exponential moving average algorithm is to the current video frame, the greater the weight is, the confidence sliding average matrix of the current video frame can be effectively determined, and the static information of each pixel point in the current video frame and its previous video frames in the video frame sequence can be effectively and comprehensively represented, so that the basis for correcting the image segmentation mask can be more accurate.

[0126] In some of the further embodiments, the embodiments of the present application are further deepened. Accordingly, the step S1300, determining the confidence correction matrix of the current video frame according to the confidence sliding average matrix and the differential sliding average matrix, includes the following steps:

[0127] Step S5100: Confidence compensation is performed on the confidence moving average matrix to obtain a confidence compensation matrix:

[0128] For the current video frame, in order to make its confidence sliding average matrix The comprehensive result of the static information of the image segmentation mask of each relevant video frame is more effectively represented, and appropriate confidence compensation can be performed on the confidence corresponding to each pixel point therein.

[0129] In one embodiment, a linear function is applied Perform linear confidence compensation. The application of this function makes the confidence sliding average matrix The lower the average confidence of each pixel in , the higher the degree of suppression, which indirectly enhances the salient features of pixels with higher average confidence.

[0130] In another embodiment, for the segmentation task, a median value of the confidence interval, such as 0.5, is usually set as a confidence threshold. Based on this, a pixel whose confidence is higher than the confidence threshold is the foreground, and a pixel whose confidence is lower than the confidence threshold is the background. Therefore, for a foreground pixel with a confidence of, for example, 0.7, no compensation is required; but for those cases where the confidence has been 0.7 (foreground) and suddenly jumps to 0.4 (background), confidence compensation plays a role in temporal stabilization. Based on this, another nonlinear function can be applied: To achieve nonlinear confidence compensation, a confidence threshold can be preset. When the confidence sliding average matrix When the average confidence of the pixel in is higher than the confidence threshold, the first constant is used to encourage the confidence of the pixel to achieve positive compensation for the confidence of the pixel. When the average confidence of the pixel is less than or equal to the confidence threshold, the second constant is used to suppress the confidence of the pixel to achieve negative compensation of the confidence of the pixel, so as to control the jump phenomenon of each pixel and make the visual segmentation effect smooth in time sequence.

[0131] In addition, those skilled in the art can also flexibly apply specific modulation functions and customize confidence compensation schemes according to the principles disclosed herein to achieve the purpose of visually smooth transition in timing.

[0132] So far, it can be known that after the confidence compensation in this step, the corresponding confidence compensation matrix can be obtained

[0133] Step S5200: Calculate the element product matrix of the confidence compensation matrix and the differential sliding average matrix:

[0134] Get the confidence compensation matrix After that, the comprehensive static information of the current video frame and its multiple previous video frames is corrected, and the differential sliding average matrix corresponding to the current video frame can be further obtained. Perform matrix element product operation to integrate static information and dynamic information of each pixel of the current video frame to obtain an element product matrix, that is:

[0135]

[0136] Step S5300: Apply a constant suppression factor to adjust the correction strength represented by the element product matrix to obtain a confidence correction matrix:

[0137] Furthermore, in order to control the correction strength represented by the element product matrix, a preset constant suppression factor is used to correct the element product matrix to obtain the confidence correction matrix Sup i ,Right now:

[0138]

[0139] It can be understood that in this embodiment, confidence compensation is first performed on the confidence sliding average matrix corresponding to the current video frame to obtain a confidence compensation matrix, and then the confidence compensation matrix and the differential sliding average matrix are multiplied to obtain an element product matrix that integrates two-way time series information. The element product matrix realizes the statistical integration of static information and dynamic information in time series. On this basis, a constant suppression factor is further used to adjust the correction strength of the element product matrix. The confidence correction matrix finally obtained will inevitably be able to effectively soften the change of the image segmentation mask of the current video frame corresponding to the image segmentation mask of the previous video frame, so that the edge contour of the object segmented from the video image is clearer and smoother.

[0140] In some further embodiments, based on the various embodiments of the present application, the image segmentation mask of the current video frame may be corrected according to the confidence correction matrix. To this end, the step S1400, correcting the image segmentation mask of the current video frame according to the confidence correction matrix, includes the following steps:

[0141] Step S6100: Obtain the confidence matrix corresponding to the image segmentation mask of the current video frame:

[0142] In order to correct the image segmentation mask of the current video frame, the confidence matrix corresponding to the image segmentation mask, i.e., M i Here, the image segmentation mask is the image segmentation mask output by the image segmentation model or other image segmentation algorithm and input into the various embodiments of the present application for calculation.

[0143] Step S6200: Calculate the sum matrix of the confidence matrix and the confidence correction matrix:

[0144] In order to realize the use of the confidence correction matrix Sup i For the confidence matrix M i The fusion of the two is performed by adding the corresponding elements to obtain the sum matrix Sum i That is: Sum i =Sup i +M i , it is not difficult to understand that the sum matrix Sum i The confidence correction matrix Sup obtained by integrating the two-way timing information i The correction of the image segmentation mask of the current video frame is realized.

[0145] Step S6300: trim the value of each element in the sum matrix to a confidence interval to obtain a modified image segmentation mask:

[0146] The confidence values ​​corresponding to the elements in the sum matrix may exceed the original confidence interval [0, 1]. Therefore, they can be pruned and the elements exceeding the confidence interval are set to 0 or 1, so that the values ​​of the elements are maintained within the confidence interval, and the corrected image segmentation mask is obtained, which can be specifically expressed by the following formula:

[0147] M′ i =Min(Max(M i +Sup i ,0),1)

[0148] According to the formula, when the confidence of a pixel in the sum matrix is ​​lower than 0, it is reset to 0; when the confidence of a pixel is higher than 0, it is reset to 1. The values ​​within the confidence interval are retained, and the matrix M′ is obtained. i , which is the confidence matrix corresponding to the modified image segmentation mask, can be used to implement other downstream tasks on the video stream.

[0149] In some embodiments of the present application, the image segmentation mask correction method of the present application can be initialized by the following steps:

[0150] Step S7100: construct a differential sliding average matrix corresponding to the first video frame in the video frame sequence, and assign values ​​to all elements of the differential sliding average matrix using arbitrarily determined initial values:

[0151] When the result of image segmentation of a video frame sequence needs to be corrected, a differential sliding average matrix can be created for the first video frame in the video frame sequence. The differential sliding average matrix sets corresponding elements corresponding to the width and height of the video frame, and each element corresponds to a pixel point of the video frame. Since there is no differential sliding average matrix before the first video frame, and the first video frame needs to provide reference data for determining the corresponding differential sliding average matrix for the subsequent video frames, 0 can be used to assign values ​​to all elements of the differential sliding average matrix of the current first video frame to complete the initialization of the differential sliding average matrix.

[0152] Step S7200: Using the confidence matrix corresponding to the image segmentation mask of the first video frame in the video frame sequence as the confidence sliding average matrix corresponding to the first video frame:

[0153] Similarly, corresponding to the first video frame, a confidence sliding average matrix needs to be created to provide reference data for subsequent video frames. Based on this, the confidence matrix corresponding to the image segmentation mask of the first video frame can be directly used as the confidence sliding average matrix of the first video frame.

[0154] Step S7300: skip the first video frame, and start from the next adjacent video frame, and perform the other steps of the method frame by frame as the current video frame:

[0155] After completing the initialization process of the above two steps, there is no need to apply the processing procedures of the previous embodiments of the present application to perform the image segmentation mask correction operation on the first video frame. Instead, the image segmentation mask of the first video frame is directly used as its corrected image segmentation mask. Then, starting from the second video frame, each is used as the current video frame of the present application, and any one of the embodiments of the present application is applied to execute the corresponding steps adopted by the embodiment on the current video frame to correct its image segmentation mask, thereby realizing the correction of the image segmentation mask of each video frame in the video frame sequence except the first video frame frame by frame.

[0156] This embodiment skips the first video frame in the video frame sequence, uses the confidence matrix of the first video frame itself as the starting point in the confidence sliding average matrix sequence, and uses a differential sliding average matrix assigned a value of 0 as a guide, so that the correction work of the image segmentation masks of subsequent video frames in the entire video frame sequence can be quickly carried out with reference.

[0157] Figure 3 An algorithm framework example implemented to integrate multiple implementations of this application, according to this example:

[0158] On the one hand, for a current video frame, take its image segmentation mask M i The image segmentation mask M of the previous frame i-1 The uncertainty matrix is ​​obtained by modulating and adding them respectively, and used as the weight matching matrix A to calculate the confidence difference matrix Dif corresponding to the frame difference information between the two frames i =A×|M i -M i-1 |.

[0159] On the other hand, obtain the differential sliding average matrix corresponding to the previous video frame and the confidence difference matrix Dif i Implement moving average to achieve weighted, thereby updating the differential sliding average matrix corresponding to the current video frame

[0160] On the other hand, obtain the confidence sliding average matrix corresponding to the previous video frame The confidence matrix M corresponding to the image segmentation mask of the current video frame i The confidence sliding average matrix corresponding to the current video frame is obtained by taking the moving average between

[0161] Finally, using the formula The differential sliding average matrix corresponding to the current video frame and the confidence sliding average matrix Perform synthesis, and then perform confidence pruning on the basis of the synthesis to obtain the corrected image segmentation mask M′ i . Clip means adjusting each confidence level in the calculation result to a confidence interval.

[0162] This example is used as a reference. When performing image segmentation mask correction on live video streams on the Internet, it is tested to obtain excellent performance in many aspects, such as Figure 4 As shown in the figure, the figure is a continuous video frame in time sequence from top to bottom. In each row, the left side is the original image, the middle is the unoptimized binary image segmentation mask, and the right side is the image segmentation mask obtained by the algorithm optimization here. Figure 4 It can be seen that this application can prevent the image segmentation effect of the live video stream from flickering, and occupies less system memory, thereby ensuring the smooth operation of the terminal device, thereby facilitating technical personnel in the field to quickly implement the technical solution of the present application, but it should not be understood as the only implementation method of the present application. Technical personnel in the field can flexibly implement it according to the principles disclosed in the present application and its various embodiments.

[0163] See also Figure 5 , an image segmentation mask correction device provided to meet one of the purposes of the present application, comprising: a differential calculation module 1100, a confidence calculation module 1200, a suppression calculation module 1300, and a mask correction module 1400, wherein: the differential calculation module 1100 is used to calculate the differential sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, and the differential sliding average matrix is ​​determined according to the moving average result of the confidence difference matrix of the image segmentation mask between two adjacent video frames in the current video frame and a plurality of previous video frames; the confidence calculation module 1200 is used to calculate the differential sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, and the confidence calculation module 1200 is used to calculate the differential sliding average matrix of the image segmentation mask between two adjacent video frames in the current video frame and a plurality of previous video frames; ...; the confidence calculation module 1200 is used to calculate the differential sliding average matrix of the image segmentation mask between two adjacent video frames; the confidence calculation module 1200 is used to calculate the differential sliding average matrix of The calculation module 1200 is used to calculate the confidence sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, and the confidence sliding average matrix is ​​determined according to the moving average result of the confidence of each pixel point in the image segmentation mask of the current video frame and its multiple previous video frames; the suppression calculation module 1300 is used to determine the confidence correction matrix of the current video frame according to the confidence sliding average matrix and the differential sliding average matrix; the mask correction module 1400 is used to correct the image segmentation mask of the current video frame according to the confidence correction matrix.

[0164] In a further embodiment, the image segmentation mask correction device of the present application also includes: a weight calculation module, which is used to modulate and add the confidence matrices of the image segmentation masks of the current video frame and its adjacent previous video frame to obtain a weight matching matrix, which is used to quantify the uncertainty of whether each pixel point of the image segmentation mask representing the current video frame belongs to the foreground or background; a frame difference constraint module, which is used to calculate the confidence difference matrix between the image segmentation masks of the current video frame and its adjacent previous video frame, and the confidence difference matrix is ​​the element product of the absolute difference matrix between the confidence matrices corresponding to the image segmentation masks of each other and the weight matching matrix.

[0165] In a further embodiment, the differential calculation module 1100 includes: a differential preparation submodule, which is used to call the confidence difference matrix of the image segmentation mask between the current video frame and its adjacent previous video frame and the differential sliding average matrix of the adjacent previous video frame according to the exponential moving average algorithm, and the confidence difference matrix is ​​used to store the confidence difference between each corresponding pixel point of the image segmentation mask between the current video frame and its adjacent previous video frame; a differential average submodule, which is used to apply the exponential moving average algorithm to obtain the moving average result of the confidence difference matrix of the current video frame and its previous video frame with a preset smoothing weight.

[0166] In a further embodiment, the confidence calculation module 1200 includes: a confidence preparation submodule, which is used to call the image segmentation mask of the current video frame and the confidence sliding average matrix corresponding to the previous video frame according to the exponential moving average algorithm, and the image segmentation mask is represented by a confidence matrix, and the confidence matrix is ​​used to store the confidence of each pixel point of the image segmentation mask; a confidence average submodule, which is used to apply the exponential moving average algorithm to obtain the moving average result of the confidence matrix of the current video frame and its previous video frame with a preset smoothing weight to obtain the confidence average matrix.

[0167] In some further embodiments, the suppression calculation module 1300 includes: a modulation implementation submodule, used to perform confidence compensation on the confidence sliding average matrix to obtain a confidence compensation matrix; a modulation suppression submodule, used to calculate the element product matrix of the confidence compensation matrix and the differential sliding average matrix; a constant suppression submodule, used to apply a constant suppression factor to adjust the correction intensity represented by the element product matrix to obtain a confidence correction matrix.

[0168] In some further embodiments, the mask correction module 1400 includes: a mask calling submodule, used to obtain the confidence matrix corresponding to the image segmentation mask of the current video frame; a difference calculation submodule, used to calculate the sum matrix of the confidence matrix and the confidence correction matrix; a normalization correction submodule, used to trim the values ​​of each element in the sum matrix to a confidence interval to obtain a corrected image segmentation mask.

[0169] In some extended embodiments, the image segmentation mask correction device of the present application includes the following initialization modules: a differential initialization module, used to construct a differential sliding average matrix corresponding to the first video frame in the video frame sequence, and to assign all elements of the differential sliding average matrix with arbitrarily determined initial values; a confidence initialization module, used to use the confidence matrix corresponding to the image segmentation mask of the first video frame in the video frame sequence as the confidence sliding average matrix corresponding to the first video frame; a subsequent execution module, used to skip the first video frame, and start from the next adjacent video frame, running each other module of the device frame by frame as the current video frame.

[0170] In order to solve the above technical problems, the present application also provides a computer device. Figure 6 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement an image segmentation mask correction method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the image segmentation mask correction method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] In this embodiment, the processor is used to execute Figure 5The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the image segmentation mask correction device of the present application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0172] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the image segmentation mask correction method of any embodiment of the present application.

[0173] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0175] In summary, the technical solution of the present application can optimize the video segmentation results generated by any segmentation model or other methods, remove sudden mis-segmentation, suppress the flickering and instability of the segmentation results, and especially suppress the continuous instability within a certain period of time. On the other hand, all the operators used in the present application are low-complexity matrix element-by-element calculations, and do not use convolution, template matching between frames and other high-complexity morphological processing methods, so the amount of calculation is relatively low. In addition, the present application is also particularly suitable for application scenarios such as live webcasts that have high timeliness requirements for image segmentation. Therefore, the technical solution of the present application has the advantages of plug-and-play, high versatility, and low computational complexity.

[0176] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0177] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image segmentation mask correction method, characterized in that: The steps include: Calculating a differential sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, wherein the differential sliding average matrix is ​​determined according to a moving average result of confidence difference matrices of image segmentation masks between two adjacent video frames in the current video frame and a plurality of previous video frames; Calculating a confidence sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, where the confidence sliding average matrix is ​​determined based on a moving average result of confidences of each pixel in the image segmentation mask of the current video frame and a plurality of previous video frames; Determining a confidence correction matrix of the current video frame according to the confidence sliding average matrix and the differential sliding average matrix, including: performing confidence compensation on the confidence sliding average matrix to obtain a confidence compensation matrix; calculating an element product matrix of the confidence compensation matrix and the differential sliding average matrix; applying a constant suppression factor to adjust the correction strength represented by the element product matrix to obtain a confidence correction matrix; The image segmentation mask of the current video frame is corrected according to the confidence correction matrix.

2. The image segmentation mask correction method according to claim 1, characterized in that: Before the step of calculating the differential sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence, the following steps are included: The confidence matrices of the image segmentation masks of the current video frame and the previous video frame adjacent thereto are modulated and added to obtain a weight matching matrix for quantifying the uncertainty of whether each pixel of the image segmentation mask of the current video frame belongs to the foreground or the background; A confidence difference matrix between the image segmentation masks of the current video frame and its adjacent previous video frame is calculated, where the confidence difference matrix is ​​the element product of the absolute difference matrix between the confidence matrices corresponding to the image segmentation masks and the weight matching matrix.

3. The image segmentation mask correction method according to claim 1, characterized in that: Calculating the confidence sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence includes the following steps: According to the exponential moving average algorithm, calling the confidence difference matrix of the image segmentation mask between the current video frame and its adjacent previous video frame and the differential sliding average matrix of the adjacent previous video frame, wherein the confidence difference matrix is ​​used to store the difference in confidence between each corresponding pixel point of the image segmentation mask between the current video frame and its adjacent previous video frame; An exponential moving average algorithm is applied to obtain a moving average result of the confidence difference matrix of the current video frame and its previous video frame with a preset smoothing weight.

4. The image segmentation mask correction method according to claim 1, characterized in that: Calculating the confidence sliding average matrix of the image segmentation mask of the current video frame in the video frame sequence includes the following steps: According to the exponential moving average algorithm, the image segmentation mask of the current video frame and the confidence sliding average matrix corresponding to the previous video frame are called, the image segmentation mask is represented by a confidence matrix, and the confidence matrix is ​​used to store the confidence of each pixel point of the image segmentation mask; An exponential moving average algorithm is applied to obtain a moving average result of the confidence matrix of the current video frame and its previous video frame with a preset smoothing weight to obtain the confidence average matrix.

5. The image segmentation mask correction method according to claim 1, characterized in that: Correcting the image segmentation mask of the current video frame according to the confidence correction matrix comprises the following steps: Get the confidence matrix corresponding to the image segmentation mask of the current video frame; Calculating a sum matrix of the confidence matrix and the confidence correction matrix; The value of each element in the sum matrix is ​​trimmed to a confidence interval to obtain a modified image segmentation mask.

6. The image segmentation mask correction method according to any one of claims 1 to 5, characterized in that: It includes the following initialization steps: Constructing a differential sliding average matrix corresponding to the first video frame in the video frame sequence, and assigning values ​​to all elements of the differential sliding average matrix by arbitrarily determining initial values; Using the confidence matrix corresponding to the image segmentation mask of the first video frame in the video frame sequence as the confidence sliding average matrix corresponding to the first video frame; The first video frame is skipped, and starting from the next adjacent video frame, other steps of the method are executed frame by frame as the current video frame.

7. An image segmentation mask correction device, characterized in that: include: A differential calculation module, used to calculate a differential sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, wherein the differential sliding average matrix is ​​determined based on a moving average result of a confidence difference matrix of an image segmentation mask between two adjacent video frames in the current video frame and a plurality of previous video frames thereof; A confidence calculation module, used to calculate a confidence sliding average matrix of an image segmentation mask of a current video frame in a video frame sequence, wherein the confidence sliding average matrix is ​​determined based on a moving average result of confidences of each pixel in the image segmentation mask of the current video frame and a plurality of previous video frames; A suppression calculation module is used to determine the confidence correction matrix of the current video frame according to the confidence sliding average matrix and the differential sliding average matrix, including: performing confidence compensation on the confidence sliding average matrix to obtain a confidence compensation matrix; calculating an element product matrix of the confidence compensation matrix and the differential sliding average matrix; applying a constant suppression factor to adjust the correction strength represented by the element product matrix to obtain a confidence correction matrix; The mask correction module is used to correct the image segmentation mask of the current video frame according to the confidence correction matrix.

8. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images