Foreground image acquisition method, foreground image acquisition device and electronic device

Abstract

The foreground image acquisition method, foreground image acquisition device and electronic device provided in the present application relate to the field of image processing technology. The foreground image acquisition method includes: performing inter-frame motion detection on the current video frame to obtain a first mask image; recognizing the current video frame through a neural network model to obtain a second mask image; and calculating the foreground image in the current video frame based on a preset calculation model, the first mask image and the second mask image. The above method can improve the problem that it is difficult to accurately and effectively extract the foreground image of the video frame using the existing foreground extraction technology.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a foreground image acquisition method, a foreground image acquisition device and an electronic device. Background Art

[0002] In some applications of image processing, it is necessary to extract foreground images. Among them, some common foreground image extraction techniques include inter-frame difference method, background difference method, ViBe algorithm, etc. The inventors have found through research that the above foreground image extraction techniques are difficult to accurately and effectively extract foreground images from video frames. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a foreground image acquisition method, a foreground image acquisition device and an electronic device to improve the problem that it is difficult to accurately and effectively extract foreground images from video frames using existing foreground extraction technologies.

[0004] To achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0005] A foreground image acquisition method, comprising:

[0006] Performing inter-frame motion detection on the obtained current video frame to obtain a first mask image;

[0007] Recognizing the current video frame through a neural network model to obtain a second mask image;

[0008] A foreground image in the current video frame is calculated based on a preset calculation model, the first mask image, and the second mask image.

[0009] In a preferred embodiment of the present application, in the foreground image acquisition method, the step of performing inter-frame motion detection on the obtained current video frame to obtain the first mask image includes:

[0010] Calculate the boundary information of each pixel point in the current video frame according to the obtained pixel value of each pixel point in the current video frame;

[0011] It is determined whether each pixel belongs to a foreground boundary point according to the boundary information of each pixel, and a first mask image is obtained according to the mask values ​​of each pixel belonging to the foreground boundary point.

[0012] In a preferred embodiment of the present application, in the foreground image acquisition method, the step of determining whether the pixel belongs to a foreground boundary point according to the boundary information of each pixel, and obtaining a first mask image according to the mask value of each pixel belonging to the foreground boundary point, includes:

[0013] For each pixel, determine the current mask value and the current frequency value of the pixel according to the boundary information of the pixel in the current video frame, the boundary information in the previous N video frames, and the boundary information in the previous M video frames, where N is not equal to M;

[0014] For each pixel point, it is determined whether the pixel point belongs to a foreground boundary point according to the current mask value and the current frequency value, and a first mask image is obtained according to the current mask value of each pixel point belonging to the foreground boundary point.

[0015] In a preferred embodiment of the present application, in the foreground image acquisition method, the neural network model includes a first network sub-model, a second network sub-model and a third network sub-model;

[0016] The step of identifying the current video frame by a neural network model to obtain a second mask image includes:

[0017] Performing semantic information extraction processing on the current video frame through the first network sub-model to obtain a first output value;

[0018] Performing a size adjustment process on the first output value by using the second network sub-model to obtain a second output value;

[0019] The second output value is subjected to mask image extraction processing by the third network sub-model to obtain a second mask image.

[0020] In a preferred embodiment of the present application, in the foreground image acquisition method, the method further includes the step of pre-constructing the first network sub-model, the second network sub-model and the third network sub-model, the step comprising:

[0021] The first network sub-model is constructed by a first convolutional layer, a plurality of second convolutional layers, and a plurality of third convolutional layers, wherein the first convolutional layer is used to perform one convolution operation, the second convolutional layer is used to perform two convolution operations, one depth-separable convolution operation, and two activation operations, and the third convolutional layer is used to perform two convolution operations, one depth-separable convolution operation, and two activation operations, and outputs the values ​​obtained by the operation together with the input values;

[0022] constructing the second network sub-model through the first convolutional layer and multiple fourth convolutional layers, wherein the fourth convolutional layer is used to perform one convolution operation, one depth-separable convolution operation and two activation operations;

[0023] The third network sub-model is constructed by multiple fourth convolutional layers and multiple upsampling layers, wherein the upsampling layers are used to perform a bilinear difference upsampling operation.

[0024] In a preferred embodiment of the present application, in the foreground image acquisition method, the step of calculating and obtaining the foreground image in the current video frame based on a preset calculation model, the first mask image and the second mask image includes:

[0025] Performing weighted sum processing on the first mask image and the second mask image according to a preset first weighting coefficient and a second weighting coefficient;

[0026] The result obtained by weighted summation is summed with a predetermined parameter to obtain a foreground image in the current video frame.

[0027] In a preferred embodiment of the present application, in the foreground image acquisition method, before executing the step of calculating the foreground image in the current video frame based on the preset calculation model, the first mask image and the second mask image, the method further includes:

[0028] Calculating a first difference between the first mask image of the current video frame and the first mask image of the previous video frame, and calculating a second difference between the second mask image of the current video frame and the second mask image of the previous video frame;

[0029] If the first difference is less than a preset difference, updating the first mask image of the current video frame to the first mask image of the previous video frame;

[0030] If the second difference is less than a preset difference, the second mask image of the current video frame is updated to the second mask image of the previous video frame.

[0031] In a preferred embodiment of the present application, in the foreground image acquisition method, the step of calculating a first difference between the first mask image of the current video frame and the first mask image of the previous video frame, and calculating a second difference between the second mask image of the current video frame and the second mask image of the previous video frame, comprises:

[0032] Performing inter-frame smoothing processing on the first mask image of the current video frame to obtain a new first mask image, and performing inter-frame smoothing processing on the second mask image of the current video frame to obtain a new second mask image;

[0033] Calculating a first difference between the new first mask image and the first mask image of the previous video frame, and calculating a second difference between the new second mask image and the second mask image of the previous video frame;

[0034] The foreground image acquisition method further includes:

[0035] If the first difference is greater than or equal to a preset difference, updating the first mask image of the current video frame to the new first mask image;

[0036] If the second difference is greater than or equal to a preset difference, the second mask image of the current video frame is updated to the new second mask image.

[0037] In a preferred embodiment of the present application, in the foreground image acquisition method, the step of performing inter-frame smoothing on the first mask image of the current video frame to obtain a new first mask image, and performing inter-frame smoothing on the second mask image of the current video frame to obtain a new second mask image, comprises:

[0038] Calculating a first mean value of the first mask images of all video frames before the current video frame, and calculating a second mean value of the second mask images of all video frames;

[0039] A new first mask image is calculated based on the first mean value and the first mask image of the current video frame, and a new second mask image is calculated based on the second mean value and the second mask image of the current video frame.

[0040] In a preferred embodiment of the present application, in the foreground image acquisition method, the step of calculating a first difference between the new first mask image and the first mask image of the previous video frame, and calculating a second difference between the new second mask image and the second mask image of the previous video frame, comprises:

[0041] Determining whether the connected regions belong to the first target region according to the area of ​​each connected region in the new first mask image, and determining whether the connected regions belong to the second target region according to the area of ​​each connected region in the new second mask image;

[0042] Calculating first barycentric coordinates of a connected area belonging to the first target area, and updating the barycentric coordinates of the new first mask image to the first barycentric coordinates;

[0043] Calculating the second barycentric coordinates of the connected area belonging to the second target area, and updating the barycentric coordinates of the new second mask image to the second barycentric coordinates;

[0044] A first difference between the first barycentric coordinates and the barycentric coordinates of the first mask image of the previous video frame is calculated, and a second difference between the second barycentric coordinates and the barycentric coordinates of the second mask image of the previous video frame is calculated.

[0045] The embodiment of the present application also provides a foreground image acquisition device, including:

[0046] A first mask image acquisition module, used for performing inter-frame motion detection on the obtained current video frame to obtain a first mask image;

[0047] A second mask image acquisition module, used for identifying the current video frame through a neural network model to obtain a second mask image;

[0048] The foreground image acquisition module is used to calculate the foreground image in the current video frame according to a preset calculation model, the first mask image and the second mask image.

[0049] Based on the above, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the foreground image acquisition method when running on the processor.

[0050] On the basis of the above, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed, the foreground image acquisition method described above is implemented.

[0051] The foreground image acquisition method, foreground image acquisition device and electronic device provided by the present application respectively perform inter-frame motion detection and neural network recognition on the same video frame, and calculate the foreground image in the video frame based on the obtained first mask image and second mask image. In this way, the basis for calculating the foreground image is increased, thereby improving the accuracy and effectiveness of the calculation results, thereby improving the problem that it is difficult to accurately and effectively extract the foreground image of the video frame using the existing foreground extraction technology, and having high practical value.

[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application.

[0054] Figure 2 A schematic diagram of application interaction of an electronic device provided in an embodiment of the present application.

[0055] Figure 3 A schematic diagram of the process of obtaining a foreground image according to an embodiment of the present application.

[0056] Figure 4 for Figure 3 Schematic diagram of the flow chart of step S110 in .

[0057] Figure 5 A structural block diagram of a neural network model provided in an embodiment of the present application.

[0058] Figure 6 This is a structural block diagram of the second convolutional layer provided in an embodiment of the present application.

[0059] Figure 7 This is a structural block diagram of the third convolutional layer provided in an embodiment of the present application.

[0060] Figure 8 This is a structural block diagram of the fourth convolutional layer provided in an embodiment of the present application.

[0061] Fig. 9 A flowchart of other steps included in the foreground image acquisition method provided in an embodiment of the present application.

[0062] Fig.10 for Fig. 9 Schematic diagram of the process of step S140 in .

[0063] Fig.11 A schematic diagram of the effect of calculating area ratio provided in an embodiment of the present application.

[0064] Fig.12 A block diagram of the functional modules included in the foreground image acquisition device provided in an embodiment of the present application.

[0065] Icons: 10 - electronic device; 12 - memory; 14 - processor; 100 - foreground image acquisition device; 110 - first mask image acquisition module; 120 - second mask image acquisition module; 130 - foreground image acquisition module. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0067] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0068] like Figure 1 As shown, an embodiment of the present application provides an electronic device 10 , which may include a memory 12 , a processor 14 , and a foreground image acquisition device 100 .

[0069] In detail, the memory 12 and the processor 14 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, they can be electrically connected to each other through one or more communication buses or signal lines. The foreground image acquisition device 100 includes at least one software function module that can be stored in the memory 12 in the form of software or firmware. The processor 14 is used to execute an executable computer program stored in the memory 12, for example, the software function module and computer program included in the foreground image acquisition device 100, so as to implement the foreground image acquisition method provided in the embodiment of the present application.

[0070] Among them, the memory 12 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0071] The processor 14 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0072] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 10 may also include Figure 1 More or fewer components as shown, or with Figure 1 The different configurations shown, for example, may also include a communication unit for exchanging information with other devices.

[0073] The specific type of the electronic device 10 is not limited, for example, it can be a terminal device with better data processing performance or a server.

[0074] In an alternative example, the electronic device 10 can be used as a live broadcast device, for example, it can be a terminal device used by the anchor during live broadcast, or it can be a background server that is communicatively connected to the terminal device used by the anchor during live broadcast.

[0075] When the electronic device 10 is used as a background server, Figure 2 As shown, the image acquisition device can send the video frames acquired by the anchor to the anchor's terminal device, and the terminal device can send the video frames to the background server for processing.

[0076] Combination Figure 3 The embodiment of the present application also provides a foreground image acquisition method applicable to the electronic device 10. The method steps defined in the process related to the foreground image acquisition method can be implemented by the electronic device 10. Figure 3 The specific process shown is described in detail.

[0077] Step S110 , performing inter-frame motion detection on the obtained current video frame to obtain a first mask image.

[0078] Step S120: Recognize the current video frame through a neural network model to obtain a second mask image.

[0079] Step S130: Calculate and obtain the foreground image in the current video frame based on a preset calculation model, the first mask image, and the second mask image.

[0080] Through the above method, based on the first mask image and the second mask image obtained by executing step S110 and step S120, the calculation basis can be increased when calculating the foreground image in executing step S130, so that the accuracy and effectiveness of the calculation result can be improved, thereby improving the problem that it is difficult to accurately and effectively obtain the foreground image of the video frame using the existing foreground extraction technology. The inventors of the present application have found that, especially in some cases (such as when obtaining video frames, if there are flashing lights, lens shaking, lens zooming, the subject is still, etc.), compared with some existing foreground image technologies, the foreground image acquisition method improved by the embodiment of the present application has better results.

[0081] It should be noted that the order of the above-mentioned step S110 and step S120 is not limited. For example, step S110 may be performed first, or step S120 may be performed first, or step S110 and step S120 may be performed simultaneously.

[0082] Optionally, the method of executing step S110 to obtain the first mask image based on the current video frame is not limited and can be selected according to actual application requirements.

[0083] For example, in an alternative example, the first mask image may be obtained by calculating the pixel value of each pixel point in the current video frame. Figure 4 Step S110 may include step S111 and step S113, the specific contents of which are described as follows.

[0084] Step S111 , calculating boundary information of each pixel point in the current video frame according to the obtained pixel value of each pixel point in the current video frame.

[0085] In this embodiment, after the captured current video frame is directly acquired through the image acquisition device or the forwarded current video frame is acquired through the connected terminal device, the current video frame can be detected to obtain the pixel value of each pixel point. Then, the boundary information of each pixel point in the current video frame is calculated based on the acquired pixel value.

[0086] It should be noted that, before detecting the current video frame to obtain pixel values, the current video frame may be converted into a grayscale image. In an alternative example, the size may be adjusted as required, for example, the size may be scaled to 256*256.

[0087] Step S113, judging whether each pixel belongs to a foreground boundary point according to the boundary information of each pixel, and obtaining a first mask image according to the mask value of each pixel belonging to the foreground boundary point.

[0088] In this embodiment, after the boundary information of each pixel point in the current video frame is obtained in step S111, it can be determined whether each pixel point belongs to a foreground boundary point according to the obtained boundary information. Then, the mask value of each pixel point belonging to the foreground boundary point is obtained, so as to obtain the first mask image based on the obtained mask values.

[0089] Optionally, the method of executing step S111 to calculate the boundary information is not limited and can be selected according to actual application requirements.

[0090] For example, in an alternative example, for each pixel point, the boundary information of the pixel point may be calculated based on the pixel values ​​of a plurality of pixel points adjacent to the pixel point.

[0091] In detail, the boundary information of each pixel can be calculated by the following calculation formula:

[0092] Gx=(fr_BW(i+1,j-1)+2*fr_BW(i+1,j)+fr_BW(i+1,j+1))_(fr_BW(i-1,j-1)+2*fr_BW(i-1,j)+fr_BW(i-1,j+1));

[0093] Gy=(fr_BW(i-1,j+1)+2*fr_BW(i,j+1)+fr_BW(i+1,j+1))-(fr_BW(i-1,j-1)+2*fr_BW(i,j-1)+fr_BW(i+1,j-1));

[0094] fr_gray(i,j)=sqrt(Gx^2+Gy^2);

[0095] Among them, fr_BW() refers to the pixel value, and fr_gray() refers to the boundary information.

[0096] Optionally, the method of executing step S113 to obtain the first mask image according to the boundary information is not limited and can be selected according to actual application requirements.

[0097] For example, in an alternative example, the current video frame may be compared with a previously acquired video frame to obtain a first mask image.

[0098] In detail, step S113 may include the following steps:

[0099] First, for each pixel point, the current mask value and the current frequency value of the pixel point are determined according to the boundary information of the pixel point in the current video frame, the boundary information in the previous N video frames, and the boundary information in the previous M video frames. Then, for each pixel point, whether the pixel point belongs to a foreground boundary point is determined according to the current mask value and the current frequency value, and a first mask image is obtained according to the current mask value of each pixel point belonging to the foreground boundary point.

[0100] In an alternative example, the current mask value and the current frequency value of the pixel point may be determined in the following manner:

[0101] First, if the boundary information of a pixel point meets the first condition, the current mask value of the pixel point can be updated to 255 and the current frequency value can be increased by 1. The first condition may include: the boundary information of the pixel point in the current video frame is greater than A1, and the difference between the boundary information of the pixel point in the current video frame and the boundary information in the previous N frames of video frames or the boundary information in the previous M frames of video frames is greater than B1;

[0102] Secondly, if the boundary information of a pixel does not meet the first condition but meets the second condition, the current mask value of the pixel can be updated to 180 and the current frequency value can be increased by 1. The second condition may include: the boundary information of the pixel in the current video frame is greater than A2, and the difference between the boundary information of the pixel in the current video frame and the boundary information in the previous N video frames or the boundary information in the previous M video frames is greater than B2;

[0103] Then, if the boundary information of a pixel does not meet the first and second conditions above, but meets the third condition, the current mask value of the pixel can be updated to 0 and the current frequency value can be increased by 1. The third condition may include: the boundary information of the pixel in the current video frame is greater than A2;

[0104] Finally, for a pixel point that does not meet the first condition, the second condition, and the third condition, the current mask value of the pixel point may be updated to 0.

[0105] It should be noted that the above current frequency value refers to the number of times the pixel point is considered to be a foreground boundary point in each video frame. For example, for pixel point (i, j), if it is considered to be a foreground boundary point in the first video frame, the current frequency value is 1; if it is also considered to be a foreground boundary point in the second video frame, the current frequency value is 2; if it is also considered to be a foreground boundary point in the third video frame, the current frequency value is 3.

[0106] The values ​​of N and M are not limited, as long as N is not equal to M. For example, in an alternative example, N may be 1 and M may be 3. That is, for each pixel point, the current mask value and the current frequency value of the pixel point may be determined according to the boundary information of the pixel point in the current video frame, the boundary information in the previous video frame, and the boundary information in the previous three video frames.

[0107] Correspondingly, the specific values ​​of A1, A2, B1 and B2 mentioned above are also not limited. For example, in an alternative example, A1 may be 30, A2 may be 20, B1 may be 12, and B2 may be 8.

[0108] Furthermore, after obtaining the current mask value and the current frequency value of the pixel point in the above manner, the pixel point whose current mask value is greater than 0 can be determined as the foreground boundary point, and the pixel point whose current mask value is equal to 0 can be determined as the background boundary point.

[0109] Furthermore, in order to further improve the accuracy of determining the foreground boundary points and the background boundary points, it is also possible to further determine whether the pixel point belongs to the foreground boundary point based on the following method, which may include:

[0110] First, for a pixel whose current mask value is greater than 0, if the ratio of the current frequency value of the pixel to the current frame number is greater than 0.6, and the difference between the boundary information in the current video frame and the boundary information in the previous video frame and the boundary information in the previous three video frames is less than 10, the pixel can be re-determined as a background boundary point;

[0111] Secondly, for a pixel whose current mask value is equal to 0, if the ratio of the current frequency value of the pixel to the current frame number is less than 0.5, and the boundary information in the current video frame is greater than 60, the pixel can be re-determined as a foreground boundary point, and the current mask value of the pixel is updated to 180;

[0112] Finally, in order to improve the accuracy of foreground image extraction of subsequent video frames, for pixels that do not meet the above two conditions, the current frequency value of the pixel can be reduced by 1.

[0113] Optionally, the method of executing step S120 to obtain the second mask image based on the current video frame is not limited and can be selected according to actual application requirements.

[0114] For example, in an alternative example, the neural network model may include multiple network sub-models to perform different processing to obtain the second mask image.

[0115] In detail, combined Figure 5 , the neural network model may include a first network sub-model, a second network sub-model and a third network sub-model. Step S120 may include the following steps:

[0116] First, the first network sub-model is used to extract semantic information from the current video frame to obtain a first output value. Secondly, the second network sub-model is used to resize the first output value to obtain a second output value. Then, the third network sub-model is used to extract a mask image from the second output value to obtain a second mask image.

[0117] The first network sub-model may be constructed by a first convolutional layer, a plurality of second convolutional layers, and a plurality of third convolutional layers. The second network sub-model may be constructed by the first convolutional layer and a plurality of fourth convolutional layers. The third network sub-model may be constructed by a plurality of the fourth convolutional layers and a plurality of upsampling layers.

[0118] It should be noted that the first convolution layer can be used to perform one convolution operation (an operation with a convolution kernel size of 3*3). The second convolution layer can be used to perform two convolution operations, one depthwise separable convolution operation and two activation operations (such as Figure 6The third convolutional layer can be used to perform two convolution operations, one depth-separable convolution operation and two activation operations, and output the value obtained by the operation together with the input value (as shown in Figure 7 The fourth convolutional layer can be used to perform one convolution operation, one depth-separable convolution operation and two activation operations (as shown in FIG. Figure 8 The upsampling layer may be used to perform a bilinear difference upsampling operation (such as an upsampling operation of 4 times).

[0119] Among them, in order to facilitate the neural network model to recognize and process the current video frame, the current video frame can also be pre-scaled to an array P of 256*256*3, and then normalized (to obtain a value between -1 and 1) through a normalization calculation formula (such as (P / 128)-1), and the processed result is input into the neural network model for recognition processing.

[0120] Optionally, the method of executing step S130 to calculate the foreground image based on a preset calculation model is not limited and can be selected according to actual application requirements.

[0121] For example, in an alternative example, step S130 may include the following steps:

[0122] First, the first mask image and the second mask image are weighted and summed according to a preset first weighting coefficient and a preset second weighting coefficient, and then the result of the weighted summing is summed with a predetermined parameter to obtain a foreground image in the current video frame.

[0123] That is, the calculation model may include:

[0124] M_fi=a1*M_fg+a2*M_c+b;

[0125] Among them, a1 is the first weighting coefficient, a2 is the second weighting coefficient, b is the parameter, M_fg is the first mask image, M_c is the second mask image, and M_fi is the foreground image.

[0126] It should be noted that the above a1, a2 and b can be determined according to the type of the specific foreground image. For example, when the foreground image is a portrait, it can be obtained by collecting multiple sample portraits for fitting.

[0127] Furthermore, considering that in some examples, the foreground image is determined for some specific display or playback control. For example, in the field of live broadcasting, in order to avoid the host portrait being blocked by the displayed or played bullet screen, it is necessary to first determine the position of the host portrait in the video frame, and when the bullet screen is played to that position, make it transparent or hidden, thereby improving the user experience.

[0128] That is to say, in some examples, the foreground image needs to be displayed or played. In order to avoid the shaking of the portrait during display or playback, a shaking elimination process can also be performed.

[0129] In detail, in an alternative example, in combination Fig. 9 Before executing step S130, the foreground image acquisition method may further include steps S140 and S150.

[0130] Step S140, calculating a first difference between the first mask image of the current video frame and the first mask image of the previous video frame, and calculating a second difference between the second mask image of the current video frame and the second mask image of the previous video frame.

[0131] Step S150: if the first difference is less than a preset difference, the first mask image of the current video frame is updated to the first mask image of the previous video frame; if the second difference is less than the preset difference, the second mask image of the current video frame is updated to the second mask image of the previous video frame.

[0132] In this embodiment, it is possible to determine whether a large change has occurred in the foreground image by calculating the amount of change between the first mask image and the second mask image in the current video frame and the previous video frame. Furthermore, when it is determined that the foreground image has not changed significantly between two adjacent frames (the current frame and the previous frame), the foreground image in the previous frame can be used to replace the foreground image in the current frame (i.e., the first mask image in the previous frame is used to replace the first mask image in the current frame, and the second mask image in the previous frame is used to replace the second mask image in the current frame), thereby avoiding the problem of inter-frame jitter.

[0133] In this way, when the foreground image (such as a portrait) changes slightly, the foreground image obtained in the current frame can be made the same as the foreground image obtained in the previous frame, thereby achieving inter-frame stability and avoiding the problem of poor user experience caused by inter-frame jitter.

[0134] That is, after executing step S150 to update the first mask image and the second mask image of the current video frame, when executing step S130, the foreground image may be calculated based on the updated first mask image and the second mask image.

[0135] Correspondingly, if the first difference is greater than or equal to the preset difference, and the second difference is greater than or equal to the preset difference, it indicates that the foreground image has changed significantly. In order to allow the live audience to effectively see the host's actions, when executing step S130, it is necessary to calculate the foreground image based on the first mask image obtained by executing step S110 and the second mask image obtained by executing step S120, so that the foreground image is different from the foreground image of the previous frame, so that the host's actions are reflected when the foreground image is played.

[0136] The method of executing step S140 to calculate the first difference and the second difference is not limited and can be selected according to actual application requirements.

[0137] The inventors of the present application have found through research that, because the smaller movements of the host are eliminated through step S150, the foreground image may jump during playback.

[0138] For example, the host's eyes are closed in the first video frame, open 0.1cm in the second video frame, and open 0.3cm in the third video frame. Since the host's eyes change little from the first video frame to the second video frame, in order to avoid inter-frame jitter, the foreground image of the second video frame and the foreground image of the first video frame will be consistent, so that the host's eyes in the foreground image of the second video frame are also closed. However, since the host's eyes change greatly from the second video frame to the third video frame, the host's eyes will open 0.3cm in the foreground image of the third video frame. In this way, the audience will see that the host's eyes change directly from closed to open 0.3cm, that is, there is a jump between frames (between the second frame and the third frame).

[0139] Considering that some viewers may not be used to the above-mentioned frame jump situation, in order to avoid the occurrence of this situation, in an alternative example, Fig.10 , step S140 may include step S141 and step S143 to calculate the first difference and the second difference.

[0140] Step S141 , performing inter-frame smoothing processing on the first mask image of the current video frame to obtain a new first mask image, and performing inter-frame smoothing processing on the second mask image of the current video frame to obtain a new second mask image.

[0141] Step S143: calculating a first difference between the new first mask image and the first mask image of the previous video frame, and calculating a second difference between the new second mask image and the second mask image of the previous video frame.

[0142] Furthermore, if the first difference is greater than or equal to the preset difference, the first mask image of the current video frame is updated to the new first mask image, so that calculation can be performed based on the new first mask image when step S150 is performed. If the second difference is greater than or equal to the preset difference, the second mask image of the current video frame is updated to the new second mask image, so that calculation can be performed based on the new second mask image when step S150 is performed.

[0143] The manner of performing inter-frame smoothing processing in step S141 is not limited. For example, in an alternative example, step S141 may include the following steps:

[0144] First, a first mean of the first mask images of all video frames before the current video frame is calculated, and a second mean of the second mask images of all video frames is calculated. Then, a new first mask image is calculated based on the first mean and the first mask image of the current video frame, and a new second mask image is calculated based on the second mean and the second mask image of the current video frame.

[0145] When a new first mask image and a new second mask image are calculated according to the first mean value and the second mean value, a specific calculation method is not limited.

[0146] In an alternative example, the new first mask image may be calculated based on a weighted summation method. For example, the new first mask image may be calculated according to the following formula:

[0147] M_k1=α1*M_k2+β1*A_k-1

[0148] A_k-1=α2*A_k-2+β2*M_k2-1

[0149] α1+β1=1, α2+β2=1;

[0150] Wherein, M_k1 is the new first mask image, M_k2 is the first mask image obtained by step S110, A_k-1 is the first mean calculated for all video frames before the current video frame, A_k-2 is the first mean calculated for all video frames before the previous video frame, M_k2-1 is the first mask image corresponding to the previous video frame, α1 can belong to [0.1, 0.9], and α2 can belong to [0.125, 0.875].

[0151] Similarly, a new second mask image may also be calculated based on a weighted summation method. The specific calculation formula may refer to the above formula, which will not be described in detail here.

[0152] It should be noted that after performing inter-frame smoothing processing using the above method to obtain a new first mask image and a new second mask image, the new first mask image and the new second mask image can also be binarized, and corresponding calculations can be performed based on the results of the binarization processing in subsequent steps.

[0153] The manner of performing the binarization process is not limited. For example, in an alternative example, the Otsu algorithm may be used for the binarization process.

[0154] It should be noted that the manner of executing step S143 to calculate the first difference and the second difference is not limited. For example, in an alternative example, step S143 may include the following steps:

[0155] First, it is determined whether the connected regions belong to the first target region according to the area of ​​each connected region in the new first mask image, and it is determined whether the connected regions belong to the second target region according to the area of ​​each connected region in the new second mask image.

[0156] Secondly, the first centroid coordinates of the connected area belonging to the first target area are calculated, and the centroid coordinates of the new first mask image are updated to the first centroid coordinates; the second centroid coordinates of the connected area belonging to the second target area are calculated, and the centroid coordinates of the new second mask image are updated to the second centroid coordinates.

[0157] Then, a first difference between the first barycentric coordinates and the barycentric coordinates of the first mask image of the previous video frame is calculated, and a second difference between the second barycentric coordinates and the barycentric coordinates of the second mask image of the previous video frame is calculated.

[0158] It should be noted that, in an alternative example, whether each connected region in the new first mask image belongs to the first target region may be determined based on the following method:

[0159] First, the area of ​​each connected region in the new first mask image can be calculated and the maximum area can be determined. Secondly, for each connected region in the new first mask image, it is determined whether the area of ​​the connected region is greater than one third of the maximum area (or other ratios can be used, which can be determined according to actual application requirements). Then, the connected region with an area greater than one third of the maximum area is determined as the first target region.

[0160] The method for judging whether each connected region in the new second mask image belongs to the second target region may refer to the above method, which will not be described in detail here.

[0161] It should be noted that, in an alternative example, the first centroid coordinates of the connected area belonging to the first target area may be calculated based on the following method:

[0162] First, determine whether the number of connected areas belonging to the first target area is greater than 2 (or other values, which can be determined according to actual application requirements). Second, if the number is greater than 2, calculate the first center of gravity coordinates based on the center of gravity coordinates of the two connected areas belonging to the first target area with the largest areas. If the number is not greater than 2, calculate the first center of gravity coordinates directly based on the center of gravity coordinates of the connected areas belonging to the first target area.

[0163] The method for calculating the second centroid coordinates of the connected area belonging to the second target area may refer to the above method, which will not be described in detail here.

[0164] It should be noted that after a new first mask image and a new second mask image are obtained by calculating the first mean and the second mean, the first mask image obtained by step S110 can be updated to the new first mask image, and the second mask image obtained by step S120 can be updated to the new second mask image.

[0165] Among them, in each of the above steps, there is an update process for the first mask image and the second mask image. Therefore, when executing each step, if the step has been updated before, then when executing the step, the step can be processed according to the first mask image and the second mask image after the most recent update process.

[0166] Furthermore, in order to avoid wasting computing resources of the processor 14 of the electronic device 10, before executing step S140, regional feature calculation processing may be performed on the first mask image obtained in step S110 and the second mask image obtained in step S120.

[0167] Among them, by calculating the area ratio of the effective area in the first mask image and the area ratio of the effective area in the second mask image, and when the area ratios do not reach the preset ratios, it can be determined that there is no foreground image in the current video frame. Therefore, subsequent steps can be chosen not to be executed, thereby reducing the amount of data calculation of the processor 14 of the electronic device 10.

[0168] Combination Fig.11 In an alternative example, the area of ​​the connected region formed by the foreground boundary points can be calculated first. Then, the connected region with the largest area is taken as the effective region. Then, the ratio of the area of ​​the effective region to the area of ​​the smallest box covering the effective region can be calculated to obtain the area ratio.

[0169] Combination Fig.12 The embodiment of the present application further provides a foreground image acquisition device 100 , which may include a first mask image acquisition module 110 , a second mask image acquisition module 120 and a foreground image acquisition module 130 .

[0170] The first mask image acquisition module 110 is used to perform inter-frame motion detection on the current video frame to obtain a first mask image. In this embodiment, the first mask image acquisition module 110 can be used to perform Figure 3 As shown in step S110 , the relevant contents about the first mask image acquisition module 110 can refer to the above description of step S110 .

[0171] The second mask image acquisition module 120 is used to identify the current video frame through a neural network model to obtain a second mask image. In this embodiment, the second mask image acquisition module 120 can be used to perform Figure 3 As shown in step S120 , the relevant contents about the second mask image acquisition module 120 can refer to the above description of step S120 .

[0172] The foreground image acquisition module 130 is used to calculate the foreground image in the current video frame according to the preset calculation model, the first mask image and the second mask image. In this embodiment, the foreground image acquisition module 130 can be used to perform Figure 3 As shown in step S130 , the relevant contents of the foreground image acquisition module 130 may refer to the above description of step S130 .

[0173] In an embodiment of the present application, corresponding to the foreground image acquisition method described above, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is run, each step of the foreground image acquisition method described above is executed.

[0174] The steps executed when the aforementioned computer program is running will not be described one by one here, and reference may be made to the explanation of the foreground image acquisition method in the previous text.

[0175] In summary, the foreground image acquisition method, foreground image acquisition device 100 and electronic device 10 provided by the present application respectively perform inter-frame motion detection and neural network recognition on the same video frame, and calculate the foreground image in the video frame based on the obtained first mask image and second mask image. In this way, the basis for calculating the foreground image is increased, thereby improving the accuracy and effectiveness of the calculation results, thereby improving the problem that it is difficult to accurately and effectively extract the foreground image of the video frame using the existing foreground extraction technology, and having high practical value.

[0176] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A foreground image acquisition method, characterized in that: include: Performing inter-frame motion detection on the obtained current video frame to obtain a first mask image; Recognizing the current video frame through a neural network model to obtain a second mask image; Obtaining a foreground image in the current video frame by calculation based on a preset calculation model, the first mask image, and the second mask image; The step of performing inter-frame motion detection on the obtained current video frame to obtain a first mask image includes: Calculate the boundary information of each pixel point in the current video frame according to the obtained pixel value of each pixel point in the current video frame; For each pixel point, determine the current mask value and the current frequency value of the pixel point according to the boundary information of the pixel point in the current video frame, the boundary information in the previous N video frames, and the boundary information in the previous M video frames, where N is not equal to M; the current frequency value is the number of times the pixel point belongs to the foreground boundary point in each video frame; For each pixel point, it is determined whether the pixel point belongs to a foreground boundary point according to the current mask value and the current frequency value, and a first mask image is obtained according to the current mask value of each pixel point belonging to the foreground boundary point.

2. The foreground image acquisition method according to claim 1, characterized in that: The neural network model includes a first network sub-model, a second network sub-model and a third network sub-model; The step of identifying the current video frame by a neural network model to obtain a second mask image includes: Performing semantic information extraction processing on the current video frame through the first network sub-model to obtain a first output value; Performing a size adjustment process on the first output value by using the second network sub-model to obtain a second output value; The second output value is subjected to mask image extraction processing by the third network sub-model to obtain a second mask image.

3. The foreground image acquisition method according to claim 2, characterized in that: The method further comprises the step of pre-constructing the first network sub-model, the second network sub-model and the third network sub-model, the step comprising: The first network sub-model is constructed by a first convolutional layer, a plurality of second convolutional layers, and a plurality of third convolutional layers, wherein the first convolutional layer is used to perform one convolution operation, the second convolutional layer is used to perform two convolution operations, one depth-separable convolution operation, and two activation operations, and the third convolutional layer is used to perform two convolution operations, one depth-separable convolution operation, and two activation operations, and outputs the values ​​obtained by the operation together with the input values; constructing the second network sub-model through the first convolutional layer and multiple fourth convolutional layers, wherein the fourth convolutional layer is used to perform one convolution operation, one depth-separable convolution operation and two activation operations; The third network sub-model is constructed by multiple fourth convolutional layers and multiple upsampling layers, wherein the upsampling layers are used to perform a bilinear interpolation upsampling operation.

4. The foreground image acquisition method according to claim 1, characterized in that: The step of calculating and obtaining the foreground image in the current video frame based on a preset calculation model, the first mask image and the second mask image comprises: Performing weighted sum processing on the first mask image and the second mask image according to a preset first weighting coefficient and a second weighting coefficient; The result obtained by weighted summation is summed with a predetermined parameter to obtain a foreground image in the current video frame.

5. The foreground image acquisition method according to any one of claims 1 to 4, characterized in that: Before executing the step of calculating and obtaining the foreground image in the current video frame based on the preset calculation model, the first mask image and the second mask image, the method further includes: Calculating a first difference between the first mask image of the current video frame and the first mask image of the previous video frame, and calculating a second difference between the second mask image of the current video frame and the second mask image of the previous video frame; If the first difference is less than a preset difference, updating the first mask image of the current video frame to the first mask image of the previous video frame; If the second difference is less than a preset difference, the second mask image of the current video frame is updated to the second mask image of the previous video frame.

6. The foreground image acquisition method according to claim 5, characterized in that: The step of calculating a first difference between the first mask image of the current video frame and the first mask image of the previous video frame, and calculating a second difference between the second mask image of the current video frame and the second mask image of the previous video frame, comprises: Performing inter-frame smoothing processing on the first mask image of the current video frame to obtain a new first mask image, and performing inter-frame smoothing processing on the second mask image of the current video frame to obtain a new second mask image; Calculating a first difference between the new first mask image and the first mask image of the previous video frame, and calculating a second difference between the new second mask image and the second mask image of the previous video frame; The foreground image acquisition method further includes: If the first difference is greater than or equal to a preset difference, updating the first mask image of the current video frame to the new first mask image; If the second difference is greater than or equal to a preset difference, the second mask image of the current video frame is updated to the new second mask image.

7. The foreground image acquisition method according to claim 6, characterized in that: The step of performing inter-frame smoothing processing on the first mask image of the current video frame to obtain a new first mask image, and performing inter-frame smoothing processing on the second mask image of the current video frame to obtain a new second mask image comprises: Calculating a first mean value of the first mask images of all video frames before the current video frame, and calculating a second mean value of the second mask images of all video frames; A new first mask image is calculated based on the first mean value and the first mask image of the current video frame, and a new second mask image is calculated based on the second mean value and the second mask image of the current video frame.

8. The foreground image acquisition method according to claim 6, characterized in that: The step of calculating a first difference between the new first mask image and the first mask image of the previous video frame, and calculating a second difference between the new second mask image and the second mask image of the previous video frame, comprises: Determining whether the connected regions belong to the first target region according to the area of ​​each connected region in the new first mask image, and determining whether the connected regions belong to the second target region according to the area of ​​each connected region in the new second mask image; Calculating first barycentric coordinates of a connected area belonging to the first target area, and updating the barycentric coordinates of the new first mask image to the first barycentric coordinates; Calculating the second barycentric coordinates of the connected area belonging to the second target area, and updating the barycentric coordinates of the new second mask image to the second barycentric coordinates; A first difference between the first barycentric coordinates and the barycentric coordinates of the first mask image of the previous video frame is calculated, and a second difference between the second barycentric coordinates and the barycentric coordinates of the second mask image of the previous video frame is calculated.

9. A foreground image acquisition device, characterized in that: include: A first mask image acquisition module, used for performing inter-frame motion detection on the obtained current video frame to obtain a first mask image; The first mask image acquisition module is also used to calculate the boundary information of each pixel point in the current video frame according to the pixel value of each pixel point in the current video frame; for each pixel point, according to the boundary information of the pixel point in the current video frame, the boundary information in the previous N video frames and the boundary information in the previous M video frames, determine the current mask value and the current frequency value of the pixel point, wherein N is not equal to M; the current frequency value is the number of times the pixel point belongs to the foreground boundary point in each video frame; for each pixel point, determine whether the pixel point belongs to the foreground boundary point according to the current mask value and the current frequency value, and obtain the first mask image according to the current mask value of each pixel point belonging to the foreground boundary point; A second mask image acquisition module, used for identifying the current video frame through a neural network model to obtain a second mask image; The foreground image acquisition module is used to calculate the foreground image in the current video frame according to a preset calculation model, the first mask image and the second mask image.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the foreground image acquisition method according to any one of claims 1 to 8 when running on the processor.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the foreground image acquisition method described in any one of claims 1 to 8 is implemented.

Images