An image processing method, apparatus and device

Abstract

The application provides an image processing method, device and equipment, the method comprising: selecting a first type of initial image from a plurality of initial images of an image sequence corresponding to a target object; wherein there are K-1 second type of initial images between two adjacent first type of initial images; inputting the first type of initial image into a target network model, outputting K foreground weight images corresponding to the first type of initial image by the target network model; and determining a foreground region image corresponding to the first type of initial image and K-1 second type of initial images behind the first type of initial image based on the K foreground weight images. According to the technical scheme of the application, only one first type of initial image needs to be input into the target network model, and K foreground region images corresponding to K initial images can be obtained, so that the calculation amount can be significantly reduced, the calculation speed can be accelerated, and the real-time requirement can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method, device and equipment. BACKGROUND

[0002] For an initial image, a region of interest in the initial image can be referred to as a foreground region, and other regions outside the foreground region can be referred to as background regions. In order to distinguish the foreground region and the background region, the initial image needs to be matting to obtain the foreground region and the background region in the initial image. In order to mat the initial image, the initial image can be input to a deep learning network model, and the deep learning network model can mat the initial image to obtain the foreground region and the background region in the initial image.

[0003] Since a video file includes a large number of initial images, when the video file is matting, each initial image in the video file is matting. Each initial image matting requires a high amount of calculation. Therefore, when a large number of initial images in the video file are matting, the amount of calculation is large, the matting speed is slow, and a long time is required to complete the matting, which cannot meet the real-time requirement. SUMMARY

[0004] The present application provides an image processing method, which comprises:

[0005] selecting a first type of initial image from a plurality of initial images of an image sequence corresponding to a target object; wherein there are K-1 second type of initial images between two adjacent first type of initial images, and K is greater than 1;

[0006] for each first type of initial image, inputting the first type of initial image to a target network model, and outputting K foreground weight images corresponding to the first type of initial image by the target network model, wherein the foreground weight image includes a probability that each pixel point in the first type of initial image belongs to a foreground region;

[0007] determining a foreground region image corresponding to the first type of initial image and K-1 second type of initial images after the first type of initial image based on the K foreground weight images.

[0008] The present application provides an image processing device, which comprises:

[0009] a selection module configured to select a first type of initial image from a plurality of initial images of an image sequence corresponding to a target object; wherein there are K-1 second type of initial images between two adjacent first type of initial images, and K is greater than 1;

[0010] The processing module is configured to, for each first initial image, input the first initial image into the target network model, and output K foreground weight images corresponding to the first initial image by the target network model, wherein each foreground weight image includes a probability that each pixel point in the first initial image belongs to a foreground region.

[0011] The determining module is configured to determine foreground region images corresponding to the first initial image and K-1 second initial images following the first initial image based on the K foreground weight images.

[0012] The present application provides an image processing device, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor; the processor is used to execute the machine executable instructions to realize the image processing method disclosed in the above examples of the present application.

[0013] As can be seen from the above technical solutions, in the embodiments of the present application, the multiple initial images in the image sequence (such as a video file) are divided into first initial images and second initial images, and there are K-1 second initial images between any two adjacent first initial images. For each first initial image, the first initial image is input into the target network model, and K foreground weight images are output by the target network model. The foreground region images corresponding to the first initial image and the K-1 second initial images following the first initial image are determined based on the K foreground weight images. Obviously, only one first initial image needs to be input into the target network model, and the foreground region images corresponding to K initial images can be obtained, that is, when performing matting operation on one initial image, the matting results of K initial images are obtained, thereby the calculation amount can be significantly reduced and the calculation speed can be accelerated, that is, the matting speed is relatively fast, and the matting can be completed in a relatively short time to meet the real-time requirement. For devices with poor performance, a relatively smooth high-precision matting effect can also be achieved, and the high-precision requirement of matting can be guaranteed. The continuity information between images can be fully utilized to assist matting, thereby improving the matting effect, the matting results of K initial images can be predicted based on historical information, the motion information of the target object can be better captured, and a higher quality matting result can be obtained. The stability and accuracy of matting can be improved, the matting quality can be improved, and the hardware cost can be saved, thereby saving hardware resources. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the description of the embodiments of the present application or the prior art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0015] Figure 1 is a flowchart of an image processing method in an embodiment of the present application;

[0016] Figures 2A-2F is a schematic diagram of each image in an embodiment of the present application;

[0017] Figure 3A and Figure 3B is a structural diagram of a network model in an embodiment of the present application;

[0018] Figure 3C is a schematic diagram of a sample weight image in an embodiment of the present application;

[0019] Figure 4 is a flowchart of an image processing method in an embodiment of the present application;

[0020] Figures 5A-5C is a schematic diagram of a foreground weight image in an embodiment of the present application;

[0021] Figure 6 is a structural diagram of an image processing apparatus in an embodiment of the present application;

[0022] Figure 7 is a hardware structural diagram of an image processing device in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments of the present application and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc. can be used herein to describe various information, but these information should not be limited to these terms. These terms are only used to distinguish one piece of information from another piece of information of the same type. For example, a first information can also be called a second information without departing from the scope of the present application, and similarly, a second information can also be called a first information. In addition, depending on the context, the word “if” used herein can be interpreted as “when” or “in response to determining” or “in response to ascertaining”.

[0025] An image processing method is proposed in the embodiments of the present application, which can be applied to any type of device, and the type of the device is not limited, such as a device with poor performance or a device with good performance. Referring to Figure 1As shown, it is a flowchart diagram of the image processing method, which can include the following steps:

[0026] Step 101, selecting a first type of initial image from a plurality of initial images of an image sequence corresponding to a target object; wherein there are K-1 second type of initial images between any two adjacent first type of initial images, and K is greater than 1.

[0027] For example, the image sequence can be divided into a first sub-sequence and a second sub-sequence, the first sub-sequence can include the first M initial images in the image sequence, and the second sub-sequence can include the remaining initial images in the image sequence except the first M initial images, and M can be greater than 1. Then, a configured frame interval (which can be configured according to experience) is obtained, the frame interval represents the number of images between any two adjacent first type of initial images, and the first type of initial image is selected from the second sub-sequence based on the frame interval.

[0028] Step 102, for each first type of initial image, inputting the first type of initial image into a target network model, outputting K foreground weight images corresponding to the first type of initial image by the target network model, and the foreground weight image includes the probability that each pixel point in the first type of initial image belongs to the foreground region.

[0029] Step 103, determining the foreground region image corresponding to the first type of initial image and the K-1 second type of initial images after the first type of initial image based on the K foreground weight images.

[0030] In a possible implementation, after the image sequence is divided into the first sub-sequence and the second sub-sequence, for each third type of initial image in the first sub-sequence, the foreground region image corresponding to the third type of initial image can also be determined. For example, the third type of initial image can be input into the target network model, and K foreground weight images corresponding to the third type of initial image can be output by the target network model; the foreground region image corresponding to the third type of initial image can be determined based on one of the K foreground weight images (such as the first foreground weight image in the K foreground weight images). Or, the third type of initial image and a preset label are input into the target network model, the preset label is used to indicate that one foreground weight image is output, and the target network model outputs one foreground weight image corresponding to the third type of initial image based on the preset label; the foreground region image corresponding to the third type of initial image is determined based on the foreground weight image.

[0031] In a possible implementation, the target network model can include a first sub-network and a second sub-network, the second sub-network including a last network layer of the target network model, and the first sub-network including remaining network layers of the target network model except the last network layer. Based on this, inputting the first type of initial image into the target network model and outputting K foreground weight images corresponding to the first type of initial image by the target network model can include but is not limited to: inputting the first type of initial image into the first sub-network, processing the first type of initial image by the first sub-network to obtain target features corresponding to the first type of initial image; and inputting the target features into K network channels of the second sub-network respectively, processing the target features by each network channel to obtain a foreground weight image corresponding to the network channel. That is, the second sub-network can include K network channels, and the K network channels can output K foreground weight images.

[0032] For example, the first sub-network can include T feature extraction network layers and N historical information network layers, T and N are positive integers, T can be greater than N, T can be equal to N, and T can be less than N. Based on this, inputting the first type of initial image into the first sub-network and processing the first type of initial image by the first sub-network to obtain target features corresponding to the first type of initial image can include but is not limited to: processing the first type of initial image by the first feature extraction network layer to obtain initial features, and inputting the initial features into the second feature extraction network layer; based on the second to the T-1th feature extraction network layer, processing the input initial features by the feature extraction network layer to obtain initial features, and inputting the initial features into the next feature extraction network layer of the feature extraction network layer; processing the input initial features by the Tth feature extraction network layer to obtain initial features, and inputting the initial features into the Nth historical information network layer; processing the input initial features and the historical features of the historical information network layer by the Nth historical information network layer to obtain candidate features, storing the candidate features as the historical features of the historical information network layer, and inputting the candidate features into the N-1th historical information network layer; based on the N-1th to the 2nd historical information network layer, processing the input candidate features and the historical features of the historical information network layer by the historical information network layer to obtain candidate features, storing the candidate features as the historical features of the historical information network layer, and inputting the candidate features into the previous historical information network layer of the historical information network layer; processing the input candidate features and the historical features of the historical information network layer by the first historical information network layer to obtain target features corresponding to the first type of initial image, and storing the target features as the historical features of the historical information network layer.

[0033] For example, for each historical information network layer, the historical information network layer can be a convolutional gated recurrent unit (convGRU) based network layer, or a convlutional long short term memory (convLSTM) based network layer, or a recurrent neural network (RNN) based network layer, or an LSTM based network layer, or a GRU based network layer. Of course, the historical information network layer can also be other network structures with memory function, and no limitation is made thereto.

[0034] In a possible implementation, before the first type of initial image is input to the target network model, a training data set can also be obtained, the training data set can include P consecutive sample images in a video stream (such as a video file) and a sample weight image corresponding to each sample image, and P can be greater than K; the configured initial network model is trained based on the training data set to obtain the trained target network model.

[0035] For training the target network model, the following manner can be used: a sample image (such as each frame of sample image) is input to the initial network model, and the initial network model outputs a foreground weight image corresponding to the sample image, and foreground weight images corresponding to K-1 sample images behind the sample image; then, based on the foreground weight image corresponding to the sample image and the sample weight image, and the foreground weight images corresponding to the K-1 sample images behind the sample image and the sample weight images, a target loss value is determined; then, based on the target loss value, the initial network model is trained to obtain the target network model.

[0036] From the above technical solution, in the embodiment of the application, the multiple frames of initial images in the image sequence (such as a video file) are divided into the first type of initial images and the second type of initial images, and there are K-1 frames of the second type of initial images between the adjacent two frames of the first type of initial images. For each frame of the first type of initial image, the first type of initial image is input to the target network model, and K foreground weight images are output by the target network model. Based on the K foreground weight images, the foreground region image corresponding to the first type of initial image and the K-1 frames of the second type of initial image behind the first type of initial image is determined. Obviously, only one frame of the first type of initial image needs to be input to the target network model, and the foreground region image corresponding to K frames of initial images can be obtained, that is, when performing matting on one frame of initial image, the matting results of K frames of initial images are obtained, thereby the calculation amount can be significantly reduced, the calculation speed can be accelerated, the matting speed is relatively fast, and the matting can be completed in a relatively short time to meet the real-time requirement. For devices with poor performance, a relatively smooth high-precision matting effect can also be achieved, and the high-precision matting requirement can be guaranteed. The continuity information between images can be fully utilized to assist matting, thereby improving the matting effect, the matting results of K frames of initial images can be predicted based on historical information, the motion information of the target object can be better captured, and a higher quality matting result can be obtained. The stability and accuracy of matting can be improved, the matting quality can be improved, no additional hardware cost is needed, and hardware resources are saved.

[0037] The above technical solution of the embodiment of the application will be described below in combination with specific application scenarios.

[0038] When performing matting on a video file, each frame of initial image in the video file needs to be matted, that is, matting is performed based on a single frame of initial image. Since each frame of initial image requires a relatively high calculation amount for matting, when performing frame-by-frame matting on a video file, the calculation amount is very large, the matting speed is relatively slow, and a relatively long time is needed to complete the matting. In some low-end devices, real-time requirements cannot be met.

[0039] In view of the above finding, in the embodiment, the continuity information between the initial images in the video file is fully utilized. Only one frame of initial image needs to be input to the target network model, and the foreground region image corresponding to K frames of initial images can be obtained, that is, when performing matting on one frame of initial image, the matting results of K frames of initial images are obtained, thereby the calculation amount can be significantly reduced, the calculation speed can be accelerated, and the high-precision matting requirement can be guaranteed.

[0040] First, the matting principle, that is, performing matting on the initial image to obtain the foreground region of the target object.

[0041] The initial image can be an image in RGB format, an image in BGR format, or a grayscale image. Taking an image in RGB format as an example, if the image to be cut out is an image in RGB format, the image is taken as the initial image, and if the image to be cut out is not an image in RGB format, such as an image in YUV format, the image in YUV format is converted into an image in RGB format, and the converted image is taken as the initial image. After obtaining the initial image, the initial image can be cut out.

[0042] Referring to Figure 2A Fig. 1 shows an example of an initial image in RGB format, which can include three channels. Taking a human object as an example, the human object region can have an irregular shape, as shown in Fig. 2. Figure 2B Fig. 2 shows an example of a human object region, which is the foreground region of the target object, i.e., the foreground region to be obtained when the initial image is cut out.

[0043] Since the initial image in RGB format includes three channels, the pixel value of each channel is [0, 255], and the position of the human object region in the three channels is the same, the prediction result of the human object region can be a single-channel grayscale image, as shown in Fig. 3. Figure 2B Fig. 3 shows an example of a single-channel grayscale image. The pixel value of the human object region can be 255, and the pixel value of the human object region can also be in the range of (0-255], i.e., excluding the pixel value 0, and the pixel value of the background region is all 0. Therefore, the pixel points with pixel values greater than 0 can be taken as the human object region, i.e., the foreground region of the target object, and the pixel points with pixel values equal to 0 can be taken as the background region.

[0044] The pixel value of the human object region is in the range of (0-255], which is not the pixel value 255. The reason is that in the edge region and transparent region of the human object, such as hair, wedding dress with transparency, etc., the foreground and background components exist at the same time, and the pixel value of this region needs to be output according to the percentage. For example, if a pixel point has a foreground percentage of 60% and a background percentage of 40%, the pixel value of the pixel point is 255*0.6=153, i.e., the pixel value of the pixel point is not 255.

[0045] In summary, when performing matting on the initial image, the pixel value of each pixel point of the initial image needs to be predicted. Assuming that the width of the initial image is W and the height is H, the pixel value of W*H pixel points needs to be predicted. For each pixel point, if the pixel value of the pixel point is not 0, the pixel point belongs to the foreground region; if the pixel value of the pixel point is 0, the pixel point belongs to the background region. When predicting the pixel value of W*H pixel points, the prediction can be a normalized value between 0 and 1, so the value can be multiplied by 255 to convert it into a grayscale image, i.e., a value between 0 and 255.

[0046] Exemplarily, a background image can also be generated, which is a virtual background image. There is no limitation on the virtual background image, see Figure 2C for an example of a virtual background image.

[0047] Exemplarily, the relationship between the foreground weight image, the foreground region image, the virtual background image, and the target image can be represented by the following formula: Image = alpha * foreground + (1-alpha) * background.

[0048] Alpha represents the prediction result, i.e., the foreground weight image, see Figure 2B for the foreground weight image, which represents the pixel ratio of the human object region, foreground represents the initial image, see Figure 2A for an initial image in RGB format, alpha * foreground represents the foreground region image, see Figure 2D for an example of a foreground region image. Background represents the virtual background image, see Figure 2C for an example of a virtual background image. Since alpha represents the proportion of the foreground region in the initial image, 1-alpha represents the proportion of the background region, and (1-alpha) * background is the new background region, see Figure 2E for an example of a background region. Then, the foreground region and the background region are added to obtain the final effect image after replacing the background, i.e., the target image, see Figure 2F for an example of a target image.

[0049] Obviously, based on the above method, the effect image (i.e., the target image) after replacing any background region (which can be selected arbitrarily according to actual needs) can be obtained. By using different backgrounds, the effect of being in different parts of the world can be achieved, which can be widely used in video live streaming, online meetings, photography special effects, etc.

[0050] Second, for video file matting, video historical information is used to enhance the matting effect.

[0051] One application scenario for image cutout is for video files, such as live video streaming and online meetings. In these scenarios, since video files contain a large number of initial images that are continuous, historical information can be used to enhance the accuracy and stability of the cutout.

[0052] For example, network structures that can remember historical information include RNN, LSTM, GRU, convLSTM, convGRU, etc. Of course, the above are just a few examples of network structures and are not intended to be limiting.

[0053] In this embodiment, these network structures can be used to record historical information of video files, thereby making full use of the continuity of a large number of initial images, and then using video historical information to enhance the matting effect.

[0054] For ease of description, this embodiment uses the convGRU network structure as an example. The input of convGRU is a two-dimensional fully convolutional layer, which can process two-dimensional initial images, take into account both temporal and spatial information at the same time, reduce the number of model parameters, and the training speed of convGRU is relatively fast.

[0055] See Figure 3A The diagram shows the structure of convGRU. The reset gate and update gate are actually normalization functions (sigmoid) that normalize the network output value to between 0 and 1, representing the weight of the input information. If it is 0, this part of the information will be lost. The closer it is to 1, the more important it is and the more it will be used.

[0056] The weights learned by convGRU are shown on the thick arrows. x represents multiplication and + represents addition. The weights after passing through the reset gate or update gate are weights between 0 and 1. Multiplying them by the input serves as a filtering function.

[0057] Regarding the working principle of convGRU, this embodiment does not impose any restrictions. The main principle is that the current frame input is combined with historical information, and after some filtering and learning, it is output. The output result ht is not only the output for the current frame, but will also be used as historical information for the input of the next frame to participate in the processing of the next frame.

[0058] When convGRU is embedded into the matting network, it can achieve "memory" of historical video information. When the network predicts the initial image t, it will also refer to the information between images 1 and t-1. This auxiliary historical information will greatly improve the quality of the network's prediction, making the results more accurate and stable.

[0059] Third, the structure of the initial network model and the target network model: the network model before training can be referred to as the initial network model, and the network model after training can be referred to as the target network model. The structure of the initial network model is the same as that of the target network model. For ease of description, the target network model will be used as an example.

[0060] For example, the target network model can be a network model based on deep learning algorithms or a network model based on neural networks. There are no restrictions on the structure of the target network model, and it can be configured arbitrarily, as long as the target network model can achieve the image matting function.

[0061] In one possible implementation, the target network model may include a first sub-network and a second sub-network. The second sub-network includes the last network layer of the target network model, and the first sub-network includes the remaining network layers of the target network model excluding the last network layer. The first sub-network may include T feature extraction network layers and N historical information network layers, where T and N are both positive integers, and T can be greater than N, equal to N, or less than N. For each historical information network layer, it can be a convGRU-based network layer, or a convLSTM-based network layer, or an RNN-based network layer, or an LSTM-based network layer, or a GRU-based network layer. For ease of description, the following example uses a convGRU-based historical information network layer.

[0062] See Figure 3B The diagram shows the structure of the target network model, which can include four feature extraction network layers (denoted as layer a1, layer a2, layer a3, and layer a4), four historical information network layers (denoted as layer b1, layer b2, layer b3, and layer b4), and network layer c. The first sub-network includes layers a1, a2, a3, a4, b1, b2, b3, and b4, and the second sub-network includes network layer c. The four feature extraction network layers and four historical information network layers are just examples; the number of feature extraction network layers can be more, such as five or six, or fewer, such as three. The number of historical information network layers can also be more or less. The number of feature extraction network layers and the number of historical information network layers can be the same or different.

[0063] For each feature extraction network layer, taking network layer a1 as an example, the network layer a1 can be composed of conv+BN (Batch Normalization) +Relu, i.e., composed of a convolutional layer, a batch normalization layer and an activation function. Of course, the above is only an example of the network layer a1, and the structure of the network layer a1 is not limited.

[0064] For each historical information network layer, taking network layer b1 as an example, the network layer b1 can be composed of a convolution unit (conv+BN+Relu) +convGRU, i.e., the network layer b1 can be composed of a convolution unit and a convolution gate recurrent unit layer. Of course, the above is only an example of the network layer b1, and the structure is not limited.

[0065] For the network layer c, the network layer c can include K network channels, and the value of K represents the frame interval. For the value of K, see the subsequent embodiments. For example, the network layer c can be a convolutional layer composed of K network channels, and for the network layer c, the K network channels are used to output K prediction results in parallel.

[0066] Referring to FIG. 1, Figure 3B As shown in FIG. 1, the input of the network layer a1 is an external image, and the output of the network layer a1 is the network layer a2 and the network layer b1. Assuming that the input resolution of the network layer a1 is 160*160, the network layer a1 will downsample the input, and the output resolution of the network layer a1 is 80*80. The input of the network layer a2 is the output of the network layer a1, and the output of the network layer a2 is the network layer a3 and the network layer b2. Assuming that the input resolution of the network layer a2 is 80*80, the network layer a2 will downsample the input, and the output resolution of the network layer a2 is 40*40. The input of the network layer a3 is the output of the network layer a2, and the output of the network layer a3 is the network layer a4 and the network layer b3. Assuming that the input resolution of the network layer a3 is 40*40, the network layer a3 will downsample the input, and the output resolution of the network layer a3 is 20*20. The input of the network layer a4 is the output of the network layer a3, and the output of the network layer a4 is the network layer b4. Assuming that the input resolution of the network layer a4 is 20*20, the network layer a4 will downsample the input, and the output resolution of the network layer a4 is 10*10.

[0067] The input of the network layer b4 is the output of the network layer a4, and the output of the network layer b4 is the network layer b3. Assuming that the input resolution of the network layer b4 is 10*10, the network layer b4 will up-sample the input, and the output resolution of the network layer b4 is 20*20. The input of the network layer b3 is the output of the network layer a3 and the output of the network layer b4, and the output of the network layer b3 is the network layer b2. Assuming that the input resolution of the network layer b3 is 20*20, the network layer b3 will up-sample the input, and the output resolution of the network layer b3 is 40*40. The input of the network layer b2 is the output of the network layer a2 and the output of the network layer b3, and the output of the network layer b2 is the network layer b1. Assuming that the input resolution of the network layer b2 is 40*40, the network layer b2 will up-sample the input, and the output resolution of the network layer b2 is 80*80. The input of the network layer b1 is the output of the network layer a1 and the output of the network layer b2, and the output of the network layer b1 is the network layer c. Assuming that the input resolution of the network layer b1 is 80*80, the network layer b1 will up-sample the input, and the output resolution of the network layer b1 is 160*160.

[0068] The input of the network layer c is the output of the network layer b1, and the output of the network layer c is the prediction result.

[0069] In summary, for each feature extraction network layer, down-sampling processing can be performed, and for each historical information network layer, up-sampling processing can be performed, so as to ensure that the resolution remains unchanged. Obviously, the convGRU can be distributed in different resolution stages. Generally, the network building process will have several down-sampling stages, and each down-sampling corresponds to a stage. The input of the convGRU includes the features extracted from the current frame and the historical information (the historical features with historical useful information obtained through the forget gate and the update gate of the convGRU). After sufficient training, a more accurate matting result can be predicted according to the historical information and the current frame. h represents historical information, which will be updated constantly and used as the input of the historical information network layer together with the input. After passing through the historical information network layer, new historical information is obtained, which is used as the input of the next frame.

[0070] Of course, Figure 3BJust an example, the structure of the target network model is not limited. For example, the input of the network layer a1 is an external image, the output of the network layer a1 is the network layer a2, the input of the network layer a2 is the output of the network layer a1, the output of the network layer a2 is the network layer a3, the input of the network layer a3 is the output of the network layer a2, the output of the network layer a3 is the network layer a4, the input of the network layer a4 is the output of the network layer a3, the output of the network layer a4 is the network layer b4, the input of the network layer b4 is the output of the network layer a4, the output of the network layer b4 is the network layer b3, the input of the network layer b3 is the output of the network layer b4, the output of the network layer b3 is the network layer b2, the input of the network layer b2 is the output of the network layer b3, the output of the network layer b2 is the network layer b1, the input of the network layer b1 is the output of the network layer b2, and the output of the network layer b1 is the network layer c.

[0071] Fourth, the training process of the target network model, for example, obtaining a training data set, the training data set can include P consecutive sample images in a video stream (such as a video file) and a sample weight image corresponding to each sample image, training the configured initial network model based on the training data set to obtain the trained target network model, the structure of the target network model can be referred to as Figure 3B shown.

[0072] For example, in order to train the target network model, the following steps can be used:

[0073] Step S11, obtaining a configured initial network model, the initial network model can include a first sub-network and a second sub-network, the second sub-network includes the last network layer of the initial network model, and the first sub-network includes the remaining network layers of the initial network model except the last network layer, the first sub-network can include T feature extraction network layers and N historical information network layers, as shown in Figure 3B shown.

[0074] Step S12, obtaining a training data set, the training data set can include P consecutive sample images in a video stream (such as a video file) and a sample weight image corresponding to each sample image.

[0075] For example, the target network model can be trained based on P consecutive sample images, the larger the value of P is, the better, but considering the performance limitation, P can take a suitable value. The images in the video stream can be used as sample images, or the images in the video stream can be converted, and the converted images can be used as sample images, such as at least one of the following conversions: enhancement, rotation, cropping, and flipping.

[0076] For each sample image, a sample weight image corresponding to the sample image also needs to be determined, the sample weight image representing the probability of each pixel point in the sample image belonging to the foreground region, that is, the sample weight image representing the pixel proportion of the human object region. Wherein, the sample weight image is the label corresponding to the sample image, see Figure 3C Fig. 3 shows an example of a sample weight image.

[0077] Step S13, training the initial network model based on the training data set to obtain the target network model.

[0078] For example, based on the training data set, the target network model can be trained by the following steps:

[0079] Step S131, for each frame of sample image in the training data set, taking a sample image as an example, the sample image is input into the initial network model, and the initial network model outputs the foreground weight image corresponding to the sample image, and the foreground weight image corresponding to the K-1 sample images behind the sample image.

[0080] For example, taking the initial network model shown in Figure 3B Fig. 1 as an example, the sample image can be input into the network layer a1, the sample image is down-sampled by the network layer a1, and the down-sampled sample image is processed, and the processing method is not limited, to obtain the initial feature 1, the initial feature 1 is input into the network layer a2 and the network layer b1. Then, the initial feature 1 is down-sampled by the network layer a2, and the down-sampled initial feature is processed to obtain the initial feature 2, the initial feature 2 is input into the network layer a3 and the network layer b2. Then, the initial feature 2 is down-sampled by the network layer a3, and the down-sampled initial feature is processed to obtain the initial feature 3, the initial feature 3 is input into the network layer a4 and the network layer b3. Then, the initial feature 3 is down-sampled by the network layer a4, and the down-sampled initial feature is processed to obtain the initial feature 4, the initial feature 4 is input into the network layer b4.

[0081] The initial feature 4 is up-sampled by the network layer b4, and the up-sampled initial feature and the historical feature of the network layer b4 are processed, and the processing process is not limited, to obtain the candidate feature 1, the candidate feature 1 is stored as the historical feature of the network layer b4, and the candidate feature 1 is input into the network layer b3.

[0082] The candidate feature 1 and the initial feature 3 are up-sampled by the network layer b3, the up-sampled candidate feature, the up-sampled initial feature and the historical feature of the network layer b3 are processed to obtain the candidate feature 2, the candidate feature 2 is stored as the historical feature of the network layer b3, and the candidate feature 2 is input into the network layer b2.

[0083] The candidate feature 2 and the initial feature 2 are up-sampled by the network layer b2, the up-sampled candidate feature, the up-sampled initial feature and the historical feature of the network layer b2 are processed to obtain a candidate feature 3, the candidate feature 3 is stored as the historical feature of the network layer b2, and the candidate feature 3 is input to the network layer b1.

[0084] The candidate feature 3 and the initial feature 1 are up-sampled by the network layer b1, the up-sampled candidate feature, the up-sampled initial feature and the historical feature of the network layer b1 are processed to obtain a target feature, the target feature is stored as the historical feature of the network layer b1, and the target feature is input to the network layer c.

[0085] For example, the network layer c can include K network channels, the target feature can be input to the first network channel of the network layer c, the target feature can be input to the second network channel of the network layer c, and the target feature can be input to the Kth network channel of the network layer c. The target feature is processed by the first network channel of the network layer c to obtain a prediction result of the foreground weight image, which is denoted as foreground weight image 1. The target feature is processed by the second network channel of the network layer c to obtain a prediction result of the foreground weight image, which is denoted as foreground weight image 2. The target feature is processed by the Kth network channel of the network layer c to obtain a prediction result of the foreground weight image, which is denoted as foreground weight image K.

[0086] For example, the network layer c can include K network channels, the target feature can be input to the first network channel of the network layer c, the target feature can be input to the second network channel of the network layer c, and the target feature can be input to the Kth network channel of the network layer c. The target feature is processed by the first network channel of the network layer c to obtain a prediction result of the foreground weight image, which is denoted as foreground weight image 1. The target feature is processed by the second network channel of the network layer c to obtain a prediction result of the foreground weight image, which is denoted as foreground weight image 2. The target feature is processed by the Kth network channel of the network layer c to obtain a prediction result of the foreground weight image, which is denoted as foreground weight image K.

[0087] In step S132, the target loss value is determined based on the foreground weight image corresponding to the sample image, the sample weight image, the foreground weight images corresponding to the K-1 sample images behind the sample image and the sample weight images.

[0088] For example, when the sample image w1 is input to the initial network model, the initial loss value 1 corresponding to the sample image w1 can be calculated based on the difference image of the foreground weight image (model output) corresponding to the sample image w1 and the sample weight image (pre-labeled), the difference image of the foreground weight image (model output) corresponding to the sample image w2 and the sample weight image (pre-labeled), and the difference image of the foreground weight image (model output) corresponding to the sample image w3 and the sample weight image (pre-labeled).

[0089] Similarly, when the sample image w2 is input to the initial network model, the initial loss value 2 corresponding to the sample image w2 can be calculated based on the difference image of the foreground weight image corresponding to the sample image w2 and the sample weight image, the difference image of the foreground weight image corresponding to the sample image w3 and the sample weight image, and the difference image of the foreground weight image corresponding to the sample image w4 and the sample weight image.

[0090] By analogy, the initial loss value corresponding to each frame of sample image in the training data set can be obtained, and then the target loss value can be obtained based on the sum of the initial loss values corresponding to all sample images.

[0091] In this embodiment, the determination manner of the initial loss value is not limited, and a loss function can be pre-configured, which is related to the above images, and the initial loss value can be calculated based on the loss function.

[0092] Step S133, training the initial network model based on the target loss value to obtain a target network model.

[0093] For example, the weights of the initial network model are adjusted based on the target loss value, and the adjustment manner is not limited, and the adjustment target is to make the target loss value smaller and smaller, and an adjusted network model is obtained. If the adjusted network model has converged, the adjusted network model is taken as the target network model. If the adjusted network model has not converged, the adjusted network model is taken as the initial network model, and the process returns to step S131.

[0094] Fifth, the detection process (i.e., the inference process) of the target network model. After obtaining the target network model, the initial image can be matting based on the target network model to obtain a matting result. Referring to FIG. 4, Figure 4 which is a schematic diagram of an image processing method based on a target network model. The process can include:

[0095] Step 401, dividing multiple frames of initial images of an image sequence (such as a video file or a video stream) corresponding to a target object into first-type initial images, second-type initial images, and third-type initial images.

[0096] Exemplarily, the target object is an object of interest of a user, for example, when the object of interest of the user is a human face, the target object is a human face, when the object of interest of the user is a human body, the target object is a human body, and when the object of interest of the user is a vehicle, the target object is a vehicle, and the type of the target object is not limited.

[0097] Exemplarily, the image sequence can include multiple frames of initial images, such as RGB type images, and each initial image can include a target object, so the image sequence can be referred to as an image sequence corresponding to the target object. The image sequence can include all initial images in a video file, or can include part of the initial images in the video file, that is, all initial images in the video file can be grouped into an image sequence, or part of the initial images in the video file can be grouped into an image sequence.

[0098] Exemplarily, the first M frames of initial images in the image sequence can be all third type initial images, and M can be greater than 1. Starting from the (M+1)th initial image, the initial images can be divided into first type initial images and second type initial images. For example, a configured frame interval K is obtained, the frame interval K represents the number of images between adjacent two frames of first type initial images, the first type initial images are selected based on the frame interval K, and there are K-1 frames of second type initial images between adjacent two frames of first type initial images, and K can be greater than 1. The value of the frame interval K is not limited.

[0099] Taking M as 4 and K as 3 as an example, the initial image 1, the initial image 2, the initial image 3 and the initial image 4 in the image sequence are all third type initial images, the initial image 5, the initial image 8, the initial image 11, the initial image 14, … are all first type initial images, and the initial image 6, the initial image 7, the initial image 9, the initial image 10, the initial image 12, the initial image 13, … are all second type initial images.

[0100] For each frame of third type initial image, the third type initial image needs to be input into the target network model to determine the foreground region image corresponding to the third type initial image through the target network model.

[0101] For each frame of first type initial image, the first type initial image needs to be input into the target network model to determine the foreground region image corresponding to the first type initial image through the target network model, and the foreground region images corresponding to the K-1 frames of second type initial images after the first type initial image are determined through the target network model.

[0102] For each frame of second type initial image, the second type initial image does not need to be input into the target network model, thereby reducing the number of initial images input into the target network model, and achieving the purpose of accelerating prediction.

[0103] Step 402: For each frame of the third type of initial image, input the third type of initial image into the target network model, and output the foreground weight image corresponding to the third type of initial image from the target network model. Based on the foreground weight image, determine the foreground region image corresponding to the third type of initial image.

[0104] For example, with Figure 3B Taking the target network model shown as an example, for each frame of the third type of initial image, the third type of initial image can be input into network layer a1. Network layer a1 processes the third type of initial image to obtain initial feature 1, which is then input into network layers a2 and b1. Network layer a2 processes initial feature 1 to obtain initial feature 2, which is then input into network layers a3 and b2. Network layer a3 processes initial feature 2 to obtain initial feature 3, which is then input into network layers a4 and b3. Network layer a4 processes initial feature 3 to obtain initial feature 4, which is then input into network layer b4.

[0105] The initial feature 4 and its historical features are processed by network layer b4 to obtain candidate feature 1. Candidate feature 1 is stored as a historical feature of network layer b4 and then input into network layer b3. Candidate feature 1, the initial feature 3, and its historical features are processed by network layer b3 to obtain candidate feature 2. Candidate feature 2 is stored as a historical feature of network layer b3 and then input into network layer b2. Candidate feature 2, the initial feature 2, and its historical features are processed by network layer b2 to obtain candidate feature 3. Candidate feature 3 is stored as a historical feature of network layer b2 and then input into network layer b1. Candidate feature 3, the initial feature 1, and its historical features are processed by network layer b1 to obtain the target feature. The target feature is stored as a historical feature of network layer b1 and then input into network layer c.

[0106] The processing of network layer c can be carried out in the following manner in this embodiment:

[0107] The network layer c can include K network channels, and the target feature can be input to the K network channels of the network layer c respectively, and the target feature can be processed by the K network channels respectively to obtain a prediction result of the foreground weight image, that is, the target network model can output K foreground weight images. As can be seen, after the third initial image is input to the target network model, the target network model can output K foreground weight images corresponding to the third initial image. On this basis, one foreground weight image can be selected from the K foreground weight images, for example, the foreground weight image output by the first network channel of the target network model is taken as the foreground weight image corresponding to the third initial image, and the foreground region image corresponding to the third initial image can be determined based on the foreground weight image.

[0108] The network layer c can include K network channels, and the target feature can be input to the K network channels of the network layer c respectively, and the target feature can be processed by the K network channels respectively to obtain a prediction result of the foreground weight image, that is, the target network model can output K foreground weight images. As can be seen, after the third initial image is input to the target network model, the target network model can output K foreground weight images corresponding to the third initial image. On this basis, one foreground weight image can be selected from the K foreground weight images, for example, the foreground weight image output by the first network channel of the target network model is taken as the foreground weight image corresponding to the third initial image, and the foreground region image corresponding to the third initial image can be determined based on the foreground weight image.

[0109] Alternatively, after obtaining the target feature and the preset label, if the preset label is used to indicate that the current initial image is not the third initial image (that is, the first initial image), the target feature is input to the K network channels of the network layer c respectively, and the target feature is processed by the K network channels to obtain a prediction result of the foreground weight image, that is, the target network model can output K foreground weight images.

[0110] The third mode, the network layer c can include K network channels. When the third type of initial image is input to the target network model, the third type of initial image and a preset label (the preset label is used to represent the frame number of the current initial image, i.e., the frame number of the initial image in the image sequence) are input to the target network model together, so that the target feature and the preset label can be input to the network layer c together. After the network layer c obtains the target feature and the preset label, if the preset label is less than or equal to M, i.e., the current initial image is the third type of initial image, only the target feature is input to the first network channel of the network layer c, and the target feature is processed by the first network channel to obtain the prediction result of the foreground weight image, i.e., the target network model can output one foreground weight image. As can be seen from the above, after the third type of initial image is input to the target network model, the target network model can output one foreground weight image corresponding to the third type of initial image. This foreground weight image can be used as the foreground weight image corresponding to the third type of initial image, and based on the foreground weight image, the foreground region image corresponding to the third type of initial image can be determined.

[0111] Alternatively, after the network layer c obtains the target feature and the preset label, if the preset label is greater than M, i.e., the current initial image is not the third type of initial image (i.e., the first type of initial image), the target feature is input to the K network channels of the network layer c respectively, and the target feature is processed by the K network channels to obtain the prediction result of the foreground weight image, i.e., the target network model can output K foreground weight images.

[0112] In step 403, for each frame of the first type of initial image, the first type of initial image is input to the target network model, and the target network model outputs K foreground weight images corresponding to the first type of initial image, and the foreground weight image includes the probability that each pixel point in the first type of initial image belongs to the foreground region. In addition, the foreground region image corresponding to the first type of initial image is determined based on the K foreground weight images, and the foreground region images corresponding to the K-1 frames of the second type of initial image after the first type of initial image are determined.

[0113] For example, in the case of the first type of initial image, the first type of initial image and the preset label are input to the target network model together, and the target feature and the preset label are input to the network layer c together. After the network layer c obtains the target feature and the preset label, if the preset label is less than or equal to M, i.e., the current initial image is the third type of initial image, only the target feature is input to the first network channel of the network layer c, and the target feature is processed by the first network channel to obtain the prediction result of the foreground weight image, i.e., the target network model can output one foreground weight image. As can be seen from the above, after the third type of initial image is input to the target network model, the target network model can output one foreground weight image corresponding to the third type of initial image. This foreground weight image can be used as the foreground weight image corresponding to the third type of initial image, and based on the foreground weight image, the foreground region image corresponding to the third type of initial image can be determined. Figure 3BThe illustrated target network model is used as an example for illustration. The first type of initial image can be input to the network layer a1, the first type of initial image is down-sampled by the network layer a1, and the down-sampled first type of initial image is processed to obtain the initial feature 1, the initial feature 1 is input to the network layer a2 and the network layer b1. Then, the initial feature 1 is down-sampled by the network layer a2, and the down-sampled initial feature is processed to obtain the initial feature 2, the initial feature 2 is input to the network layer a3 and the network layer b2. Then, the initial feature 2 is down-sampled by the network layer a3, and the down-sampled initial feature is processed to obtain the initial feature 3, the initial feature 3 is input to the network layer a4 and the network layer b3. Then, the initial feature 3 is down-sampled by the network layer a4, and the down-sampled initial feature is processed to obtain the initial feature 4, the initial feature 4 is input to the network layer b4.

[0114] The initial feature 4 is up-sampled by the network layer b4, and the up-sampled initial feature and the historical feature of the network layer b4 are processed to obtain the candidate feature 1, the candidate feature 1 is stored as the historical feature of the network layer b4, and the candidate feature 1 is input to the network layer b3. Then, the candidate feature 1 and the initial feature 3 are up-sampled by the network layer b3, the up-sampled candidate feature, the up-sampled initial feature, and the historical feature of the network layer b3 are processed to obtain the candidate feature 2, the candidate feature 2 is stored as the historical feature of the network layer b3, and the candidate feature 2 is input to the network layer b2. Then, the candidate feature 2 and the initial feature 2 are up-sampled by the network layer b2, the up-sampled candidate feature, the up-sampled initial feature, and the historical feature of the network layer b2 are processed to obtain the candidate feature 3, the candidate feature 3 is stored as the historical feature of the network layer b2, and the candidate feature 3 is input to the network layer b1. Then, the candidate feature 3 and the initial feature 1 are up-sampled by the network layer b1, the up-sampled candidate feature, the up-sampled initial feature, and the historical feature of the network layer b1 are processed to obtain the target feature, the target feature is stored as the historical feature of the network layer b1, and the target feature is input to the network layer c.

[0115] For example, the network layer c can include K network channels, the target feature can be input to the first network channel of the network layer c, the target feature can be input to the second network channel of the network layer c, and the target feature can be input to the Kth network channel of the network layer c. The target feature is processed by the first network channel of the network layer c to obtain the prediction result of the foreground weight image, which is denoted as foreground weight image 1. The target feature is processed by the second network channel of the network layer c to obtain the prediction result of the foreground weight image, which is denoted as foreground weight image 2. The target feature is processed by the Kth network channel of the network layer c to obtain the prediction result of the foreground weight image, which is denoted as foreground weight image K.

[0116] In summary, after the first type of initial image is input to the target network model, K foreground weight images can be obtained, wherein the foreground weight image 1 can be used as the foreground weight image corresponding to the first type of initial image, the foreground weight image 2 can be used as the foreground weight image corresponding to the first type of initial image followed by the first second type of initial image, and so on, and the foreground weight image K can be used as the foreground weight image corresponding to the first type of initial image followed by the K-1 second type of initial image. Based on the foreground weight image corresponding to the first type of initial image, the foreground region image corresponding to the first type of initial image can be determined. For each second type of initial image, based on the foreground weight image corresponding to the second type of initial image, the foreground region image corresponding to the second type of initial image can be determined. In summary, the foreground region image corresponding to the first type of initial image and the foreground region images corresponding to the K-1 second type of initial images followed by the first type of initial image can be determined based on the K foreground weight images.

[0117] For example, for each frame of initial image (such as the first type of initial image, the second type of initial image, and the third type of initial image), after obtaining the foreground region image corresponding to the initial image, the target image corresponding to the initial image can also be determined, such as by the following formula representing the relationship between each image: Image = alpha * foreground + (1-alpha) * background. alpha represents the prediction result, i.e., the foreground weight image, foreground represents the initial image, alpha * foreground represents the foreground region image, background represents the virtual background image, 1-alpha represents the proportion of the background region, (1-alpha) * background is the new background region, and Image is the final effect image after replacing the background, i.e., the target image.

[0118] In the above embodiment, since the network layer b1, the network layer b2, the network layer b3, and the network layer b4 include the convGRU, in the initialization stage, the history information of the convGRU is empty, and as the initial images are continuously input, the history information extracted by the convGRU will be more and more accurate, and these history information will be continuously updated to participate in the prediction process, i.e., the above-mentioned history features, so that the prediction can be performed based on the history information, and the foreground weight images of multiple frames of initial images (i.e., K frames of initial images, such as one frame of first type of initial image and K-1 frames of second type of initial image) can be predicted based on the history information. Since multiple frames of initial images are predicted at the same time, the computational burden is reduced, and real-time performance can be achieved.

[0119] Among them, due to the continuity between video frames, the historical information contains the matte information of historical frames, and these matte information contains useful information such as the motion trajectory of the target object. Therefore, when the target network model makes auxiliary predictions on the current initial image based on the historical information, it can be considered to predict the matte results of the next K - 1 frames while predicting the matte result of the current initial image. Taking K = 2 as an example, the target network model will simultaneously predict the matte result of the current initial image t and the matte result of the next initial image t + 1 based on the current initial image t and the historical information, that is, in one run, the matte results of two initial images are obtained at the same time. Obviously, when predicting the next initial image t + 1, it can be directly skipped and the prediction of the initial image t + 2 can be continued, so as to achieve twice - speed inference and save computing resources.

[0120] For example, referring to Figure 5A as shown, when K is 1, the target network model outputs a single - channel alpha map. Assuming the width and height of the image are W and H, then Figure 5A the dimension of is 1 * W * H, and the "1" represents 1 channel, indicating that the alpha image is a single - channel grayscale image. When K is 2, only the number of output channels of the target network model needs to be modified. Referring to Figure 5B as shown, the target network model outputs a two - channel alpha map, that is, the dimension is 2 * W * H. When K is 3, referring to Figure 5C as shown, the target network model outputs a three - channel alpha map, that is, the dimension is 3 * W * H. And so on, when K is t, the target network model outputs an alpha map with a dimension of t * W * H. Each dimension represents the prediction results of the current - frame initial image and the initial images of the next i (0 < i < t) frames in sequence, and the prediction result is the foreground weight image.

[0121] As can be seen from the above technical solutions, in this embodiment, only one frame of the first type of initial image needs to be input into the target network model to obtain the foreground region image corresponding to K frames of initial images. That is, when performing a matting operation on one frame of initial image, the matting result of K frames of initial images is obtained, thereby significantly reducing the amount of computation and speeding up the computation. The matting speed is relatively fast, and the matting can be completed in a short time, meeting real-time requirements. Even on devices with poor performance, a smooth and high-precision matting effect can be achieved, while ensuring the high precision requirement of matting. It can fully utilize the continuity information between images to assist in matting, thereby improving the matting effect. It can combine historical information to simultaneously predict the matting result of K frames of initial images, better capturing the motion information of the target object and obtaining higher quality matting results. It can improve the stability and accuracy of matting, improve the matting quality, without increasing hardware costs, and save hardware resources. It fully utilizes the continuity information between video frames, uses a memory module to enable the network to learn and remember historical information, and uses the continuity of video to assist in video matting, thereby improving the video matting effect. By combining historical information, the results of the target object in the current frame and subsequent frames can be predicted simultaneously, which can better capture the motion information of the target object and obtain higher quality matting prediction results.

[0122] Based on the same concept as the above method, this application proposes an image processing apparatus, see [link to relevant documentation]. Figure 6 The diagram shown is a structural schematic of the image processing device, which may include:

[0123] The selection module 61 is used to select a first type of initial image from multiple initial images of the image sequence corresponding to the target object; there are K-1 second type of initial images between two adjacent first type of initial images, where K is greater than 1; the processing module 62 is used to input the first type of initial image into the target network model for each first type of initial image, and the target network model outputs K foreground weight images corresponding to the first type of initial image, the foreground weight images including the probability that each pixel in the first type of initial image belongs to the foreground region; the determination module 63 is used to determine the foreground region images corresponding to the first type of initial image and the K-1 second type of initial images following the first type of initial image based on the K foreground weight images.

[0124] Illustratively, the selecting module 61 is specifically configured to: divide the image sequence into a first sub-sequence and a second sub-sequence when selecting the first type of initial image from the multiple initial images of the image sequence corresponding to the target object, the first sub-sequence including M initial images in front of the image sequence, the second sub-sequence including the remaining initial images of the image sequence other than the M initial images, M being greater than 1; obtain a configured frame interval, the frame interval representing the number of images between two adjacent first type of initial images, and select the first type of initial image from the second sub-sequence based on the frame interval.

[0125] Illustratively, the processing module 62 is further configured to: for each third type of initial image in the first sub-sequence, determine a foreground region image corresponding to the third type of initial image; and specifically configured to: input the third type of initial image into the target network model, output K foreground weight images corresponding to the third type of initial image by the target network model; determine the foreground region image corresponding to the third type of initial image based on one of the K foreground weight images; or input the third type of initial image and a preset label into the target network model, output one foreground weight image corresponding to the third type of initial image by the target network model based on the preset label; and determine the foreground region image corresponding to the third type of initial image based on the foreground weight image.

[0126] Illustratively, the target network model includes a first sub-network and a second sub-network, the second sub-network including a last network layer of the target network model, and the first sub-network including the remaining network layers of the target network model other than the last network layer; and the processing module 62 is specifically configured to: input the first type of initial image into the first sub-network, process the first type of initial image by the first sub-network to obtain a target feature corresponding to the first type of initial image; and input the target feature into K network channels of the second sub-network respectively, process the target feature by each network channel to obtain a foreground weight image corresponding to the network channel.

[0127] Exemplarily, the first sub-network includes T feature extraction network layers and N history information network layers, K and N are both positive integers; the processing module 62 inputs the first type of initial image into the first sub-network, and processes the first type of initial image by the first sub-network to obtain the target feature corresponding to the first type of initial image, and specifically for: processing the first type of initial image by the first feature extraction network layer to obtain an initial feature, and inputting the initial feature into the second feature extraction network layer; based on the second to the T-1th feature extraction network layer, processing the input initial feature by the feature extraction network layer to obtain an initial feature, and inputting the initial feature into the next feature extraction network layer of the feature extraction network layer; processing the input initial feature by the Tth feature extraction network layer to obtain an initial feature, and inputting the initial feature into the Nth history information network layer; processing the input initial feature and the historical feature of the history information network layer by the Nth history information network layer to obtain a candidate feature, storing the candidate feature as the historical feature of the history information network layer, and inputting the candidate feature into the N-1th history information network layer; based on the N-1th to the 2nd history information network layer, processing the input candidate feature and the historical feature of the history information network layer by the history information network layer to obtain a candidate feature, storing the candidate feature as the historical feature of the history information network layer, and inputting the candidate feature into the previous history information network layer of the history information network layer; processing the input candidate feature and the historical feature of the history information network layer by the first history information network layer to obtain the target feature corresponding to the first type of initial image, and storing the target feature as the historical feature of the history information network layer.

[0128] Exemplarily, for each history information network layer, the history information network layer is a network layer based on convGRU, or a network layer based on convLSTM, or a network layer based on RNN, or a network layer based on LSTM, or a network layer based on GRU.

[0129] The processing module 62 is further configured to: obtain a training data set, the training data set comprising P consecutive sample images in a video stream and a sample weight image corresponding to each sample image, P being greater than K; train the configured initial network model based on the training data set to obtain the trained target network model; and specifically configured to: input a sample image into the initial network model, and output a foreground weight image corresponding to the sample image and K-1 foreground weight images corresponding to K-1 sample images behind the sample image by the initial network model; determine a target loss value based on the foreground weight image corresponding to the sample image and the sample weight image, and the K-1 foreground weight images corresponding to the K-1 sample images behind the sample image and the sample weight images; and train the initial network model based on the target loss value to obtain the target network model.

[0130] Based on the same application concept as the above method, an image processing device is provided in the embodiments of the present application, as shown in Figure 7 The image processing device comprises a processor 71 and a machine readable storage medium 72, and the machine readable storage medium 72 stores machine executable instructions that can be executed by the processor 71; and the processor 71 is configured to execute the machine executable instructions to implement the image processing method disclosed in the above examples of the present application.

[0131] Based on the same application concept as the above method, the embodiments of the present application further provide a machine readable storage medium, and the machine readable storage medium stores a plurality of computer instructions, and the computer instructions can implement the image processing method disclosed in the above examples of the present application when executed by a processor.

[0132] The machine readable storage medium can be any electronic, magnetic, optical, or other physical storage device, and can contain or store information such as executable instructions, data, and the like. For example, the machine readable storage medium can be a RAM (Random Access Memory), a volatile memory, a non-volatile memory, a flash memory, a storage drive (such as a hard disk drive), a solid state drive, any type of storage disk (such as a compact disk, a DVD, and the like), or similar storage medium, or a combination thereof.

[0133] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transmitting and receiving device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0134] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0139] The above merely provides an example of the present application, but is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Select a first type of initial image from multiple initial images in the image sequence corresponding to the target object; wherein, there are K-1 second type initial images between two adjacent first type initial images, where K is greater than 1; For each frame of the first type of initial image, the first type of initial image is input into the target network model, and the target network model outputs K foreground weight images corresponding to the first type of initial image. Each foreground weight image includes the probability that each pixel in the first type of initial image belongs to the foreground region. The target network model includes a first sub-network and a second sub-network. The second sub-network includes the last network layer of the target network model, and the first sub-network includes the remaining network layers of the target network model excluding the last network layer. The first sub-network includes T feature extraction network layers and N historical information network layers, where T and N are both positive integers. The process of inputting the first type of initial image into the target network model and having the target network model output the K foreground weight images corresponding to the first type of initial image includes: processing the first type of initial image based on each feature extraction network layer to obtain initial features. The input of the first feature extraction network layer is the first type of initial image. For example, the input of any feature extraction network layer from the 2nd to the Tth feature extraction network layer is the initial feature output of the previous feature extraction network layer; based on each historical information network layer, the input of each historical information network layer is processed, and the output of each historical information network layer is stored as the historical feature of each historical information network layer. The input of the Nth historical information network layer is the output of the Tth feature extraction network layer and the historical feature of this historical information network layer. The input of any historical information network layer from the (N-1)th to the 1st historical information network layer is the output of the next historical network layer and the historical feature of this historical network layer. The output of the 1st historical network layer is the target feature corresponding to the first type of initial image; the target feature is input to the K network channels of the second sub-network, and each network channel processes the target feature to obtain the foreground weight image corresponding to that network channel; Based on the K foreground weight images, the foreground region images corresponding to the first type of initial image and the K-1 frames of the second type of initial image following the first type of initial image are determined.

2. The method according to claim 1, characterized in that, The step of selecting a first type of initial image from multiple initial images of the image sequence corresponding to the target object includes: The image sequence is divided into a first subsequence and a second subsequence. The first subsequence includes the initial M frames of the preceding image sequence, and the second subsequence includes the remaining initial images in the image sequence excluding the initial M frames. M is greater than 1. Obtain the configured frame interval, which represents the number of images between two adjacent first-type initial images, and select first-type initial images from the second sub-sequence based on the frame interval.

3. The method according to claim 2, characterized in that, After dividing the image sequence into a first subsequence and a second subsequence, the method further includes: for each frame of the third type of initial image in the first subsequence, determining the foreground region image corresponding to the third type of initial image; specifically including: The third type of initial image is input into the target network model, which outputs K foreground weight images corresponding to the third type of initial image; based on one of the K foreground weight images, the foreground region image corresponding to the third type of initial image is determined; or, The third type of initial image and the preset label are input into the target network model, and the target network model outputs a foreground weight image corresponding to the third type of initial image based on the preset label; the foreground region image corresponding to the third type of initial image is determined based on the foreground weight image.

4. The method according to claim 1, characterized in that, For each historical information network layer, the historical information network layer is a network layer based on convGRU, or a network layer based on convLSTM, or a network layer based on RNN, or a network layer based on LSTM, or a network layer based on GRU.

5. The method according to claim 1, characterized in that, Before inputting the first type of initial image into the target network model, the method further includes: Obtain a training dataset, which includes P consecutive sample images from the video stream and a sample weight image corresponding to each sample image, where P is greater than K; train the configured initial network model based on the training dataset to obtain the trained target network model; specifically including: The sample image is input into the initial network model, and the initial network model outputs the foreground weight image corresponding to the sample image and the foreground weight images corresponding to the K-1 sample images following the sample image. Based on the foreground weight image and sample weight image corresponding to the sample image, and the foreground weight images and sample weight images corresponding to the K-1 sample images following the sample image, the target loss value is determined. The initial network model is trained based on the target loss value to obtain the target network model.

6. An image processing apparatus, characterized in that, The device includes: The selection module is used to select a first type of initial image from multiple initial images in the image sequence corresponding to the target object; there are K-1 second type of initial images between two adjacent first type initial images, where K is greater than 1; The processing module is used to input the first type of initial image into the target network model for each frame, and output K foreground weight images corresponding to the first type of initial image. Each foreground weight image includes the probability that each pixel in the first type of initial image belongs to the foreground region. The target network model includes a first sub-network and a second sub-network. The second sub-network includes the last network layer of the target network model, and the first sub-network includes the remaining network layers of the target network model excluding the last network layer. The first sub-network includes T feature extraction network layers and N historical information network layers, where T and N are both positive integers. The step of inputting the first type of initial image into the target network model and outputting the K foreground weight images corresponding to the first type of initial image includes: processing the first type of initial image based on each feature extraction network layer to obtain initial features. The input of the first feature extraction network layer is the first type of initial image. For the initial image, the input of any feature extraction network layer from the 2nd to the Tth feature extraction network layer is the initial feature output of the previous feature extraction network layer. Based on each historical information network layer, the input of each historical information network layer is processed, and the output of each historical information network layer is stored as the historical feature of that historical information network layer. The input of the Nth historical information network layer is the output of the Tth feature extraction network layer and the historical feature of this historical information network layer. The input of any historical information network layer from the (N-1)th to the 1st historical information network layer is the output of the next historical information network layer and the historical feature of this historical information network layer. The output of the 1st historical information network layer is the target feature corresponding to the first type of initial image. The target feature is then input to the K network channels of the second sub-network, and each network channel processes the target feature to obtain the foreground weight image corresponding to that network channel. The determination module is used to determine the foreground region image corresponding to the first type of initial image and the K-1 frames of the second type of initial image following the first type of initial image based on the K foreground weight images.

7. The apparatus according to claim 6, Its features are, in, When the selection module selects a first type of initial image from multiple initial images of the image sequence corresponding to the target object, it is specifically used to: divide the image sequence into a first subsequence and a second subsequence, wherein the first subsequence includes the preceding M initial images in the image sequence, and the second subsequence includes the remaining initial images in the image sequence excluding the M initial images, wherein M is greater than 1; obtain the configured frame interval, wherein the frame interval represents the number of images between two adjacent first type initial images, and select a first type of initial image from the second subsequence based on the frame interval; The processing module is further configured to: for each frame of the third type of initial image in the first sub-sequence, determine the foreground region image corresponding to the third type of initial image; specifically, it is configured to: input the third type of initial image to the target network model, and have the target network model output K foreground weight images corresponding to the third type of initial image; determine the foreground region image corresponding to the third type of initial image based on one of the K foreground weight images; or, input the third type of initial image and a preset label to the target network model, and have the target network model output a foreground weight image corresponding to the third type of initial image based on the preset label; determine the foreground region image corresponding to the third type of initial image based on the foreground weight image. For each historical information network layer, the historical information network layer is either a network layer based on convGRU, or a network layer based on convLSTM, or a network layer based on RNN, or a network layer based on LSTM, or a network layer based on GRU. The processing module is further configured to: acquire a training dataset, which includes P consecutive sample images from the video stream and a sample weight image corresponding to each sample image, where P is greater than K; train the configured initial network model based on the training dataset to obtain the trained target network model; specifically, it is configured to: input a sample image into the initial network model, and have the initial network model output the foreground weight image corresponding to the sample image and the foreground weight images corresponding to the K-1 sample images following the sample image; determine the target loss value based on the foreground weight image and sample weight image corresponding to the sample image, and the foreground weight image and sample weight image corresponding to the K-1 sample images following the sample image; and train the initial network model based on the target loss value to obtain the target network model.

8. An image processing device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the steps of the method according to any one of claims 1-5.

Images