Video target segmentation method and device, video target segmentation model training method

Abstract

The present disclosure relates to a video target segmentation method and device, and a training method for a video target segmentation model. The video target segmentation method includes: obtaining a video to be processed; for each image frame in the video to be processed, inputting the target mask of at least one adjacent image frame adjacent to the image frame and the pixel matrix of the image frame into the video target segmentation model to obtain the target mask of the image frame; based on the target mask of each image frame in the video to be processed, performing target segmentation on the video to be processed; wherein the video target segmentation model is trained in the following manner: adjusting the parameters of the video target segmentation model based on the estimated mask obtained by the video target segmentation model for each image frame in the training video sample and the actual target mask of each image frame in the training video sample.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing, and in particular to a video target segmentation method and device, and a video target segmentation model training method. Background Art

[0002] Video target segmentation technology covers a wide range of applications and has huge market potential. It can be applied to short video intelligent editing, special effects production, and short video creation. At present, the video target segmentation technology based on deep learning uses optical flow or feature correlation to capture temporal context information in order to obtain a smoother segmentation result by using temporal information. Although it also obtains better segmentation results, it introduces a huge amount of calculation, reduces the speed performance of the algorithm used by the model, and creates a speed bottleneck for actual deployment applications, especially for mobile applications. Summary of the invention

[0003] The present disclosure provides a video target segmentation method and device, and a video target segmentation model training method, so as to at least solve the problems of large computational complexity and slow convergence speed of the video target segmentation technology based on deep learning in the related art.

[0004] According to a first aspect of an embodiment of the present disclosure, a video target segmentation method is provided, characterized in that it includes: obtaining a video to be processed; for each image frame in the video to be processed, inputting a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame into a video target segmentation model to obtain a target mask of the image frame; based on the target mask of each image frame in the video to be processed, performing target segmentation on the video to be processed; wherein the video target segmentation model is trained in the following manner: adjusting parameters of the video target segmentation model based on an estimated mask obtained by the video target segmentation model for each image frame in a training video sample and an actual target mask for each image frame in the training video sample.

[0005] Optionally, a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: determining a guide coding matrix of at least one adjacent image frame based on the target mask of at least one adjacent image frame adjacent to the image frame, wherein each guide coding matrix of the guide coding matrix of at least one adjacent image frame includes at least one of the following: guide foreground information, guide background information and invalid guide information, the guide foreground information is used to forward guide the video target segmentation model to output the target mask of the image frame, the guide background information is used to reversely guide the video target segmentation model to output the target mask of the image frame, and the invalid guide information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain a target mask of the image frame.

[0006] Optionally, based on a target mask of at least one adjacent image frame adjacent to the image frame, determining a guide coding matrix of at least one adjacent image frame includes: for an image frame in at least one adjacent image frame, obtaining a target mask of the image frame; determining a first position in the target mask of the image frame having a value greater than or equal to a first threshold, using a position in an initial matrix corresponding to the first position as a position of guide foreground information, and adjusting the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; determining a second position in the target mask of the image frame having a value less than or equal to a second threshold, using a position in the initial matrix corresponding to the second position as a position of guide background information, and adjusting the value of the position of the guide background information to a second preset value; determining a third position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold, using a position in the initial matrix corresponding to the third position as a position of guide invalid information, and adjusting the value of the position of the invalid guide information to a third preset value; using the adjusted initial matrix as the guide coding matrix of the image frame.

[0007] Optionally, determining a first position in a target mask of the image frame having a value greater than or equal to a first threshold includes: performing an erosion operation on the target mask of the image frame, and determining a first position in the target mask after the erosion operation having a value greater than or equal to the first threshold.

[0008] Optionally, determining a second position in a target mask of the image frame having a value less than or equal to a second threshold includes: performing a dilation operation on the target mask of the image frame, and determining a second position in a target mask after the dilation operation having a value less than or equal to the second threshold.

[0009] Optionally, a pixel matrix of an image frame and a guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: normalizing the pixel matrix of the image frame to obtain a pixel matrix after processing the image frame; connecting the pixel matrix after processing the image frame and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; and inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0010] Optionally, the pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: inputting the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; inputting the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; and inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0011] According to a second aspect of an embodiment of the present disclosure, a method for training a video target segmentation model is provided, which is characterized by comprising: obtaining a training sample set, wherein the training sample set includes multiple training videos and an actual target mask of each image frame in each training video; for each image frame in the training video, inputting a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame into a video target segmentation model to obtain an estimated mask of the image frame; and training the video target segmentation model based on the estimated mask of each image frame in the training video and the actual target mask of each image frame in the training video.

[0012] Optionally, a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame are input into a video target segmentation model to obtain an estimated mask of the image frame, including: determining a guide coding matrix of at least one adjacent image frame based on the target mask of at least one adjacent image frame adjacent to the image frame, wherein each guide coding matrix of the guide coding matrix of at least one adjacent image frame includes at least one of the following: guide foreground information, guide background information and invalid guide information, the guide foreground information is used to forward guide the video target segmentation model to output the target mask of the image frame, the guide background information is used to reversely guide the video target segmentation model to output the target mask of the image frame, and the invalid guide information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame.

[0013] Optionally, based on a target mask of at least one adjacent image frame adjacent to the image frame, determining a guide coding matrix of at least one adjacent image frame includes: for an image frame in at least one adjacent image frame, obtaining a target mask of the image frame; determining a first position in the target mask of the image frame having a value greater than or equal to a first threshold, using a position in an initial matrix corresponding to the first position as a position of guide foreground information, and adjusting the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; determining a second position in the target mask of the image frame having a value less than or equal to a second threshold, using a position in the initial matrix corresponding to the second position as a position of guide background information, and adjusting the value of the position of the guide background information to a second preset value; determining a third position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold, using a position in the initial matrix corresponding to the third position as a position of guide invalid information, and adjusting the value of the position of the invalid guide information to a third preset value; using the adjusted initial matrix as the guide coding matrix of the image frame.

[0014] Optionally, determining a first position in a target mask of the image frame having a value greater than or equal to a first threshold includes: performing an erosion operation on the target mask of the image frame, and determining a first position in the target mask after the erosion operation having a value greater than or equal to the first threshold.

[0015] Optionally, determining a second position in a target mask of the image frame having a value less than or equal to a second threshold includes: performing a dilation operation on the target mask of the image frame, and determining a second position in a target mask after the dilation operation having a value less than or equal to the second threshold.

[0016] Optionally, a pixel matrix of an image frame and a guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: normalizing the pixel matrix of the image frame to obtain a pixel matrix after processing the image frame; connecting the pixel matrix after processing the image frame and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; and inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0017] Optionally, the pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: inputting the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; inputting the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; and inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0018] Optionally, before training, the parameters of the first three channels of the first convolutional layer of the video target segmentation model are set as the parameters of the first convolutional layer of the image target segmentation model, wherein the image target segmentation model is trained in advance.

[0019] According to three aspects of the embodiments of the present disclosure, a video target segmentation device is provided, characterized in that it includes: a video acquisition unit, configured to acquire a video to be processed; a mask acquisition unit, configured to input, for each image frame in the video to be processed, a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame into a video target segmentation model to obtain a target mask of the image frame; a segmentation unit, configured to perform target segmentation on the video to be processed based on the target mask of each image frame in the video to be processed; wherein the video target segmentation model is trained in the following manner: adjusting parameters of the video target segmentation model based on an estimated mask obtained by the video target segmentation model for each image frame in a training video sample and an actual target mask for each image frame in the training video sample.

[0020] Optionally, the mask acquisition unit is further configured to determine a guiding coding matrix of at least one adjacent image frame based on a target mask of at least one adjacent image frame adjacent to the image frame, wherein each guiding coding matrix of the guiding coding matrix of at least one adjacent image frame includes at least one of the following: guiding foreground information, guiding background information and invalid guiding information, the guiding foreground information is used to forward guide the target mask of the image frame output by the video target segmentation model, the guiding background information is used to reversely guide the target mask of the image frame output by the video target segmentation model, and the invalid guiding information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guiding coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame.

[0021] Optionally, the mask acquisition unit is further configured to acquire a target mask of an image frame for at least one adjacent image frame; determine a first position in the target mask of the image frame having a value greater than or equal to a first threshold, use a position in the initial matrix corresponding to the first position as the position of the guided foreground information, and adjust the value of the position of the guided foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; determine a second position in the target mask of the image frame having a value less than or equal to a second threshold, use a position in the initial matrix corresponding to the second position as the position of the guided background information, and adjust the value of the position of the guided background information to a second preset value; determine a third position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold, use a position in the initial matrix corresponding to the third position as the position of the invalid guided information, and adjust the value of the position of the invalid guided information to a third preset value; and use the adjusted initial matrix as the guided coding matrix of the image frame.

[0022] Optionally, the mask acquisition unit is further configured to perform an erosion operation on the target mask of the image frame, and determine a first position in the target mask after the erosion operation whose value is greater than or equal to a first threshold.

[0023] Optionally, the mask acquisition unit is further configured to perform a dilation operation on the target mask of the image frame, and determine a second position having a value less than or equal to a second threshold in the target mask after the dilation operation.

[0024] Optionally, the mask acquisition unit is further configured to normalize the pixel matrix of the image frame to obtain a pixel matrix after the image frame is processed; connect the pixel matrix after the image frame is processed and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0025] Optionally, the mask acquisition unit is further configured to input the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; input the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connect the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a training device for a video target segmentation model is provided, characterized in that it includes: a sample set acquisition unit, configured to acquire a training sample set, wherein the training sample set includes multiple training videos and an actual target mask of each image frame in each training video; a mask acquisition unit, configured to, for each image frame in the training video, input a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame into the video target segmentation model to obtain an estimated mask of the image frame; and a training unit, configured to train the video target segmentation model based on the estimated mask of each image frame in the training video and the actual target mask of each image frame in the training video.

[0027] Optionally, the mask acquisition unit is further configured to determine a guiding coding matrix of at least one adjacent image frame based on a target mask of at least one adjacent image frame adjacent to the image frame, wherein each guiding coding matrix of the guiding coding matrix of at least one adjacent image frame includes at least one of the following: guiding foreground information, guiding background information and invalid guiding information, the guiding foreground information is used to forward guide the target mask of the image frame output by the video target segmentation model, the guiding background information is used to reversely guide the target mask of the image frame output by the video target segmentation model, and the invalid guiding information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guiding coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame.

[0028] Optionally, the mask acquisition unit is further configured to acquire a target mask of an image frame for at least one adjacent image frame; determine a first position in the target mask of the image frame having a value greater than or equal to a first threshold, use a position in the initial matrix corresponding to the first position as the position of the guided foreground information, and adjust the value of the position of the guided foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; determine a second position in the target mask of the image frame having a value less than or equal to a second threshold, use a position in the initial matrix corresponding to the second position as the position of the guided background information, and adjust the value of the position of the guided background information to a second preset value; determine a third position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold, use a position in the initial matrix corresponding to the third position as the position of the invalid guided information, and adjust the value of the position of the invalid guided information to a third preset value; and use the adjusted initial matrix as the guided coding matrix of the image frame.

[0029] Optionally, the mask acquisition unit is further configured to perform an erosion operation on the target mask of the image frame, and determine a first position in the target mask after the erosion operation whose value is greater than or equal to a first threshold.

[0030] Optionally, the mask acquisition unit is further configured to perform a dilation operation on the target mask of the image frame, and determine a second position having a value less than or equal to a second threshold in the target mask after the dilation operation.

[0031] Optionally, the mask acquisition unit is further configured to normalize the pixel matrix of the image frame to obtain a pixel matrix after the image frame is processed; connect the pixel matrix after the image frame is processed and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0032] Optionally, the mask acquisition unit is further configured to input the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; input the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connect the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0033] Optionally, before training, the parameters of the first three channels of the first convolutional layer of the video target segmentation model are set as the parameters of the first convolutional layer of the image target segmentation model, wherein the image target segmentation model is trained in advance.

[0034] According to the fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement the video target segmentation method and / or the video target segmentation model training method according to the present disclosure.

[0035] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the video target segmentation method and / or the video target segmentation model training method according to the present disclosure.

[0036] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the video target segmentation method and / or the video target segmentation model training method according to the present disclosure.

[0037] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0038] According to the video target segmentation method and device, and the training method of the video target segmentation model disclosed in the present invention, when obtaining the target mask of the image frame, the guide coding matrix of the image frame adjacent to the image frame in the video is introduced, that is, the timing information is effectively integrated in the target segmentation, so that the stability of the video target segmentation result can be significantly improved and the flickering can be reduced without substantially increasing the amount of calculation and time consumption, thereby obtaining a more refined and stable segmentation result. Therefore, the present invention solves the problem of large amount of calculation and slow convergence speed of the video target segmentation technology based on deep learning in the related art.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0041] Figure 1 is a schematic diagram showing an implementation scenario of a video object segmentation method according to an exemplary embodiment of the present disclosure;

[0042] Figure 2 is a flow chart of a video object segmentation method according to an exemplary embodiment;

[0043] Figure 3 is a flowchart of a method for training a video object segmentation model according to an exemplary embodiment;

[0044] Figure 4 is a block diagram of a video object segmentation device according to an exemplary embodiment;

[0045] Figure 5 is a block diagram of a training device for a video object segmentation model according to an exemplary embodiment;

[0046] Figure 6 is a block diagram of an electronic device 600 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0049] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0050] In response to the above problems, the present disclosure provides a video target segmentation model training method and a video target segmentation method, which can solve the problems of large amount of calculation and slow convergence speed in related technologies. The following is an example of segmenting faces in videos for explanation.

[0051] Figure 1 is a schematic diagram showing an implementation scenario of a video object segmentation method according to an exemplary embodiment of the present disclosure, such as Figure 1 The implementation scenario includes a server 100, a user terminal 110 and a user terminal 120, wherein the user terminals are not limited to two, and include but are not limited to mobile phones, personal computers and other devices. The user terminal can be equipped with a camera for acquiring video. The server can be a single server, or a server cluster composed of several servers, or a cloud computing platform or a virtualization center.

[0052] After receiving the request for training the video target segmentation model sent by the user terminals 110 and 120, the server 100 counts the videos received from the user terminals 110 and 120 in the past and annotates the faces in the counted videos to obtain the target mask with the face as the target in each video, and merges the annotated videos together as a training sample set, wherein the training sample set includes multiple training videos and the actual target mask of each image frame in each training video. After obtaining the training sample set, for each image frame in each training video, the server 100 inputs the target mask of at least one adjacent image frame adjacent to the image frame and the pixel matrix of the image frame into the video target segmentation model to obtain the estimated mask of the image frame, and then determines the target loss function based on the estimated mask of each image frame in the training video and the actual target mask of each image frame in the training video, and then adjusts the parameters of the video target segmentation model through the target loss function to train the video target segmentation model. After completing the training of the video target segmentation model, the target mask of any input video to be processed can be obtained through the trained video target segmentation model, so as to accurately segment the target of the video to be processed.

[0053] Next, we will refer to Figures 2 to 5 A video object segmentation method and apparatus, and a video object segmentation model training method according to exemplary embodiments of the present disclosure are described in detail.

[0054] Figure 2 is a flow chart of a video object segmentation method according to an exemplary embodiment. Figure 2 As shown, the video target segmentation method includes the following steps:

[0055] In step S201, a video to be processed is obtained. The video to be processed can be any video that needs to be segmented.

[0056] In step S202, for each image frame in the video to be processed, the target mask of at least one adjacent image frame and the pixel matrix of the image frame are input into the video object segmentation model to obtain the target mask of the image frame. For example, take one of the image frames (hereinafter collectively referred to as the current image frame) as an example for explanation, assuming that the pixel length of the current image frame is H, the pixel width is W, and the target mask of the image frame N frames adjacent to the current image frame has been obtained, denoted as M k ∈R H ×W,k=1,2,...N. Generally speaking, when N is 1 or 2, better segmentation results can be obtained in practical applications. Of course, different frame numbers can be tried according to actual needs. Adjacent frames here refer to the image frames before or after the current image frame in the training video. Different choices can be made according to actual application conditions, such as the N frames before the current image frame, the N frames after the current image frame, or the frames before and after the current image frame to form N frames. Here, The value of the i-th row and j-th column of the target mask of the k-th image frame can be in the range of [0,1]. Assuming that the current image frame is in RGB format (it can also be BGR storage format, etc.), the pixel matrix of the RGB image can be expressed as X∈R H×W×3 In general, the value range of each pixel of X is [0,255].

[0057] According to an exemplary embodiment of the present disclosure, the target mask of at least one adjacent image frame adjacent to the image frame and the pixel matrix of the image frame are input into a video target segmentation model to obtain the target mask of the image frame, including: based on the target mask of at least one adjacent image frame adjacent to the image frame, determining the guide coding matrix of at least one adjacent image frame, wherein each guide coding matrix of the guide coding matrix of at least one adjacent image frame includes at least one of the following: guide foreground information, guide background information and invalid guide information, the guide foreground information is used to forward guide the target mask of the image frame output by the video target segmentation model, the guide background information is used to reversely guide the target mask of the image frame output by the video target segmentation model, and the invalid guide information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame. Through this embodiment, the guide coding matrix can be flexibly set according to different guided objects. For example, the above-mentioned guide coding matrix can be determined based on the target mask of the image frame of N frames adjacent to the current image frame, that is, according to M k , k=1,2,...N to obtain N groups of guided coding matrices. The function of the guided coding matrix is ​​to encode the target mask of the image frame that is N frames adjacent to the current image frame and use it as reference information for calculating the target mask of the current image frame to guide the generation of the target mask of the current image frame, thereby improving the temporal stability of the segmentation result and reducing flickering.

[0058] Specifically, according to different guided objects, the above-mentioned guided coding matrix may include at least one of the following information:

[0059] 1) Guided foreground information: Through specific coding, it is used to provide a target mask of the target object to be segmented in time sequence for the current image frame, and positively guide the video target segmentation module to generate the target mask of the current image frame;

[0060] 2) Guiding background information: Through specific encoding, it is used to provide a target mask as the background in time sequence for the current image frame, and reversely guide the video target segmentation model to generate the target mask of the current image frame;

[0061] 3) Invalid guidance information: This information does not guide the video target segmentation model. It is hoped that the video target segmentation model can play the role of the image target segmentation model, that is, this information can be applied in the following scenarios: when there is no target mask of the adjacent frame, but it is still hoped that the video target segmentation model can operate normally, or the video target segmentation model can simultaneously segment static images and videos. It should be noted that for invalid guidance information, a probability p can be set in the video target segmentation model (the experimental default value is 0.3, which can be adjusted according to the specific situation). The probability p represents a 30% probability of guiding the encoding matrix when encoding an image frame that is N frames adjacent to the current image frame. The full coding is invalid guide coding, that is, the guide coding matrix only contains invalid guide information, thereby improving the robustness of the model.

[0062] The above three types of information are independent of each other and can be combined in different forms according to specific needs. Commonly used combinations are as follows: guide foreground + invalid guide, guide foreground + guide background + invalid guide, guide foreground + guide background.

[0063] According to an exemplary embodiment of the present disclosure, based on a target mask of at least one adjacent image frame adjacent to an image frame, determining a guide coding matrix of at least one adjacent image frame includes: for an image frame in at least one adjacent image frame, obtaining a target mask of the image frame; determining a first position in the target mask of the image frame with a value greater than or equal to a first threshold, using a position corresponding to the first position in the initial matrix as a position of guide foreground information, and adjusting the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix with the same number of rows and columns as the target mask of the image frame; determining a second position in the target mask of the image frame with a value less than or equal to a second threshold, using a position corresponding to the second position in the initial matrix as a position of guide background information, and adjusting the value of the position of the guide background information to a second preset value; determining a third position in the target mask of the image frame with a value greater than the second threshold and less than the first threshold, using a position corresponding to the third position in the initial matrix as a position of guide invalid information, and adjusting the value of the position of the invalid guide information to a third preset value; using the adjusted initial matrix as a guide coding matrix of the image frame. According to this embodiment, at least one adjacent image frame can be easily and quickly converted into a guide coding matrix so as to guide the target segmentation of the current image frame.

[0064] For example, for each image frame in at least one predetermined image frame, the value of the position representing the guided foreground information in the guided coding matrix of the image frame is b+σ, the value representing the position representing the guided background information is b-σ, and the value representing the position representing the invalid guided information is b, wherein the position representing the guided foreground information is a position corresponding to a position in the target mask of the image frame having a value greater than or equal to a first threshold, the position representing the guided background information is a position corresponding to a position in the target mask of the image frame having a value less than or equal to a second threshold, and the position representing the invalid guided information is a position corresponding to a position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold, and b and σ are positive integers. The guided coding matrix determined by this embodiment can conveniently and quickly obtain the required information.

[0065] Specifically, we can remember k=1, 2, ...N is the guide coding matrix for the kth adjacent frame of the current image frame X. In the first coding mode: The value set is {b-σ, b, b+σ}, where b+σ is used to encode the foreground information, b-σ is used to encode the background information, and b is used to encode the invalid information. k The position predicted as the foreground in the image frame (i.e., the position whose value in the target mask of the above image frame is greater than or equal to the first threshold), and then The same position in is set to b+σ; M can be extracted k The position predicted as the background in the target mask of the above image frame (i.e., the position whose value is less than or equal to the second threshold) is then The same position in is set to b-σ; if the background and foreground do not need to be guided, M can be extracted k The position where the value is greater than the second threshold and less than the first threshold The same position in is set to b. In the above, b can be 0 and σ can be 1.

[0066] For another example, for each image frame in at least one predetermined image frame, the value of the position representing the guide foreground information in the guide coding matrix of the image frame is [0, 1], the value of the position representing the guide background information is [1, 0], and the value of the position representing the invalid guide information is [0, 0], wherein the position representing the guide foreground information is a position corresponding to a position in the target mask of the image frame having a value greater than or equal to a first threshold, the position representing the guide background information is a position corresponding to a position in the target mask of the image frame having a value less than or equal to a second threshold, and the position representing the invalid guide information is a position corresponding to a position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold. The guide coding matrix determined by this embodiment can conveniently and quickly obtain the required information.

[0067] Specifically, still remember k=1, 2, ...N is the guide coding matrix for the kth adjacent frame of the current image frame X. In the second coding mode: The value set is {b, b+σ}; for the i-th row and j-th column, if the two elements are [0, 0], it means that the code is invalid guidance information; if the two elements are [0, 1], it means that the code is guidance foreground information; if the two elements are [1, 0], it means that the code is guidance background information. Specifically, M can be extracted k The position predicted as the foreground in the image frame (i.e., the position whose value in the target mask of the above image frame is greater than or equal to the first threshold), and then The same position in is set to [0, 1]; M can be extracted k The position predicted as the background in the target mask of the above image frame (i.e., the position whose value is less than or equal to the second threshold) is then The same position in is set to [1, 0]; if you do not need to guide the background and foreground, you can extract M k The position where the value is greater than the second threshold and less than the first threshold The same position in is set to [0, 0]. In the above, b can be 0 and σ can be 1.

[0068] According to an exemplary embodiment of the present disclosure, determining a first position in a target mask of an image frame having a value greater than or equal to a first threshold includes: performing an erosion operation on the target mask of the image frame, and determining a first position in the target mask after the erosion operation having a value greater than or equal to the first threshold. Through this embodiment, a more optimal guide coding matrix can be obtained.

[0069] According to an exemplary embodiment of the present disclosure, determining a second position in a target mask of an image frame whose value is less than or equal to a second threshold includes: performing an expansion operation on the target mask of the image frame, and determining a second position in a target mask after the expansion operation whose value is less than or equal to the second threshold. Through this embodiment, a more optimal guide coding matrix can be obtained.

[0070] For example, if the guided coding matrix includes guided foreground information, guided background information, and invalid guided information, a feasible method is to convert M k Perform an image corrosion operation and then extract the M after the corrosion operation k The position predicted as the foreground in is set to b+σ; k Perform image dilation operation and extract M after dilation operation k The position predicted as background in M ​​is set to b-σ; k The other areas are set to b.

[0071] According to an exemplary embodiment of the present disclosure, a pixel matrix of an image frame and a guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: normalizing the pixel matrix of the image frame to obtain a pixel matrix after the image frame is processed; connecting the pixel matrix after the image frame is processed and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; and inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame. Through this embodiment, fusion can be performed conveniently and quickly.

[0072] Specifically, after obtaining the guide coding matrix of adjacent N frames After that, the RGB image pixel matrix X and k = 1, 2, ... N, there are many fusion methods, and the present disclosure does not limit this. For example, a simple and feasible method is to directly combine X with the normalized X and the preprocessed X before inputting the video target segmentation model. k=1, 2, ... N are concatenated in the channel dimension. Therefore, in the first encoding mode, the number of input channels of the video object segmentation model is increased from 3 to 3+N. In the second encoding mode, the number of input channels of the video object segmentation model is increased from 3 to 3+2*N. The above normalization process can be to normalize the pixel value of X to between [b-σ, b+σ] (usually, b=0, σ=1, that is, the pixel value of X is normalized to between [-1, 1]).

[0073] According to an exemplary embodiment of the present disclosure, a pixel matrix of an image frame and a guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: inputting the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; inputting the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0074] Specifically, we can first k=1, 2, ...N are sent to several convolutional layers to obtain the output results, and then the pixel matrix X is input into the image target segmentation network (that is, the input of the model is a 3-channel image) to obtain the intermediate output results, and then the above output results and the intermediate output results are concatenated with the channel dimension (channel), and then sent to the subsequent video target segmentation model.

[0075] In step S203, target segmentation is performed on the video to be processed based on the target mask of each image frame in the video to be processed; wherein the video target segmentation model is trained in the following manner: parameters of the video target segmentation model are adjusted based on the estimated mask obtained by the video target segmentation model for each image frame in the training video sample and the actual target mask of each image frame in the training video sample.

[0076] Figure 3 is a flowchart of a method for training a video object segmentation model according to an exemplary embodiment. Figure 3 As shown, the training method of the video object segmentation model includes the following steps:

[0077] In step S301, a training sample set is obtained, wherein the training sample set includes multiple training videos and an actual target mask of each image frame in each training video. For example, the videos received from the user terminals 110 and 120 in the past can be counted as training videos, and the predetermined objects in the counted training videos are annotated to obtain the target mask with the predetermined object as the target in each training video, and the annotated videos are merged together as a training sample set, wherein the predetermined object is the target object to be segmented determined according to actual needs. For another example, the annotated video resources are relatively few and the annotation is time-consuming and laborious, so annotated static pictures can be used to simulate two or three frames of video. In short, the static pictures and their annotated target masks can be randomly enhanced in the same manner through data enhancement methods such as translation, distortion, rotation, radial transformation and thin plate spline interpolation to simulate the scenes of movement, shaking, blurring and the like of objects in the video, thereby obtaining the training videos and corresponding annotated target masks required for training.

[0078] In step S302, for each image frame in the training video, the target mask of at least one adjacent image frame and the pixel matrix of the image frame are input into the video object segmentation model to obtain the estimated mask of the image frame. For example, take one of the image frames (hereinafter collectively referred to as the current image frame) as an example for explanation, assuming that the pixel length of the current image frame is H, the pixel width is W, and the target mask of the image frame N frames adjacent to the current image frame has been obtained, denoted as M k ∈R H×W, k = 1, 2, ... N. Generally speaking, when N is 1 or 2, better segmentation results can be obtained in practical applications. Of course, different frame numbers can be tried according to actual needs. Adjacent frames here refer to the image frames before or after the current image frame in the training video. Different choices can be made according to actual application situations, such as the N frames before the current image frame, the N frames after the current image frame, or a few frames before and after the current image frame to form N frames. Here, The value of the i-th row and j-th column of the target mask of the k-th image frame can be in the range of [0, 1]. Assuming that the current image frame is in RGB format (it can also be BGR storage format, etc.), the pixel matrix of the RGB image can be expressed as X∈R H×W×3 In general, the value range of each pixel of X is [0, 255].

[0079] According to an exemplary embodiment of the present disclosure, the target mask of at least one adjacent image frame adjacent to the image frame and the pixel matrix of the image frame are input into a video target segmentation model to obtain an estimated mask of the image frame, including: based on the target mask of at least one adjacent image frame adjacent to the image frame, determining the guide coding matrix of at least one adjacent image frame, wherein each guide coding matrix of the guide coding matrix of at least one adjacent image frame includes at least one of the following: guide foreground information, guide background information and invalid guide information, the guide foreground information is used to forward guide the target mask of the image frame output by the video target segmentation model, the guide background information is used to reversely guide the target mask of the image frame output by the video target segmentation model, and the invalid guide information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame. Through this embodiment, the guide coding matrix can be flexibly set according to different guided objects. For example, the above-mentioned guide coding matrix can be determined based on the target mask of the image frame of N frames adjacent to the current image frame, that is, according to M k , k = 1, 2, ... N to obtain N groups of guided coding matrices. The role of the guided coding matrix is ​​to encode the target mask of the image frame that is N frames adjacent to the current image frame and use it as reference information for calculating the target mask of the current image frame to guide the generation of the target mask of the current image frame, thereby improving the temporal stability of the segmentation result and reducing flickering.

[0080] Specifically, according to different guided objects, the above-mentioned guided coding matrix may include at least one of the following information:

[0081] 1) Guided foreground information: Through specific coding, it is used to provide a target mask of the target object to be segmented in time sequence for the current image frame, and positively guide the video target segmentation module to generate the target mask of the current image frame;

[0082] 2) Guiding background information: Through specific encoding, it is used to provide a target mask as the background in time sequence for the current image frame, and reversely guide the video target segmentation model to generate the target mask of the current image frame;

[0083] 3) Invalid guidance information: This information does not guide the video target segmentation model. It is hoped that the video target segmentation model can play the role of the image target segmentation model, that is, this information can be applied in the following scenarios: when there is no target mask of the adjacent frame, but it is still hoped that the video target segmentation model can operate normally, or the video target segmentation model can simultaneously segment static images and videos. It should be noted that for invalid guidance information, a probability p can be set in the video target segmentation model (the experimental default value is 0.3, which can be adjusted according to the specific situation). The probability p represents a 30% probability of guiding the encoding matrix when encoding the image frames of N adjacent frames. The full coding is invalid guide coding, that is, the guide coding matrix only contains invalid guide information, thereby improving the robustness of the model.

[0084] The above three types of information are independent of each other and can be combined in different forms according to specific needs. Commonly used combinations are as follows: guide foreground + invalid guide, guide foreground + guide background + invalid guide, guide foreground + guide background.

[0085] According to an exemplary embodiment of the present disclosure, based on a target mask of at least one adjacent image frame adjacent to an image frame, determining a guide coding matrix of at least one adjacent image frame includes: for an image frame in at least one adjacent image frame, obtaining a target mask of the image frame; determining a first position in the target mask of the image frame with a value greater than or equal to a first threshold, using a position corresponding to the first position in the initial matrix as a position of guide foreground information, and adjusting the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix with the same number of rows and columns as the target mask of the image frame; determining a second position in the target mask of the image frame with a value less than or equal to a second threshold, using a position corresponding to the second position in the initial matrix as a position of guide background information, and adjusting the value of the position of the guide background information to a second preset value; determining a third position in the target mask of the image frame with a value greater than the second threshold and less than the first threshold, using a position corresponding to the third position in the initial matrix as a position of guide invalid information, and adjusting the value of the position of the invalid guide information to a third preset value; using the adjusted initial matrix as a guide coding matrix of the image frame. According to this embodiment, at least one adjacent image frame can be easily and quickly converted into a guide coding matrix so as to guide the target segmentation of the current image frame.

[0086] For example, for each image frame in at least one predetermined image frame, the value of the position representing the guide foreground information in the guide coding matrix of the image frame is b+σ, the value representing the position of the guide background information is b-σ, and the value representing the position of the invalid guide information is b, wherein the position representing the guide foreground information is the position corresponding to c, the position representing the guide background information is the position corresponding to the position in the target mask of the image frame whose value is less than or equal to the second threshold, the position representing the invalid guide information is the position corresponding to the position in the target mask of the image frame whose value is greater than the second threshold and less than the first threshold, and b and σ are positive integers. The guide coding matrix determined by this embodiment can easily and quickly obtain the required information.

[0087] Specifically, we can remember k=1, 2, ...N is the guide coding matrix for the kth adjacent frame of the current image frame X. In the first coding mode: The value set is {b-σ, b, b+σ}, where b+σ is used to encode the foreground information, b-σ is used to encode the background information, and b is used to encode the invalid information. k The position predicted as the foreground in the image frame (i.e., the position whose value in the target mask of the above image frame is greater than or equal to the first threshold), and then The same position in is set to b+σ; M can be extracted k The position predicted as the background in the target mask of the above image frame (i.e., the position whose value is less than or equal to the second threshold) is then The same position in is set to b-σ; if the background and foreground do not need to be guided, M can be extracted k The position where the value is greater than the second threshold and less than the first threshold The same position in is set to b. In the above, b can be 0 and σ can be 1.

[0088] For another example, for each image frame in at least one predetermined image frame, the value of the position representing the guide foreground information in the guide coding matrix of the image frame is [0, 1], the value of the position representing the guide background information is [1, 0], and the value of the position representing the invalid guide information is [0, 0], wherein the position representing the guide foreground information is a position corresponding to a position in the target mask of the image frame having a value greater than or equal to a first threshold, the position representing the guide background information is a position corresponding to a position in the target mask of the image frame having a value less than or equal to a second threshold, and the position representing the invalid guide information is a position corresponding to a position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold. The guide coding matrix determined by this embodiment can conveniently and quickly obtain the required information.

[0089] Specifically, still remember k=1, 2, ...N is the guide coding matrix for the kth adjacent frame of the current image frame X. In the second coding mode: The value set is {b, b+σ}; for the i-th row and j-th column, if the two elements are [0, 0], it means that the code is invalid guidance information; if the two elements are [0, 1], it means that the code is guidance foreground information; if the two elements are [1, 0], it means that the code is guidance background information. Specifically, M can be extracted k The position predicted as the foreground in the image frame (i.e., the position whose value in the target mask of the above image frame is greater than or equal to the first threshold), and then The same position in is set to [0, 1]; M can be extracted k The position predicted as the background in the target mask of the above image frame (i.e., the position whose value is less than or equal to the second threshold) is then The same position in is set to [1, 0]; if you do not need to guide the background and foreground, you can extract M k The position where the value is greater than the second threshold and less than the first threshold The same position in is set to [0, 0]. In the above, b can be 0 and σ can be 1.

[0090] According to an exemplary embodiment of the present disclosure, determining a first position in a target mask of an image frame having a value greater than or equal to a first threshold includes: performing an erosion operation on the target mask of the image frame, and determining a first position in the target mask after the erosion operation having a value greater than or equal to the first threshold. Through this embodiment, a more optimal guide coding matrix can be obtained.

[0091] According to an exemplary embodiment of the present disclosure, determining a second position in a target mask of an image frame whose value is less than or equal to a second threshold includes: performing an expansion operation on the target mask of the image frame, and determining a second position in a target mask after the expansion operation whose value is less than or equal to the second threshold. Through this embodiment, a more optimal guide coding matrix can be obtained.

[0092] For example, if the guided coding matrix includes guided foreground information, guided background information, and invalid guided information, a feasible method is to convert M k Perform an image corrosion operation and then extract the M after the corrosion operation k The position predicted as the foreground in is set to b+σ; k Perform image dilation operation and extract M after dilation operation k The position predicted as background in M ​​is set to b-σ; k The other areas are set to b.

[0093] According to an exemplary embodiment of the present disclosure, a pixel matrix of an image frame and a guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: normalizing the pixel matrix of the image frame to obtain a pixel matrix after the image frame is processed; connecting the pixel matrix after the image frame is processed and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; and inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame. Through this embodiment, fusion can be performed conveniently and quickly.

[0094] Specifically, after obtaining the guide coding matrix of adjacent N frames After that, the RGB image pixel matrix X and k = 1, 2, ... N, there are many fusion methods, and the present disclosure does not limit this. For example, a simple and feasible method is to directly combine X with the normalized X and the preprocessed X before inputting the video target segmentation model. k=1, 2, ... N are concatenated in the channel dimension. Therefore, in the first encoding mode, the number of input channels of the video object segmentation model is increased from 3 to 3+N. In the second encoding mode, the number of input channels of the video object segmentation model is increased from 3 to 3+2*N. The above normalization process can be to normalize the pixel value of X to between [b-σ, b+σ] (usually, b=0, σ=1, that is, the pixel value of X is normalized to between [-1, 1]).

[0095] According to an exemplary embodiment of the present disclosure, the pixel matrix of an image frame and the guide coding matrix of at least one adjacent image frame are input into a video target segmentation model to obtain a target mask of the image frame, including: inputting the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; inputting the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame. Through this embodiment, a better fusion result can be obtained.

[0096] Specifically, we can first k=1, 2, ...N are sent to several convolutional layers to obtain the output results, and then the pixel matrix X is input into the image target segmentation network (that is, the input of the model is a 3-channel image) to obtain the intermediate output results, and then the above output results and the intermediate output results are concatenated with the channel dimension (channel), and then sent to the subsequent video target segmentation model.

[0097] return Figure 3 In step S303, the video target segmentation model is trained based on the estimated mask of each image frame in the training video and the actual target mask of each image frame in the training video. In step S303, the target loss function can be determined based on the estimated mask of each image frame in the training video and the annotated target mask corresponding to the training video, and then the parameters of the video target segmentation model are adjusted by minimizing the target loss to perform training.

[0098] According to an exemplary embodiment of the present disclosure, before training, the parameters of the first three channels of the first convolutional layer of the video target segmentation model are set as the parameters of the first convolutional layer of the image target segmentation model, wherein the image target segmentation model is trained in advance. Through this embodiment, the parameters of the first three channels of the first layer of the video target segmentation model before training can be directly copied to the image target segmentation model, and training on the basis of good parameters can obtain better training results.

[0099] Specifically, when there is already an image target segmentation model (i.e., without integrating the above-mentioned guided coding matrix In the model with k=1, 2, ...N, the number of input channels of the model is still 3), in order to make the convergence speed of training the video target segmentation model faster and obtain the excellent performance of the image target segmentation model when using invalid guidance messages, the parameters in the image target segmentation model can be initialized to the video target segmentation model, that is, for the first convolutional layer of the model, it is only necessary to copy the parameters of the first convolutional layer in the image target segmentation model to the positions of the first three channel feature layers of the first convolutional layer of the video target segmentation model.

[0100] According to an exemplary embodiment of the present disclosure, an image target segmentation model is trained in the following manner: obtaining a second training sample set, wherein the second training sample set includes multiple training images and annotated target masks corresponding to the multiple training images; inputting the training images into the image segmentation model to obtain estimated masks of the training images; determining a target loss function based on the estimated masks and the annotated target masks corresponding to the training images; and adjusting the parameters of the image target segmentation model through the target loss function to complete the training of the image target segmentation model.

[0101] In summary, the present disclosure proposes a video target segmentation method based on adjacent frames to address the stability and other issues in video target segmentation technology, that is, obtaining the target masks of N adjacent frames (N is an integer greater than 0) of the current image frame in the video, and after guide encoding, merging them with the RGB picture of the current image frame (generally refers to a picture, which can be stored in RGB mode, BGR mode, etc.) and sending them to the video target segmentation model, calculating the target mask of the current image frame, and then performing target segmentation on the video based on the target mask of each image frame, effectively integrating the timing information in the video, and can significantly improve the stability of the video target segmentation result without basically increasing the amount of calculation and time consumption, significantly reduce flickering, and obtain more refined and stable segmentation results. Moreover, the scheme is extremely universal and can be applied to various subdivision directions of video processing.

[0102] Figure 4 is a block diagram of a video object segmentation device according to an exemplary embodiment. Figure 4 The device includes a video acquisition unit 40, a mask acquisition unit 42 and a segmentation unit 44.

[0103] The video acquisition unit 40 is configured to acquire the video to be processed; the mask acquisition unit 42 is configured to input the target mask of at least one adjacent image frame adjacent to the image frame and the pixel matrix of the image frame into the video target segmentation model for each image frame in the video to be processed, so as to obtain the target mask of the image frame; the segmentation unit 44 is configured to perform target segmentation on the video to be processed based on the target mask of each image frame in the video to be processed; wherein the video target segmentation model is trained in the following manner: adjusting the parameters of the video target segmentation model based on the estimated mask obtained by the video target segmentation model for each image frame in the training video sample and the actual target mask of each image frame in the training video sample.

[0104] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 42 is further configured to determine a guiding coding matrix of at least one adjacent image frame based on a target mask of at least one adjacent image frame adjacent to the image frame, wherein each guiding coding matrix of the guiding coding matrix of at least one adjacent image frame includes at least one of the following: guiding foreground information, guiding background information and invalid guiding information, the guiding foreground information is used to forward guide the target mask of the image frame output by the video target segmentation model, the guiding background information is used to reversely guide the target mask of the image frame output by the video target segmentation model, and the invalid guiding information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guiding coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame.

[0105] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 42 is further configured to acquire a target mask of an image frame in at least one adjacent image frame; determine a first position in the target mask of the image frame having a value greater than or equal to a first threshold, use the position corresponding to the first position in the initial matrix as the position of the guided foreground information, and adjust the value of the position of the guided foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; determine a second position in the target mask of the image frame having a value less than or equal to a second threshold, use the position corresponding to the second position in the initial matrix as the position of the guided background information, and adjust the value of the position of the guided background information to a second preset value; determine a third position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold, use the position corresponding to the third position in the initial matrix as the position of the guided invalid information, and adjust the value of the position of the invalid guided information to a third preset value; use the adjusted initial matrix as the guided coding matrix of the image frame.

[0106] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 42 is further configured to perform an erosion operation on the target mask of the image frame, and determine a first position in the target mask after the erosion operation whose value is greater than or equal to a first threshold.

[0107] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 42 is further configured to perform a dilation operation on the target mask of the image frame, and determine a second position having a value less than or equal to a second threshold in the target mask after the dilation operation.

[0108] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 42 is further configured to normalize the pixel matrix of the image frame to obtain a pixel matrix after the image frame is processed; connect the pixel matrix after the image frame is processed and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0109] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 42 is further configured to input the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; input the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connect the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0110] Figure 5is a block diagram of a training device for a video object segmentation model according to an exemplary embodiment. Figure 5 The device includes a sample set acquisition unit 50, a mask acquisition unit 52 and a training unit 54.

[0111] The sample set acquisition unit 50 is configured to acquire a training sample set, wherein the training sample set includes multiple training videos and an actual target mask of each image frame in each training video; the mask acquisition unit 52 is configured to, for each image frame in the training video, input the target mask of at least one adjacent image frame adjacent to the image frame and the pixel matrix of the image frame into the video target segmentation model to obtain an estimated mask of the image frame; the training unit 54 is configured to train the video target segmentation model based on the estimated mask of each image frame in the training video and the actual target mask of each image frame in the training video.

[0112] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 52 is further configured to determine a guiding coding matrix of at least one adjacent image frame based on a target mask of at least one adjacent image frame adjacent to the image frame, wherein each guiding coding matrix of the guiding coding matrix of at least one adjacent image frame includes at least one of the following: guiding foreground information, guiding background information and invalid guiding information, the guiding foreground information is used to forward guide the target mask of the image frame output by the video target segmentation model, the guiding background information is used to reversely guide the target mask of the image frame output by the video target segmentation model, and the invalid guiding information has no guiding effect on the target mask of the image frame output by the video target segmentation model; the pixel matrix of the image frame and the guiding coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame.

[0113] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 52 is further configured to acquire a target mask of an image frame in at least one adjacent image frame; determine a first position in the target mask of the image frame having a value greater than or equal to a first threshold, use the position corresponding to the first position in the initial matrix as the position of the guided foreground information, and adjust the value of the position of the guided foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; determine a second position in the target mask of the image frame having a value less than or equal to a second threshold, use the position corresponding to the second position in the initial matrix as the position of the guided background information, and adjust the value of the position of the guided background information to a second preset value; determine a third position in the target mask of the image frame having a value greater than the second threshold and less than the first threshold, use the position corresponding to the third position in the initial matrix as the position of the guided invalid information, and adjust the value of the position of the invalid guided information to a third preset value; use the adjusted initial matrix as the guided coding matrix of the image frame.

[0114] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 52 is further configured to perform an erosion operation on the target mask of the image frame, and determine a first position in the target mask after the erosion operation whose value is greater than or equal to a first threshold.

[0115] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 52 is further configured to perform a dilation operation on the target mask of the image frame, and determine a second position having a value less than or equal to a second threshold in the target mask after the dilation operation.

[0116] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 52 is further configured to normalize the pixel matrix of the image frame to obtain a pixel matrix after the image frame is processed; connect the pixel matrix after the image frame is processed and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0117] According to an exemplary embodiment of the present disclosure, the mask acquisition unit 52 is further configured to input the guide coding matrix of at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of at least one adjacent image frame; input the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connect the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

[0118] According to an exemplary embodiment of the present disclosure, before training, the parameters of the first three channels of the first convolutional layer of the video target segmentation model are set as the parameters of the first convolutional layer of the image target segmentation model, wherein the image target segmentation model is trained in advance.

[0119] According to an embodiment of the present disclosure, an electronic device may be provided. Figure 6 It is a block diagram of an electronic device 600 according to an embodiment of the present disclosure, and the electronic device includes at least one memory 601 and at least one processor 602, wherein the at least one memory stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor, the training method of the video target segmentation model and the video target segmentation method according to the embodiment of the present disclosure are executed.

[0120] As an example, the electronic device 600 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device 600 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0121] In the electronic device 600, the processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 602 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0122] The processor 602 can execute instructions or codes stored in the memory, wherein the memory 601 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0123] The memory 601 may be integrated with the processor 602, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 601 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 601 and the processor 602 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 602 can read files stored in the memory 601.

[0124] In addition, the electronic device 600 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0125] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the video target segmentation model and the video target segmentation method of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0126] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the video target segmentation model training method and video target segmentation method of the embodiment of the present disclosure.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video object segmentation method, characterized in that: include: Get the video to be processed; For each image frame in the video to be processed, inputting a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame into a video target segmentation model to obtain a target mask of the image frame; Based on the target mask of each image frame in the video to be processed, performing target segmentation on the video to be processed; The video target segmentation model is trained in the following manner: adjusting the parameters of the video target segmentation model based on the estimated mask obtained by the video target segmentation model for each image frame in the training video sample and the actual target mask of each image frame in the training video sample; The step of inputting the target mask of at least one adjacent image frame to the image frame and the pixel matrix of the image frame into the video target segmentation model to obtain the target mask of the image frame includes: Based on a target mask of at least one adjacent image frame adjacent to the image frame, determining a guide coding matrix of the at least one adjacent image frame, wherein each guide coding matrix of the at least one adjacent image frame includes at least one of the following: guide foreground information, guide background information, and invalid guide information, the guide foreground information is used to forward guide the video target segmentation model to output the target mask of the image frame, the guide background information is used to reversely guide the video target segmentation model to output the target mask of the image frame, and the invalid guide information has no guiding effect on the video target segmentation model to output the target mask of the image frame; The pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into the video object segmentation model to obtain the object mask of the image frame.

2. The video object segmentation method according to claim 1, characterized in that: The step of determining a guide coding matrix of at least one adjacent image frame based on a target mask of at least one adjacent image frame adjacent to the image frame comprises: For an image frame in the at least one adjacent image frame, Acquire a target mask of the image frame; Determine a first position in the target mask of the image frame whose value is greater than or equal to a first threshold, use a position in the initial matrix corresponding to the first position as a position of guide foreground information, and adjust the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; Determine a second position in the target mask of the image frame whose value is less than or equal to a second threshold, use the position in the initial matrix corresponding to the second position as the position of the guiding background information, and adjust the value of the position of the guiding background information to a second preset value; Determine a third position in the target mask of the image frame whose value is greater than the second threshold and less than the first threshold, use the position in the initial matrix corresponding to the third position as the position of invalid guidance information, and adjust the value of the position of the invalid guidance information to a third preset value; The adjusted initial matrix is ​​used as the guide coding matrix of the image frame.

3. The video object segmentation method according to claim 2, characterized in that: The determining of a first position in the target mask of the image frame having a value greater than or equal to a first threshold comprises: performing an erosion operation on the target mask of the image frame, A first position in the target mask after the etching operation having a value greater than or equal to a first threshold is determined.

4. The video object segmentation method according to claim 2, characterized in that: The determining of a second position in the target mask of the image frame having a value less than or equal to a second threshold comprises: Performing a dilation operation on the target mask of the image frame, A second position in the target mask after the dilation operation and having a value less than or equal to a second threshold is determined.

5. The video object segmentation method according to claim 1, characterized in that: The step of inputting a pixel matrix of the image frame and a guide coding matrix of at least one adjacent image frame into the video object segmentation model to obtain an object mask of the image frame comprises: Normalizing the pixel matrix of the image frame to obtain a pixel matrix after processing the image frame; Connecting the processed pixel matrix of the image frame and the guide coding matrix of the at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; The fused matrix is ​​input into the video object segmentation model to obtain the object mask of the image frame.

6. The video object segmentation method according to claim 1, characterized in that: The step of inputting a pixel matrix of the image frame and a guide coding matrix of at least one adjacent image frame into the video object segmentation model to obtain an object mask of the image frame comprises: Inputting the guided coding matrix of the at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guided coding matrix of the at least one adjacent image frame; Inputting the pixel matrix of the image frame into an image object segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; Connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; The fused matrix is ​​input into the video object segmentation model to obtain the object mask of the image frame.

7. A method for training a video object segmentation model, characterized in that: include: Acquire a training sample set, wherein the training sample set includes a plurality of training videos and an actual target mask of each image frame in each training video; For each image frame in the training video, inputting the target mask of at least one adjacent image frame adjacent to the image frame and the pixel matrix of the image frame into the video target segmentation model to obtain an estimated mask of the image frame; Training the video object segmentation model based on the estimated mask of each image frame in the training video and the actual object mask of each image frame in the training video; The step of inputting the target mask of at least one adjacent image frame to the image frame and the pixel matrix of the image frame into the video target segmentation model to obtain the estimated mask of the image frame includes: Based on a target mask of at least one adjacent image frame adjacent to the image frame, determining a guide coding matrix of the at least one adjacent image frame, wherein each guide coding matrix of the at least one adjacent image frame includes at least one of the following: guide foreground information, guide background information, and invalid guide information, the guide foreground information is used to forward guide the video target segmentation model to output the target mask of the image frame, the guide background information is used to reversely guide the video target segmentation model to output the target mask of the image frame, and the invalid guide information has no guiding effect on the video target segmentation model to output the target mask of the image frame; The pixel matrix of the image frame and the guide coding matrix of at least one adjacent image frame are input into the video object segmentation model to obtain the object mask of the image frame.

8. The training method according to claim 7, characterized in that: The step of determining a guide coding matrix of at least one adjacent image frame based on a target mask of at least one adjacent image frame adjacent to the image frame comprises: For an image frame in the at least one adjacent image frame, Acquire a target mask of the image frame; Determine a first position in the target mask of the image frame whose value is greater than or equal to a first threshold, use a position in the initial matrix corresponding to the first position as a position of guide foreground information, and adjust the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix having the same number of rows and columns as the target mask of the image frame; Determine a second position in the target mask of the image frame whose value is less than or equal to a second threshold, use the position in the initial matrix corresponding to the second position as the position of the guiding background information, and adjust the value of the position of the guiding background information to a second preset value; Determine a third position in the target mask of the image frame whose value is greater than the second threshold and less than the first threshold, use the position in the initial matrix corresponding to the third position as the position of invalid guidance information, and adjust the value of the position of the invalid guidance information to a third preset value; The adjusted initial matrix is ​​used as the guide coding matrix of the image frame.

9. The training method according to claim 8, characterized in that: The determining of a first position in the target mask of the image frame having a value greater than or equal to a first threshold comprises: performing an erosion operation on the target mask of the image frame, A first position in the target mask after the etching operation having a value greater than or equal to a first threshold is determined.

10. The training method according to claim 8, characterized in that: The determining of a second position in the target mask of the image frame having a value less than or equal to a second threshold comprises: Performing a dilation operation on the target mask of the image frame, A second position in the target mask after the dilation operation and having a value less than or equal to a second threshold is determined.

11. The training method according to claim 7, characterized in that: The step of inputting a pixel matrix of the image frame and a guide coding matrix of at least one adjacent image frame into the video object segmentation model to obtain an object mask of the image frame comprises: Normalizing the pixel matrix of the image frame to obtain a pixel matrix after processing the image frame; Connecting the processed pixel matrix of the image frame and the guide coding matrix of the at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; The fused matrix is ​​input into the video object segmentation model to obtain the object mask of the image frame.

12. The training method according to claim 7, characterized in that: The step of inputting a pixel matrix of the image frame and a guide coding matrix of at least one adjacent image frame into the video object segmentation model to obtain an object mask of the image frame comprises: Inputting the guided coding matrix of the at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guided coding matrix of the at least one adjacent image frame; Inputting the pixel matrix of the image frame into an image object segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; Connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; The fused matrix is ​​input into the video object segmentation model to obtain the object mask of the image frame.

13. The training method according to claim 7, characterized in that: Before training, the parameters of the first three channels of the first convolutional layer of the video target segmentation model are set as the parameters of the first convolutional layer of the image target segmentation model, wherein the image target segmentation model is trained in advance.

14. A video object segmentation device, characterized in that: include: A video acquisition unit, configured to acquire a video to be processed; A mask acquisition unit is configured to input, for each image frame in the video to be processed, a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame into a video target segmentation model to obtain a target mask of the image frame; A segmentation unit is configured to perform target segmentation on the video to be processed based on the target mask of each image frame in the video to be processed; The video target segmentation model is trained in the following manner: adjusting the parameters of the video target segmentation model based on the estimated mask obtained by the video target segmentation model for each image frame in the training video sample and the actual target mask of each image frame in the training video sample; Wherein, the mask acquisition unit is further configured to determine the guiding coding matrix of at least one adjacent image frame adjacent to the image frame based on the target mask of at least one adjacent image frame, wherein each guiding coding matrix in the guiding coding matrix of the at least one adjacent image frame includes at least one of the following: guiding foreground information, guiding background information and invalid guiding information, the guiding foreground information is used to forward guide the video target segmentation model to output the target mask of the image frame, the guiding background information is used to reversely guide the video target segmentation model to output the target mask of the image frame, and the invalid guiding information has no guiding effect on the video target segmentation model to output the target mask of the image frame; the pixel matrix of the image frame and the guiding coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame.

15. The video object segmentation device according to claim 14, characterized in that: The mask acquisition unit is further configured to acquire a target mask of an image frame in the at least one adjacent image frame; Determine a first position in the target mask of the image frame whose value is greater than or equal to a first threshold, use the position in the initial matrix corresponding to the first position as the position of the guide foreground information, and adjust the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix with the same number of rows and columns as the target mask of the image frame; determine a second position in the target mask of the image frame whose value is less than or equal to a second threshold, use the position in the initial matrix corresponding to the second position as the position of the guide background information, and adjust the value of the position of the guide background information to a second preset value; determine a third position in the target mask of the image frame whose value is greater than the second threshold and less than the first threshold, use the position in the initial matrix corresponding to the third position as the position of the guide invalid information, and adjust the value of the position of the invalid guide information to a third preset value; use the adjusted initial matrix as the guide coding matrix of the image frame.

16. The video object segmentation device according to claim 15, characterized in that: The mask acquisition unit is further configured to perform an erosion operation on the target mask of the image frame, and determine a first position in the target mask after the erosion operation whose value is greater than or equal to a first threshold.

17. The video object segmentation device according to claim 15, characterized in that: The mask acquisition unit is further configured to perform a dilation operation on the target mask of the image frame, and determine a second position in the target mask after the dilation operation whose value is less than or equal to a second threshold.

18. The video object segmentation device according to claim 14, characterized in that: The mask acquisition unit is further configured to normalize the pixel matrix of the image frame to obtain the pixel matrix of the image frame after processing; connect the pixel matrix of the image frame after processing and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain the target mask of the image frame.

19. The video object segmentation device according to claim 14, characterized in that: The mask acquisition unit is further configured to input the guide coding matrix of the at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of the at least one adjacent image frame; Inputting the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

20. A training device for a video object segmentation model, characterized in that: include: A sample set acquisition unit is configured to acquire a training sample set, wherein the training sample set includes a plurality of training videos and an actual target mask of each image frame in each training video; A mask acquisition unit is configured to input, for each image frame in the training video, a target mask of at least one adjacent image frame adjacent to the image frame and a pixel matrix of the image frame into a video target segmentation model to obtain an estimated mask of the image frame; A training unit, configured to train the video object segmentation model based on the estimated mask of each image frame in the training video and the actual object mask of each image frame in the training video; Wherein, the mask acquisition unit is further configured to determine the guiding coding matrix of at least one adjacent image frame adjacent to the image frame based on the target mask of at least one adjacent image frame, wherein each guiding coding matrix in the guiding coding matrix of the at least one adjacent image frame includes at least one of the following: guiding foreground information, guiding background information and invalid guiding information, the guiding foreground information is used to forward guide the video target segmentation model to output the target mask of the image frame, the guiding background information is used to reversely guide the video target segmentation model to output the target mask of the image frame, and the invalid guiding information has no guiding effect on the video target segmentation model to output the target mask of the image frame; the pixel matrix of the image frame and the guiding coding matrix of at least one adjacent image frame are input into the video target segmentation model to obtain the target mask of the image frame.

21. The training device according to claim 20, characterized in that The mask acquisition unit is further configured to acquire a target mask of an image frame in the at least one adjacent image frame; Determine a first position in the target mask of the image frame whose value is greater than or equal to a first threshold, use the position in the initial matrix corresponding to the first position as the position of the guide foreground information, and adjust the value of the position of the guide foreground information to a first preset value, wherein the initial matrix is ​​a unit matrix with the same number of rows and columns as the target mask of the image frame; determine a second position in the target mask of the image frame whose value is less than or equal to a second threshold, use the position in the initial matrix corresponding to the second position as the position of the guide background information, and adjust the value of the position of the guide background information to a second preset value; determine a third position in the target mask of the image frame whose value is greater than the second threshold and less than the first threshold, use the position in the initial matrix corresponding to the third position as the position of the guide invalid information, and adjust the value of the position of the invalid guide information to a third preset value; use the adjusted initial matrix as the guide coding matrix of the image frame.

22. The training device according to claim 21, characterized in that The mask acquisition unit is further configured to perform an erosion operation on the target mask of the image frame, and determine a first position in the target mask after the erosion operation whose value is greater than or equal to a first threshold.

23. The training device according to claim 21, characterized in that The mask acquisition unit is further configured to perform a dilation operation on the target mask of the image frame, and determine a second position in the target mask after the dilation operation whose value is less than or equal to a second threshold.

24. The training device according to claim 20, characterized in that The mask acquisition unit is further configured to normalize the pixel matrix of the image frame to obtain the pixel matrix of the image frame after processing; connect the pixel matrix of the image frame after processing and the guide coding matrix of at least one adjacent image frame in the dimension of the number of channels to obtain a fused matrix; input the fused matrix into the video target segmentation model to obtain the target mask of the image frame.

25. The training device according to claim 20, characterized in that The mask acquisition unit is further configured to input the guide coding matrix of the at least one adjacent image frame into a predetermined number of convolutional layers to obtain a processed guide coding matrix of the at least one adjacent image frame; Inputting the pixel matrix of the image frame into the image target segmentation model to obtain a predetermined intermediate output result, wherein the predetermined intermediate output result has the same resolution as the processed guide coding matrix; connecting the predetermined intermediate output result and the processed guide coding matrix in the dimension of the number of channels to obtain a fused matrix; inputting the fused matrix into the video target segmentation model to obtain a target mask of the image frame.

26. The training device according to claim 20, characterized in that Before training, the parameters of the first three channels of the first convolutional layer of the video target segmentation model are set as the parameters of the first convolutional layer of the image target segmentation model, wherein the image target segmentation model is trained in advance.

27. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video target segmentation method according to any one of claims 1 to 6 and / or the training method of the video target segmentation model according to any one of claims 7 to 13.

28. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the video target segmentation method described in any one of claims 1 to 6 and / or the training method of the video target segmentation model described in any one of claims 7 to 13.

29. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the video target segmentation method according to any one of claims 1 to 6 and / or the training method of the video target segmentation model according to any one of claims 7 to 13 are implemented.

Images