A video face segmentation method, device and equipment based on time sequence information

Abstract

The present invention discloses a method, device, equipment and storage medium for video face segmentation based on temporal information, which comprises: constructing and training a face segmentation model; wherein the face segmentation model is provided with a TemporalModule module for capturing the temporal information between video frames; inputting face video data into the face segmentation model and performing video face segmentation processing based on the temporal information to obtain a target face segmentation result. It can achieve the effect of no jump between different frames in video segmentation, and improve the stability of video segmentation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video face segmentation method, device and equipment based on time sequence information. Background Art

[0002] Scene parsing is to assign a label to each pixel in an image. It is a basic research topic in computer vision and has many applications in image editing, autonomous driving and other fields. Face segmentation can be regarded as a basic task of scene parsing. The results of face segmentation can be used for various tasks such as face editing. The segmentation task in real scenes is actually based on video rather than static (image). With the continuous emergence of various short video applications, massive video content has also been generated. Video parsing has also become a new demand. Therefore, obtaining high-quality video face segmentation is a very important task. However, the existing video face segmentation method ignores the timing information of the video by directly using the image segmentation model, which causes the problem of jumps between different frames of the video. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a video face segmentation method, device and equipment based on time sequence information, aiming to solve the problem of jumps between different frames of the video caused by the existing video face segmentation.

[0004] To achieve the above object, the present invention provides a video face segmentation method based on time sequence information, the method comprising:

[0005] Constructing and training a face segmentation model; wherein the face segmentation model is provided with a TemporalModule module for capturing timing information between video frames;

[0006] The face video data is input into the face segmentation model and the video face segmentation processing is performed based on the timing information to obtain the target face segmentation result.

[0007] Preferably, the face segmentation model is constructed based on the Encoder-Decoder framework, and the face segmentation model includes a Segment Module module and a Refine Module module, and the Segment Module module includes a Temporal Module module for capturing timing information between video frames; wherein,

[0008] The input video data and image data are roughly segmented by the SegmentModule module to obtain a Coarse-Mask, wherein the image data includes a face image in each video frame in the video data;

[0009] Fine-tune the Coarse-Mask through the Refine Module module to obtain a Refine-Mask;

[0010] The Coarse-Mask and the Refine-Mask are added to obtain a predicted Mask result.

[0011] Preferably, the training process of the face segmentation model includes:

[0012] Inputting the image data into the DECA model to generate a corresponding face video sequence to obtain expanded video data;

[0013] After data enhancement is performed on the expanded video data and the image data respectively, the data are input into the face segmentation model for joint training.

[0014] Preferably, the step of inputting the image data into a DECA model to generate a corresponding face video sequence to obtain expanded video data includes:

[0015] Inputting the image data into the DECA model to obtain corresponding 3D information;

[0016] The facial posture is adjusted by controlling the pitch angle, yaw angle and roll angle of the image data, and random illumination transformation is added to obtain a 3D face corresponding to continuous angle changes;

[0017] The 3D face is projected into a 2D face, and the 2D face is combined with the image data to generate a corresponding face video sequence to obtain the expanded video data.

[0018] Preferably, after respectively performing data enhancement on the expanded video data and the image data, inputting the data into the face segmentation model for joint training comprises:

[0019] The face segmentation model is trained in sequence according to the input video data and the image data in an interval training manner, wherein the dimension of the video data is [B, T, C, H, W], the dimension of the image data is [B, C, H, W], and B, T, C, H, and W represent batch size, sequence length, number of channels, height, and width, respectively;

[0020] Using the first objective function L v and the second objective function L i The video data and the image data are supervised during the training process.

[0021] Preferably, the step of training the face segmentation model in sequence according to the input video data and the image data in an interval training manner includes:

[0022] The number of training steps is determined, and when the number of training steps is an even number, the face segmentation model is trained according to the video data; otherwise, the face segmentation model is trained according to the image data.

[0023] Preferably, the first objective function L is used respectively v and the second objective function L i Supervising the video data and the image data during the training process includes:

[0024] L v =0.5L dice +0.5L curricular-topk +2L temporal-coherence #(2-1), L i =0.5L dice +0.5L curricular-topk #(2-2), where L dice represents the Dice loss, L curricular-TopK represents the cross entropy loss for improving difficult negative example mining based on curriculum learning, L temporal-coherence Indicates the loss of timing consistency.

[0025] To achieve the above object, the present invention further provides a video face segmentation device based on time sequence information, the device comprising:

[0026] A model building unit is used to build and train a face segmentation model; wherein the face segmentation model is provided with a TemporalModule module for capturing timing information between video frames;

[0027] The segmentation processing unit is used to input the face video data into the face segmentation model and perform video face segmentation processing based on the timing information to obtain a target face segmentation result.

[0028] In order to achieve the above objectives, the present invention also proposes a device, including a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a video face segmentation method based on timing information as described in the above embodiment.

[0029] In order to achieve the above-mentioned object, the present invention also proposes a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a video face segmentation method based on timing information as described in the above embodiment.

[0030] Beneficial effects:

[0031] The above scheme sets a Temporal Module in the face segmentation model to capture the timing information between video frames, and uses the timing information between frames to perform video face segmentation on the face video data, so as to achieve the effect of no jumps between different frames in the video segmentation, thereby improving the stability of video segmentation.

[0032] The above scheme, through the Refine Module module and objective function designed in the face segmentation model, can achieve a more refined segmentation effect and save calculation amount, thereby improving segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic flow chart of a method for video face segmentation based on time series information provided by an embodiment of the present invention.

[0035] Figure 2 A schematic diagram of the network structure of a face segmentation model provided by an embodiment of the present invention.

[0036] Figure 3 A schematic diagram of the overall process of training a face segmentation model provided by an embodiment of the present invention.

[0037] Figure 4 A schematic structural diagram of a video face segmentation device based on time series information provided by an embodiment of the present invention.

[0038] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0041] The content of the present invention is described in detail below with reference to embodiments.

[0042] With the development of deep learning, semantic segmentation tasks have also changed from traditional algorithms such as graph cuts to deep learning-based algorithms represented by FCN. Compared with traditional algorithms, deep neural networks do not require feature design, have higher robustness, and larger network capacity, and have achieved the best results on various benchmark datasets. The current optimal semantic segmentation algorithms are all designed for image segmentation. Compared with images, videos have increased temporal information. Other current video segmentation schemes still have shortcomings: (1) semantic segmentation is performed by introducing optical flow information, but this will cause the inference speed to become very slow and the accuracy is relatively low; (2) segmentation is performed by using the temporal information of the previous and next frames through the LSTM / GRU module, which reduces the redundancy in video segmentation, but the accuracy of segmentation will be reduced.

[0043] Based on this, the present invention proposes a video face segmentation method based on temporal information, which can maintain low reasoning loss while obtaining high segmentation accuracy.

[0044] Reference Figure 1 FIG. 1 is a flow chart of a method for video face segmentation based on time sequence information provided by an embodiment of the present invention.

[0045] In this embodiment, the method includes:

[0046] S11, constructing and training a face segmentation model; wherein the face segmentation model is provided with a Temporal Module module for capturing timing information between video frames.

[0047] The face segmentation model is constructed based on the Encoder-Decoder framework, and the face segmentation model includes a Segment Module and a Refine Module. The Segment Module includes a Temporal Module for capturing timing information between video frames.

[0048] The input video data and image data are roughly segmented by the Segment Module module to obtain a Coarse-Mask, wherein the image data includes a face image in each video frame in the video data;

[0049] Fine-tune the Coarse-Mask through the Refine Module module to obtain a Refine-Mask;

[0050] The Coarse-Mask and the Refine-Mask are added to obtain a predicted Mask result.

[0051] Reference Figure 2 The schematic diagram of the network structure is shown. In this embodiment, the Encoder-Decoder framework is used to build a face segmentation model, and the entire model includes two sub-modules: Segment module and Refine Module. Segmentmodule can be regarded as a baseline segmentation network, which is responsible for coarse segmentation of the input image data / video data to obtain a coarse-mask. Refine Module can be regarded as a refinement network, which is responsible for fine-tuning the coarse-mask processed by the Segment module to obtain a refine-mask. Finally, the coarse-mask and refine-mask are added at the pixel level to obtain the predicted mask.

[0052] Specifically, in the Segment module, in order to utilize the timing information, the Temporal Module is introduced at different scales of the Decoder. The Temporal Module uses ConvGRU to capture the timing information. Compared with ConvLSTM, ConvGRU has only two gates: update gate and reset gate. It has a simple structure and is easier to train. In the Temporal Module, the feature f is first obtained by the Conv-BN-Relu module to obtain f'; in order to reduce the amount of calculation, f' is obtained by splitting (split) operation to obtain f'1 and f'2; f'1 is obtained by passing ConvGRU Finally Concatenate with f'2 to get f * As the output of the module.

[0053] The training process of the face segmentation model includes:

[0054] Inputting the image data into the DECA model to generate a corresponding face video sequence to obtain expanded video data;

[0055] After data enhancement is performed on the expanded video data and the image data respectively, the data are input into the face segmentation model for joint training.

[0056] Furthermore, the image data is input into the DECA model to generate a corresponding face video sequence to obtain the expanded video data, including:

[0057] Inputting the image data into the DECA model to obtain corresponding 3D information;

[0058] The facial posture is adjusted by controlling the pitch angle, yaw angle and roll angle of the image data, and random illumination transformation is added to obtain a 3D face corresponding to continuous angle changes;

[0059] The 3D face is projected into a 2D face, and the 2D face is combined with the image data to generate a corresponding face video sequence to obtain the expanded video data.

[0060] In this embodiment, in order to expand the face video training data, the face 3D model DECA is introduced, and a single-frame face image is used to generate a corresponding face video sequence (i.e., face video). That is, by using a single face image data to generate a corresponding face video sequence, the face video training data is expanded, thereby reducing the annotation cost of the face video training data. The specific steps include:

[0061] (1) Input a single-frame face image into the DECA model to obtain the 3D information of the face image;

[0062] (2) By controlling the pitch, yaw and roll angles, adjusting the facial posture and adding random illumination transformations, a series of 3D faces corresponding to continuous angle transformations are obtained. The pitch, yaw and roll angles controlled above refer to the angles of the facial images containing 3D information generated by the DECA model; the angles can be further limited to [-15°, 15°].

[0063] (3) All the obtained 3D faces are projected back to 2D faces, and then the 2D faces are pasted into any image, where the face is the foreground and the other areas in the image are the background. The combined 2D face image frames generate the corresponding face video sequence (the 2D face image at this time can show the sequence of 2D images of the face under rotation compared with the initial face image in (1)).

[0064] In terms of data enhancement, different data enhancement methods are mainly used according to the characteristics of video data and image data: Due to the small number of video data sets, different data enhancement strategies are designed to expand the video data samples. Including: adding disturbances to the face through transmission transformation to simulate the angle change of the face; sampling the video frames at intervals to simulate the fast playback of the video and the fast change of the face; and expanding the number of samples by randomly flipping the video sequence. For the image data set, conventional data gain methods (including color jitter, random occlusion, rotation transformation, etc.) are used.

[0065] Furthermore, after respectively performing data enhancement on the expanded video data and the image data, the data are input into the face segmentation model for joint training, including:

[0066] The face segmentation model is trained in sequence according to the input video data and the image data in an interval training manner, wherein the dimension of the video data is [B, T, C, H, W], the dimension of the image data is [B, C, H, W], and B, T, C, H, and W represent batch size, sequence length, number of channels, height, and width, respectively;

[0067] Using the first objective function L v and the second objective function L i The video data and the image data are supervised during the training process.

[0068] The step of training the face segmentation model in sequence according to the input video data and the image data in an interval training manner includes:

[0069] The number of training steps is determined, and when the number of training steps is an even number, the face segmentation model is trained according to the video data; otherwise, the face segmentation model is trained according to the image data.

[0070] Wherein, the first objective function L is used respectively v and the second objective function L i Supervising the video data and the image data during the training process includes:

[0071] L v =0.5L dice +0.5L curricular-topk +2L temporal-coherence #(2-1), L i =0.5L dice +0.5L curricular-topk #(2-2), where L dice represents the Dice loss, L curricular-topk represents the cross entropy loss for improving difficult negative example mining based on curriculum learning, L temporal-coherence Indicates the loss of timing consistency.

[0072] Reference Figure 3 The overall flow diagram of model training is shown. In this embodiment, the image data set and the video data set are used to jointly train the model. First, due to the cost of data set annotation, the number of video data sets is small, while the currently public image data sets are relatively easy to obtain, and the resolution and segmentation accuracy of the image data sets are relatively high; secondly, the video data set is relatively small due to the limitations of the scene, while the image data set usually includes a rich background; the image data set can be regarded as a special video sequence containing only a single frame of image. The video sequence data and image data are input in sequence in an interval training manner to train the model (the benefit of this training method is that the model can be trained using image data and video data at the same time, and can maintain the continuous learning of the temporal information of the model corresponding to the video sequence). Optionally, the video data is trained in even times and the image data is trained in odd times. The dimensions of the video data and image data input to the model during training are [B, T, C, H, W] and [B, C, H, W], respectively, where B, T, C, H, and W represent the batch size, sequence length, number of channels, height, and width, respectively.

[0073] In addition, different objective functions are used for supervision of video data and image data, corresponding to L v and L i , video data loss L v The image data loss will be L i One more time-series correlation loss L tc, which is used to solve the problem of jumps or flickers between different frames in a video sequence. The formula is as follows:

[0074] L v =0.5L dice +0.5L curricular-topk +2L temporal-coherence #(2-1);

[0075] L i =0.5L dice +0.5L curricular-topk #(2-2);

[0076] Among them, L dice Represents Dice loss, the specific formula is as follows:

[0077] Where i represents the pixel number; N = H × W represents the number of pixels; c represents the pixel category number, and C = 2 represents the number of categories; Indicates that the i-th pixel corresponds to the true label of the c-th class; represents the predicted probability that the i-th pixel corresponds to the c-th class.

[0078] L curricular-TopK It represents the cross entropy loss of improved difficult negative example mining based on curriculum learning. The specific formula is as follows:

[0079]

[0080] Where i represents the pixel number; N = H × W represents the number of pixels; c represents the pixel category number, and C = 2 represents the number of categories; Indicates that the i-th pixel corresponds to the true label of the c-th class; It represents the predicted probability that the i-th pixel corresponds to the c-th class. t represents the threshold of the first k minimum predicted probabilities. The specific formula is as follows:

[0081]

[0082] Here, k represents an adaptive parameter. Based on the principle of curriculum learning, the value of k decreases as the number of training steps increases, and the corresponding learning objectives also go from difficult to easy, ultimately achieving better optimization results. That is,

[0083]

[0084] Where k start =80 represents the initial k value, k end =10 indicates the final k value, step indicates the current number of training steps, and total_step indicates the total number of training steps.

[0085] Ltc L represents the temporal consistency loss, which represents the mean square error loss between the difference between the predicted previous and next frames and the difference between the previous and next frames of the true label. tc It is used to supervise the consistency between two frames of a video sequence to solve the problems of video jitter and flicker. The formula is as follows:

[0086]

[0087] Among them, always S t Represents the prediction result at timestamp t; The true label at timestamp t, where t represents the video frame number of the same video sequence.

[0088] S12, inputting the face video data into the face segmentation model and performing video face segmentation processing based on the timing information to obtain a target face segmentation result.

[0089] In summary, by proposing a temporal enhancement module to capture video temporal information and proposing a new objective function based on the characteristics of video segmentation, the accuracy and stability of video face segmentation can be greatly improved. In order to utilize temporal information, in terms of network structure design, this scheme uses the Coarse-Refine network structure from coarse to fine, and proposes the Temporal Module temporal enhancement module to capture the temporal information between video frames. This scheme can optimize the face segmentation model using both video data and image data, and design different objective functions for image data and video data respectively. According to the principle of course learning, an improved cross entropy loss function is proposed to improve the accuracy of segmentation; in the flicker problem of video segmentation, consistency loss is introduced to improve the stability of video segmentation. In practical applications, the network structure and objective function based on temporal information proposed in this scheme, and the video data and image data are used for joint training at the same time, can achieve good results in the accuracy and stability of face video segmentation.

[0090] Reference Figure 4 FIG. 4 is a schematic diagram of the structure of a video face segmentation device based on time sequence information provided by an embodiment of the present invention.

[0091] In this embodiment, the device 40 includes:

[0092] A model building unit 41 is used to build and train a face segmentation model; wherein the face segmentation model is provided with a Temporal Module module for capturing timing information between video frames;

[0093] The segmentation processing unit 42 is used to input the face video data into the face segmentation model and perform video face segmentation processing based on the timing information to obtain a target face segmentation result.

[0094] Furthermore, in the model building unit 41, the face segmentation model is built based on the Encoder-Decoder framework, and the face segmentation model includes a Segment Module module and a Refine Module module, and the Segment Module module includes a Temporal Module module for capturing timing information between video frames; wherein,

[0095] The input video data and image data are roughly segmented by the Segment Module module to obtain a Coarse-Mask, wherein the image data includes a face image in each video frame in the video data;

[0096] Fine-tune the Coarse-Mask through the Refine Module module to obtain a Refine-Mask;

[0097] The Coarse-Mask and the Refine-Mask are added to obtain a predicted Mask result.

[0098] Furthermore, in the model building unit 41, the training process of the face segmentation model includes:

[0099] A data expansion unit, used for inputting the image data into a DECA model, generating a corresponding face video sequence, and obtaining expanded video data;

[0100] The joint training unit is used to perform data enhancement on the expanded video data and the image data respectively, and then input them into the face segmentation model for joint training.

[0101] Furthermore, the data expansion unit further includes:

[0102] An input unit, used for inputting the image data into the DECA model to obtain corresponding 3D information;

[0103] An adjustment unit, configured to adjust the facial posture by controlling the pitch angle, yaw angle and roll angle of the image data, and adding random illumination transformation to obtain a 3D face corresponding to continuous angle changes;

[0104] The combining unit is used to project the 3D face into a 2D face, and combine the 2D face with the image data to generate a corresponding face video sequence to obtain the expanded video data.

[0105] Furthermore, the joint training unit further includes:

[0106] An interval training unit, used for training the face segmentation model according to the input video data and the image data in sequence by using an interval training method, wherein the dimension of the video data is [B, T, C, H, W], the dimension of the image data is [B, C, H, W], and B, T, C, H, and W represent batch size, sequence length, number of channels, height, and width, respectively;

[0107] The training supervision unit is used to respectively use the first objective function L v and the second objective function L i The video data and the image data are supervised during the training process.

[0108] Furthermore, the interval training unit comprises:

[0109] The number of training steps is determined, and when the number of training steps is an even number, the face segmentation model is trained according to the video data; otherwise, the face segmentation model is trained according to the image data.

[0110] Furthermore, the training supervision unit comprises:

[0111] L v =0.5L dice +0.5L curricular-topk +2L temporal-coherence #(2-1), L i =0.5L dice +0.5L curricular-topk #(2-2), where L dice represents the Dice loss, L curricular-TopK represents the cross entropy loss for improving difficult negative example mining based on curriculum learning, L temporal-coherence Indicates the loss of timing consistency.

[0112] Each unit module of the device 40 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0113] The embodiment of the present invention further provides a device, which includes the video face segmentation device based on time sequence information as described above, wherein the video face segmentation device based on time sequence information can be used Figure 4 The structure of the embodiment can be executed accordingly. Figure 1The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0114] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0115] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0116] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0117] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0118] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The video face segmentation method based on time sequence information is shown. The computer readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0119] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0120] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0121] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A video face segmentation method based on temporal information, characterized in that: The method comprises: Constructing and training a face segmentation model; wherein the face segmentation model is provided with a Temporal Module module for capturing timing information between video frames; Inputting the face video data into the face segmentation model and performing video face segmentation processing based on the timing information to obtain a target face segmentation result; The face segmentation model is constructed based on the Encoder-Decoder framework. The face segmentation model includes a Segment Module and a Refine Module. The Segment Module includes a Temporal Module for capturing timing information between video frames. The input video data and image data are roughly segmented by the Segment Module module to obtain a Coarse-Mask, wherein the image data includes a face image in each video frame in the video data; Fine-tune the Coarse-Mask through the Refine Module module to obtain a Refine-Mask; Add the Coarse-Mask and the Refine-Mask to obtain a predicted Mask result; The training process of the face segmentation model includes: Inputting the image data into the DECA model to generate a corresponding face video sequence to obtain expanded video data; After data enhancement is performed on the expanded video data and the image data, the data are input into the face segmentation model for joint training; wherein, Inputting the image data into the DECA model to generate a corresponding face video sequence to obtain expanded video data; further comprising: Inputting the image data into the DECA model to obtain corresponding 3D information; The facial posture is adjusted by controlling the pitch angle, yaw angle and roll angle of the image data, and random illumination transformation is added to obtain a 3D face corresponding to continuous angle changes; Projecting the 3D face into a 2D face, and combining the 2D face with the image data to generate a corresponding face video sequence, thereby obtaining the expanded video data; After data enhancement is performed on the expanded video data and the image data respectively, the data are input into the face segmentation model for joint training; further comprising: The face segmentation model is trained in sequence according to the input video data and the image data in an interval training manner, wherein the dimension of the video data is [B, T, C, H, W], the dimension of the image data is [B, C, H, W], and B, T, C, H, and W represent batch size, sequence length, number of channels, height, and width, respectively; Using the first objective function L v and the second objective function L i The video data and the image data are supervised during the training process.

2. The video face segmentation method based on time series information according to claim 1, characterized in that: The step of training the face segmentation model in sequence according to the input video data and the image data in an interval training manner includes: The number of training steps is determined, and when the number of training steps is an even number, the face segmentation model is trained according to the video data; otherwise, the face segmentation model is trained according to the image data.

3. The video face segmentation method based on time sequence information according to claim 1, characterized in that: The first objective function L is used respectively v and the second objective function L i Supervising the video data and the image data during the training process includes: L v =0.5L dice +0.5L curricular-topk +2L temporal-coherence , L i =0.5L dice +0.5L curricular-topk ,That In, L dice represents the Dice loss, L curricular-topk represents the cross entropy loss for improving difficult negative example mining based on curriculum learning, L temporal-coherence Indicates the loss of timing consistency.

4. A video face segmentation device based on time sequence information, characterized in that: The device comprises: A model building unit is used to build and train a face segmentation model; wherein the face segmentation model is provided with a Temporal Module module for capturing timing information between video frames; A segmentation processing unit, used for inputting the face video data into the face segmentation model and performing video face segmentation processing based on the timing information to obtain a target face segmentation result; In the model building unit, the face segmentation model is built based on the Encoder-Decoder framework, and the face segmentation model includes a Segment Module module and a Refine Module module, and the Segment Module module includes a Temporal Module module for capturing timing information between video frames; wherein, The input video data and image data are roughly segmented by the Segment Module module to obtain a Coarse-Mask, wherein the image data includes a face image in each video frame in the video data; Fine-tune the Coarse-Mask through the Refine Module module to obtain a Refine-Mask; Add the Coarse-Mask and the Refine-Mask to obtain a predicted Mask result; The model building unit comprises: A data expansion unit, used for inputting the image data into a DECA model, generating a corresponding face video sequence, and obtaining expanded video data; A joint training unit, used for performing data enhancement on the expanded video data and the image data respectively, and then inputting the data into the face segmentation model for joint training; Furthermore, the data expansion unit further includes: An input unit, used for inputting the image data into the DECA model to obtain corresponding 3D information; An adjustment unit, configured to adjust the facial posture by controlling the pitch angle, yaw angle and roll angle of the image data, and adding random illumination transformation to obtain a 3D face corresponding to continuous angle changes; A combining unit, used for projecting the 3D face into a 2D face, and combining the 2D face with the image data to generate a corresponding face video sequence, thereby obtaining the expanded video data; Furthermore, the joint training unit further includes: An interval training unit, used for training the face segmentation model according to the input video data and the image data in sequence by using an interval training method, wherein the dimension of the video data is [B, T, C, H, W], the dimension of the image data is [B, C, H, W], and B, T, C, H, and W represent batch size, sequence length, number of channels, height, and width, respectively; The training supervision unit is used to respectively use the first objective function L v and the second objective function L i The video data and the image data are supervised during the training process.

5. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a video face segmentation method based on time sequence information as claimed in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of a video face segmentation method based on time sequence information as described in any one of claims 1 to 3.

Images