A method and device for detecting and generating lip shape of a person in a video
By using preset timing data network model and continuous frame subtraction filtering processing in video character lip shape detection, the lip jitter problem caused by inter-frame gap is solved, the accuracy and efficiency of lip shape detection are improved, and it is suitable for speech recognition and Multimedia system.
Patent Information
- Application Number
- CN202011474640.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-12-14
AI Technical Summary
In video-based lip shape detection, inter-frame gaps lead to jitter and unnatural phenomena in lip-form video generation. Traditional methods will affect the feature extraction effect when improving efficiency.
The lip detection generation method based on the preset timing data network model is adopted, combined with continuous frame subtraction method and mean filtering processing, adjust the lip angle and plane rotation, determine whether to perform filtering according to the pixel point threshold, and extract feature points in combination with deep learning technology .
It reduces the jitter phenomenon in lip-form video, improves the accuracy and efficiency of lip-form detection, and is suitable for speech recognition and multimedia systems.
Smart Images

Figure CN112651310B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of computer vision feature detection, and in particular to a method and device for detecting and generating lip shape of a person in a video. Background Art
[0002] Lip detection technology refers to the process of extracting lip feature information and confirming the detection of lip feature point trajectories from images or videos. Lip detection has high application value in many fields such as speech recognition, identity recognition, and multimedia systems. Video-based lip detection is different from image-based lip detection. In addition to extracting lip features, it is also necessary to regenerate the lip edges into the video. This process will affect the overall lip generation effect due to the gap between different frames in the video, such as jitter and unnatural phenomena. Therefore, how to reduce the mutual influence between frames is a very important part of the lip generation of foreground characters based on video.
[0003] The core problem of lip edge detection for foreground characters in videos is how to reduce the mutual influence between adjacent frames while effectively extracting lip key points, lip edges and lip motion trajectories, thereby reducing the jitter problem in the video that generates the lip edges.
[0004] The traditional way to extract and regenerate the lip edge is to extract features from each frame of the video to obtain feature points, and then reshape the lip edge based on the feature points. The problem with this approach is that when the corresponding difference between the pixels between frames is too large, the generated lip trajectory video will show obvious jitter, especially when the foreground person is constantly speaking. If each frame is smoothed, it will directly affect the efficiency of feature extraction. Summary of the invention
[0005] The purpose of this application is to overcome the above problems or at least partially solve or alleviate the above problems.
[0006] According to one aspect of the present application, a method for detecting and generating lip shape of a person in a video is provided, comprising:
[0007] Extract video frame images from input video;
[0008] Extracting the facial region and background of the video frame image based on the global features and local features of the video frame image;
[0009] Using a preset time series data network model to perform edge extraction on the facial area and background of the video frame image, and generate a lip shape generation image corresponding to the video frame image;
[0010] For the lip shape generated image, a continuous frame subtraction method is used to calculate the difference between two adjacent video frame images, and filtering is performed on the lip shape generated image according to the difference;
[0011] The filtered lip shape generation image is subjected to video generation processing.
[0012] Preferably, performing video generation processing on the filtered lip shape generation image comprises:
[0013] Adjusting the lip angle of the filtered lip shape generated image so that the rotation angle in the lip shape plane is zero;
[0014] The audio data extracted from the input video is synthesized with the lip shape generated image after the balance to obtain a lip shape video.
[0015] Preferably, for the lip shape generated image, using a continuous frame subtraction method to calculate the difference between two adjacent video frame images, and filtering the lip shape generated image according to the difference includes:
[0016] The difference between two adjacent video frame images is calculated using a continuous frame subtraction method. When the difference is greater than or equal to a pixel threshold, the lip shape generated image is processed by a mean filter; when the difference is less than the pixel threshold, no filtering is performed.
[0017] Preferably, the mean filtering process includes:
[0018] The N×N convolution kernel is used to average the difference between two adjacent video frames.
[0019] Preferably, obtaining a preset time series data network model includes:
[0020] Using deep learning technology, a time series data network model based on feature points is trained.
[0021] In a second aspect, the present invention further provides a device for detecting and generating lip shape of a person in a video, comprising:
[0022] A video image module, configured to extract video frame images from an input video;
[0023] A face extraction module, configured to extract a face region and a background of the video frame image based on global features and local features of the video frame image;
[0024] A lip shape generation module is configured to use a preset temporal data network model to perform edge extraction on the facial area and background of the video frame image to generate a lip shape generation image corresponding to the video frame image;
[0025] A filtering module is configured to calculate the difference between two adjacent video frame images using a continuous frame subtraction method for the lip shape generated image, and filter the lip shape generated image according to the difference;
[0026] The synthesis module is configured to perform video generation processing on the filtered lip shape generation image.
[0027] Preferably, the synthesis module performs video generation processing on the filtered lip shape generated image, including:
[0028] Adjusting the lip angle of the filtered lip shape generated image so that the rotation angle in the lip shape plane is zero;
[0029] The audio data extracted from the input video is synthesized with the lip shape generated image after the balance to obtain a lip shape video.
[0030] Preferably, the filtering module calculates the difference between two adjacent video frame images using a continuous frame subtraction method for the lip shape generated image, and filtering the lip shape generated image according to the difference comprises:
[0031] The difference between two adjacent video frame images is calculated using a continuous frame subtraction method. When the difference is greater than or equal to a pixel threshold, the lip shape generated image is processed by a mean filter; when the difference is less than the pixel threshold, no filtering is performed.
[0032] Preferably, the filtering module adopts mean filtering processing including:
[0033] The N×N convolution kernel is used to average the difference between two adjacent video frames.
[0034] Preferably, the lip shape generation module acquires a preset time series data network model including:
[0035] Using deep learning technology, a time series data network model based on feature points is trained.
[0036] The method and device for video character lip shape detection and generation of the present application combine mean filter smoothing processing after threshold processing with a method for extracting lip shape feature points based on deep learning. The lip shape detection and generation of foreground characters in the video can be widely used in speech recognition, multimedia systems, character recognition and other fields.
[0037] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:
[0039] Figure 1 is a schematic flow chart of a method for detecting and generating lip shape of a person in a video according to an embodiment of the present application;
[0040] Figure 2 is a structural diagram of a time series data network model according to an embodiment of the present application;
[0041] Figure 3 It is a structural schematic diagram of a device for detecting and generating lip shape of a person in a video according to an embodiment of the present application;
[0042] Figure 4 is a schematic diagram of a video frame image extracted from a video according to an embodiment of the present application;
[0043] Figure 5 is a schematic diagram of a smoothed lip edge according to an embodiment of the present application;
[0044] Figure 6 is a schematic diagram of the inter-frame difference of the lip edge smoothed according to an embodiment of the present application; wherein, Figure 6 (a) Figure 5 Schematic diagram of the inter-frame difference of the lip edge in the first two images; Figure 6 (b) Figure 5 Schematic diagram of the inter-frame difference of the lip edge in the last two images;
[0045] Figure 7 This is a flowchart of video character lip shape detection generation according to an embodiment of the present application;
[0046] Figure 8 is a schematic structural block diagram of a first computer-readable storage medium according to an embodiment of the present application;
[0047] Fig. 9 It is a schematic structural block diagram of a second computer-readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION
[0048] Figure 1 The flowchart of the method for detecting and generating lip shape of a person in a video according to an embodiment of the present application may include the following steps:
[0049] S101, extracting video frame images from an input video;
[0050] S102, extracting the facial area and background of the video frame image based on the global features and local features of the video frame image;
[0051] S103, using a preset time series data network model to perform edge extraction on the facial area and background of the video frame image, and generate a lip shape generated image corresponding to the video frame image;
[0052] S104, for the lip shape generated image, using a continuous frame subtraction method to calculate the difference between two adjacent video frame images, and performing filtering processing on the lip shape generated image according to the difference;
[0053] S105, performing video generation processing on the filtered lip shape generation image.
[0054] In an embodiment of the present invention, deep learning technology is used to train a time series data network model based on feature points. The input of the model is each frame image in the video, and the output is the lip shape generation image corresponding to the image. Secondly, for the lip shape generation images, the continuous frame subtraction method is used to calculate the difference between each other. When the difference is greater than the threshold, mean filtering is used, and then the convolution kernel is used to output the final difference. If the difference is less than the threshold, no filtering is performed. Finally, the image after filtering and smoothing is used to generate a video. Compared with the traditional video frame direct generation method, the method for video character lip shape detection and generation in the embodiment of the present invention has improved accuracy and efficiency.
[0055] In the embodiment of the present invention, the feature extracted in step S103 is the key point of the lip shape, and the lip shape feature can be quantitatively evaluated by performing edge extraction on the facial area and background of the video frame image using a preset temporal data network model.
[0056] In the embodiment of the present invention, in step S105, performing video generation processing on the filtered lip shape generation image includes:
[0057] Adjusting the lip angle of the filtered lip shape generated image so that the rotation angle in the lip shape plane is zero;
[0058] The audio data extracted from the input video is synthesized with the lip shape generated image after the balance to obtain a lip shape video.
[0059] In the embodiment of the present invention, the video for generating lip shape can be used for generating the full face of the virtual anchor. Accurate lip shape generation plays a vital role in generating the full face of the anchor.
[0060] In the embodiment of the present invention, in step S104, for the lip shape generated image, using a continuous frame subtraction method to calculate the difference between two adjacent video frame images, and filtering the lip shape generated image according to the difference includes:
[0061] The difference between two adjacent video frame images is calculated using a continuous frame subtraction method. When the difference is greater than or equal to a pixel threshold, the lip shape generated image is processed by a mean filter; when the difference is less than the pixel threshold, no filtering is performed.
[0062] In the embodiment of the present invention, when the pixel size of the extracted lip shape image is within a predetermined range, the pixel threshold may be 80,000 pixels.
[0063] In the embodiment of the present invention, the mean value filtering process includes:
[0064] The N×N convolution kernel is used to average the difference between two adjacent video frames.
[0065] In the embodiment of the present invention, TCN can fully record the time series data information, that is, the TCN time series time model can accurately record the movement trajectory of the lip edge, but the boundary information has a relatively large impact on TCN. If the lip movement amplitude is large, severe lip shaking may occur. Therefore, in the embodiment of the present invention, the frame image in the video is processed by the smoothing process of the mean filter, and each frame output of the time series data network model feature learning is smoothed, which improves the problem of severe lip shaking. The experiment verifies that the scheme improves the stability and smoothness of the lip fluctuation.
[0066] In the embodiment of the present invention, step S103, obtaining a preset time series data network model includes:
[0067] Using deep learning technology, a time series data network model based on feature points is trained.
[0068] In the embodiment of the present invention, a temporal convolutional network model (TCN) is used, such as Figure 2 As shown in the figure, there is a causal relationship between the Conv layers of TCN, that is, there will be no loss of historical information or changes in future frames. Therefore, applying the TCN temporal network model to the generation of lip edges in foreground character videos can improve accuracy and production efficiency.
[0069] like Figure 3 As shown, the embodiment of the present invention also provides a device for detecting and generating lip shape of a person in a video, comprising:
[0070] The video image module 100 is configured to extract video frame images from an input video;
[0071] A face extraction module 200 is configured to extract a face region and a background of the video frame image based on global features and local features of the video frame image;
[0072] The lip shape generation module 300 is configured to perform edge extraction on the facial area and background of the video frame image using a preset temporal data network model to generate a lip shape generation image corresponding to the video frame image;
[0073] The filtering module 400 is configured to calculate the difference between two adjacent video frames using a continuous frame subtraction method for the lip shape generated image, and perform filtering processing on the lip shape generated image according to the difference;
[0074] The synthesis module 500 is configured to perform video generation processing on the filtered lip shape generation image.
[0075] In the embodiment of the present invention, the synthesis module 500 performs video generation processing on the filtered lip shape generated image, including:
[0076] Adjusting the lip angle of the filtered lip shape generated image so that the rotation angle in the lip shape plane is zero;
[0077] The audio data extracted from the input video is synthesized with the lip shape generated image after the balance to obtain a lip shape video.
[0078] In the embodiment of the present invention, the filtering module 400 calculates the difference between two adjacent video frame images using a continuous frame subtraction method for the lip shape generated image, and performs filtering processing on the lip shape generated image according to the difference, including:
[0079] The difference between two adjacent video frame images is calculated using a continuous frame subtraction method. When the difference is greater than or equal to a pixel threshold, the lip shape generated image is processed by a mean filter; when the difference is less than the pixel threshold, no filtering is performed.
[0080] In the embodiment of the present invention, the filtering module 400 adopts mean filtering processing including:
[0081] The N×N convolution kernel is used to average the difference between two adjacent video frames.
[0082] In the embodiment of the present invention, the lip shape generation module 300 obtains the preset time series data network model including:
[0083] Using deep learning technology, a time series data network model based on feature points is trained.
[0084] like Figures 4 to 7 As shown, the method of the embodiment of the present invention performs lip shape detection and generation based on deep learning, realizes the lip shape extraction of the foreground character in the video and generates a new lip shape video. The specific process is as follows:
[0085] 1. Extract the input video into two formats: audio and image, with the audio retaining its original format.
[0086] 2. Based on global features and local features, each frame of the video is processed to extract the facial area and background.
[0087] 3. Based on the time series data network model for lip feature points obtained through training, the edge of the lip is restored based on the feature points to generate a lip edge image.
[0088] 4. The lip edge image generated in step 3 is processed for video synthesis. Here, time smoothing can be performed. The error between the previous and next frames is calculated using the continuous frame subtraction method. The threshold of the pixel difference is set to 80,000. If the pixel error between frames is less than 80,000, video synthesis is performed directly; otherwise, mean filtering is performed to reduce the error between frames before video synthesis.
[0089] 5. Using video synthesis technology, the audio data of step 1 is synthesized with the generated lip edge video to obtain the final result, that is, the generated lip video.
[0090] The present application also provides a computing device, referring to Figure 8 The computing device includes a memory 1120, a processor 1110, and a computer program stored in the memory 1120 and executable by the processor 1110. The computer program is stored in a space 1130 for program code in the memory 1120. When the computer program is executed by the processor 1110, it is implemented to execute any one of the method steps 1131 according to the present invention.
[0091] The present application also provides a computer-readable storage medium. Fig. 9 The computer-readable storage medium comprises a storage unit for program code, the storage unit being provided with a program 1131 ′ for executing the method steps according to the present invention, the program being executed by a processor.
[0092] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is run on a computer, the computer is caused to execute the method steps according to the present invention.
[0093] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer loads and executes the computer program instructions, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0094] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0095] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof.
[0096] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for detecting and generating lip shape of a person in a video. include: Extract video frame images from input video; Extracting the facial region and background of the video frame image based on the global features and local features of the video frame image; Using a preset time series data network model to perform edge extraction on the facial area and background of the video frame image, and generate a lip shape generation image corresponding to the video frame image; For the lip shape generated image, a continuous frame subtraction method is used to calculate the difference between two adjacent video frame images, and filtering is performed on the lip shape generated image according to the difference; Performing video generation processing on the filtered lip shape generation image; For the lip shape generated image, using a continuous frame subtraction method to calculate the difference between two adjacent video frame images, and filtering the lip shape generated image according to the difference includes: The difference between two adjacent video frame images is calculated by using a continuous frame subtraction method, and when the difference is greater than or equal to a pixel threshold, the lip shape generated image is processed by a mean filter; When the difference is smaller than the pixel threshold, no filtering is performed.
2. The method according to claim 1, It is characterized in that The video generation process of the filtered lip shape generated image includes: Adjusting the lip angle of the filtered lip shape generated image so that the rotation angle in the lip shape plane is zero; The audio data extracted from the input video is synthesized with the lip shape generated image after the balance to obtain a lip shape video.
3. The method according to claim 1, It is characterized in that The mean filter processing includes: The N×N convolution kernel is used to average the difference between two adjacent video frames.
4. The method according to claim 1, It is characterized in that Obtaining the preset time series data network model includes: Using deep learning technology, a time series data network model based on feature points is trained.
5. A device for detecting and generating lip shape of a person in a video, include: A video image module, configured to extract video frame images from an input video; A face extraction module, configured to extract a face region and a background of the video frame image based on global features and local features of the video frame image; A lip shape generation module is configured to use a preset temporal data network model to perform edge extraction on the facial area and background of the video frame image to generate a lip shape generation image corresponding to the video frame image; A filtering module is configured to calculate the difference between two adjacent video frame images using a continuous frame subtraction method for the lip shape generated image, and filter the lip shape generated image according to the difference; A synthesis module, configured to perform video generation processing on the filtered lip shape generated image; The filtering module calculates the difference between two adjacent video frame images using a continuous frame subtraction method for the lip shape generated image, and performs filtering processing on the lip shape generated image according to the difference, including: The difference between two adjacent video frame images is calculated by using a continuous frame subtraction method, and when the difference is greater than or equal to a pixel threshold, the lip shape generated image is processed by a mean filter; When the difference is smaller than the pixel threshold, no filtering is performed.
6. The device according to claim 5, It is characterized in that The synthesis module performs video generation processing on the filtered lip shape generated image, including: Adjusting the lip angle of the filtered lip shape generated image so that the rotation angle in the lip shape plane is zero; The audio data extracted from the input video is synthesized with the lip shape generated image after the balance to obtain a lip shape video.
7. The device according to claim 5, It is characterized in that The filtering module adopts mean filtering processing including: The N×N convolution kernel is used to average the difference between two adjacent video frames.
8. The device according to claim 5, It is characterized in that The lip shape generation module obtains a preset time series data network model including: Using deep learning technology, a time series data network model based on feature points is trained.
