A video portrait lip-reading analysis method based on classic lip reading

Abstract

This invention discloses a video portrait lip-reading analysis method based on classic lip-reading. The main steps are as follows: Both the sample video and the evidence video are segmented into frames; frames from the sample video where a specific statement is uttered, and frames from the evidence video where the statement to be examined is uttered, are extracted; key feature points are extracted from the faces in each frame, obtaining 68 feature points; the distance between the 62nd and 66th feature points in each frame is recorded, forming two feature vectors; the two feature vectors are normalized to their maximum and minimum values; if the two feature vectors have the same length, the cosine similarity between them is calculated, and whether the statement in the evidence video is consistent with the sample video is determined based on whether the cosine similarity is higher than a standard threshold; if the two feature vectors have different lengths, the minimum cumulative distance between them after dynamic time warping is calculated, and whether the statement in the evidence video is consistent with the sample video is determined based on whether the minimum cumulative distance is less than a standard threshold.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and investigation technology, and specifically to a video portrait lip-reading analysis method based on classic lip-reading. Background Technology

[0002] The absence of audio in videos is quite common, and videos lacking this crucial information cannot be directly used in scenarios such as law enforcement supervision, legal proceedings, and news reporting. Even worse, videos lacking audio may be misinterpreted and maliciously disseminated, leading to serious public opinion risks and negative social impacts. Therefore, improving the accuracy of lip-reading analysis methods can effectively expand evidence acquisition channels, maintain social harmony and stability, and has significant research value and urgent practical needs.

[0003] Currently, lip-reading analysis methods mainly fall into two categories. The first involves extracting traditional visual features from mouth images and analyzing them using feature comparison or machine learning. The second utilizes deep learning technology, employing deep networks for end-to-end learning and analysis of mouth images. The former has limited feature extraction capabilities and often requires high-quality video recordings, thus limiting its practical application. While the latter can improve feature extraction capabilities through large-scale sample training to handle lip-reading analysis under various conditions, obtaining sufficient large-scale sample data in practice is often difficult, and the lack of training data significantly limits the lip-reading analysis capabilities of deep network models.

[0004] Due to the aforementioned limitations, existing lip-reading methods are difficult to use effectively in practice, especially in specific application scenarios such as law enforcement process supervision and legal litigation evidence, where the success rate of existing methods remains very limited. This indicates that lip-reading technology still faces many challenges in practical applications. Summary of the Invention

[0005] To address the shortcomings of existing lip-reading techniques, this invention provides a video portrait lip-reading analysis method based on classic lip-reading. This method, tailored to specific application needs, effectively determines whether the statements made by a person in the forensic video match those in the sample video by extracting and matching features from the forensic video with known spoken statements. This provides strong evidentiary support for applications such as process review and litigation.

[0006] The objective of this invention can be achieved by adopting the following technical solutions:

[0007] A video portrait lip-reading analysis method based on classic lip reading, characterized by the following steps:

[0008] S1. Both the evidence video and the sample video are segmented into frames. Frames from the sample video where specific statements are uttered, and frames from the evidence video where statements to be examined are uttered, are extracted. The frames extracted from the sample video are denoted as follows: Where N is the total number of frames extracted from the sample video, and each frame extracted from the evidence video is denoted as... Where M is the total number of frames extracted from the video of the sample;

[0009] S2. Using the facial landmark detection method, 68 key feature points of the face in each frame are detected and located.

[0010] S3. Record the distance between feature points 62 and 66 in each frame, forming two feature vectors, where the feature vector corresponding to the sample is A = (a1, a2, ..., a...). N The feature vector corresponding to the sample is B = (b1, b2, ..., b M );

[0011] S4. Perform maximum / minimum value normalization on eigenvectors A and B respectively. For eigenvector A, the i-th element after normalization... for:

[0012]

[0013] Where a i Let a represent the i-th element of the original feature vector A. min a represents the minimum value among the original elements of eigenvector A. max The maximum value among the original elements of feature vector A is used to obtain the normalized feature vector A of the sample video. * And the normalized feature vector B of the sample video was obtained in the same way. * ;

[0014] S5. If the feature vectors of the sample and the evidence have the same dimension, i.e., N = M, then calculate the cosine similarity S between the two normalized feature vectors:

[0015]

[0016] If S is higher than the standard threshold, the statements in the evidence video are determined to be consistent with the sample video; otherwise, they are determined to be inconsistent.

[0017] S6. If the feature vectors of the sample and the evidence have different dimensions, i.e. N≠M, calculate the minimum cumulative distance ∑D between the two after dynamic time warping. If ∑D is less than the standard threshold, it is determined that the statements in the evidence video are consistent with the sample video; otherwise, it is determined that the two are inconsistent.

[0018] As a preferred technical solution, in step S2, the Convolutional Experts Constrained Local Model (CE-CLM) is used to detect and locate 68 key feature points of the face.

[0019] As a preferred technical solution, in step S5, the standard threshold for cosine similarity is 0.9469.

[0020] As a preferred technical solution, in step S6, the standard threshold for the minimum cumulative distance after dynamic time warping is 1.9085.

[0021] The present invention has the following advantages and effects compared with the prior art:

[0022] 1. This invention provides a video portrait lip reading analysis method based on classic lip reading, which solves the problems of low video quality and insufficient training samples in practical applications by performing paired analysis of the evidence and sample videos.

[0023] 2. This invention can effectively handle situations where the length of the evidence and the sample video are the same or different, and has a wide range of applications.

[0024] 3. This invention is simple to calculate, has low computational complexity, requires fewer resources, and effectively improves the application efficiency of the method. Attached Figure Description

[0025] Figure 1 This is an overall flowchart of the video portrait lip reading analysis method based on classic lip reading disclosed in this invention;

[0026] Figure 2 This is a schematic diagram of 68 key facial feature points extracted in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0028] This embodiment discloses a video portrait lip-reading analysis method based on classic lip-reading. The implementation process of this invention is described in detail below using self-shot sample videos (i.e., videos to be tested) A, B, C, and D, and corresponding sample videos E, F, G, and H as examples. Specifically, A and E have the same number of frames and the same utterance; B and F have the same number of frames but different utterances; C and G have different number of frames but the same utterance; and D and H have different number of frames but different utterances.

[0029] The overall implementation process is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0030] Each of the eight video segments is divided into frames, and the frames of video A are sequentially denoted as follows: Let each frame of video B be sequentially denoted as... And so on.

[0031] Using a facial landmark detection method, 68 key feature points of the face in each frame are detected and located.

[0032] Record the distance between feature points 62 and 66 in each frame to form the feature vector for each video segment. The feature vector for video A is A = (a1, a2, ..., a...). N The feature vector corresponding to video B is B = (b1, b2, ..., bb2). M ), and so on.

[0033] Perform maximum / minimum value normalization on each eigenvector. Taking eigenvector A as an example, the i-th element after normalization... for:

[0034]

[0035] Where a i Let a represent the i-th element of the original feature vector A. min a represents the minimum value among the original elements of eigenvector A. max The maximum value among the original elements of feature vector A is used to obtain the normalized feature vector A of the sample video. * And the normalized feature vector B of other videos is obtained in the same way. * C * 、D * 、E * F * , G * H * .

[0036] For A with the same feature vector dimension * and E * and B * and F *Calculate the cosine similarity separately to obtain A. * and E * The cosine similarity between them is 0.9732, which is greater than the standard threshold of 0.9469, therefore the statements made by the two are considered to be identical; B * and F * The cosine similarity between the two statements is 0.8341, which is less than the standard threshold of 0.9469, indicating that the statements made by the two statements are inconsistent.

[0037] For C with different feature vector dimensions * and G * , and D * and H * Calculate the minimum cumulative distance between the two after dynamic time warping, and obtain C. * and G * The minimum cumulative distance between them is 1.2013, which is less than the standard threshold of 1.9085, therefore the statements made by both are considered to be consistent; D * and H * The minimum cumulative distance between them is 2.5236, which is greater than the standard threshold of 1.9085, so it is determined that the statements made by the two are inconsistent.

[0038] The experimental results above demonstrate that the method of the present invention has high accuracy in determining whether the statements made by the sample and the evidence are consistent, thus proving the effectiveness of the method of the present invention.

[0039] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A video portrait lip-reading analysis method based on classic lip-reading, characterized in that, Includes the following steps: S1. Both the evidence video and the sample video are segmented into frames. Frames from the sample video where specific statements are uttered, and frames from the evidence video where statements to be examined are uttered, are extracted. The frames extracted from the sample video are denoted as follows: Where N is the total number of frames extracted from the sample video, and each frame extracted from the evidence video is denoted as... Where M is the total number of frames extracted from the video of the sample; S2. Using the facial landmark detection method, 68 key feature points of the face in each frame are detected and located. S3. Record the distance between feature points 62 and 66 in each frame, forming two feature vectors, where the feature vector corresponding to the sample is A = (a1, a2, ..., a...). N The feature vector corresponding to the sample is B = (b1, b2, ..., b M ); S4. Perform maximum / minimum value normalization on eigenvectors A and B respectively. For eigenvector A, the i-th element after normalization... for: Where a i Let a represent the i-th element of the original feature vector A. min a represents the minimum value among the original elements of eigenvector A. max The maximum value among the original elements of feature vector A is used to obtain the normalized feature vector A of the sample video. * And the normalized feature vector B of the sample video was obtained in the same way. * ; S5. If the feature vectors of the sample and the evidence have the same dimension, i.e., N = M, then calculate the cosine similarity S between the two normalized feature vectors: If S is higher than the standard threshold, the statements in the evidence video are determined to be consistent with the sample video; otherwise, they are determined to be inconsistent. S6. If the feature vectors of the sample and the evidence have different dimensions, i.e. N≠M, calculate the minimum cumulative distance ∑D between the two after dynamic time warping. If ∑D is less than the standard threshold, it is determined that the statements in the evidence video are consistent with the sample video; otherwise, it is determined that the two are inconsistent.

2. The video portrait lip-reading analysis method based on classic lip-reading as described in claim 1, characterized in that, In step S2, the Convolutional Experts Constrained Local Model (CE-CLM) is used to detect and locate 68 key feature points of the face.

3. The video portrait lip-reading analysis method based on classic lip-reading according to claim 1, characterized in that, In step S5, the standard threshold for cosine similarity is 0.9469.

4. The video portrait lip-reading analysis method based on classic lip-reading as described in claim 1, characterized in that, In step S6, the standard threshold for the minimum cumulative distance after dynamic time warping is 1.9085.

Images