Sound and lip synchronous identification method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the technical field of audio and video image processing, and relates to an audio and lip synchronous recognition method and device, computer equipment and a storage medium, and the method comprises the steps: firstly, extracting the visual features of a video through a visual 3D convolutional network, and extracting the audio features through an audio 2D convolutional network; the lip shape change and the voice information are respectively captured by the two characteristics; then, a gated recurrent neural network is ingeniously applied to deeply fuse visual and audio features to generate global features; and finally, based on the fused features, identification is carried out through a sound and lip synchronous classifier. According to the method, the time sequence relation between the audio and the lip shape can be effectively captured, the accuracy of audio and lip synchronous recognition is greatly improved, and reliable technical support is provided for video quality evaluation and anomaly detection.

Description

Technical Field

[0001] The present application relates to the technical field of audio and video image processing, and in particular to a lip synchronization recognition method, device, computer equipment and storage medium. Background Art

[0002] Judging whether the voice and mouth are synchronized is widely used in reality. For example, in an interview scenario, by checking whether the interviewer's voice and lips are synchronized, it is possible to prevent non-compliant phenomena such as answering questions on behalf of others and guiding the interview.

[0003] However, the applicant has found that traditional lip synchronization recognition methods have difficulty in accurately capturing the subtle associations between audio and visual information, resulting in low synchronization recognition accuracy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a lip synchronization recognition method, apparatus, computer equipment and storage medium to solve the problem of low synchronization recognition accuracy in traditional lip synchronization recognition methods.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a lip synchronization recognition method, which adopts the following technical solution:

[0006] Obtain the original video clip to be identified;

[0007] Preprocessing the original video clip to obtain a preprocessed video clip;

[0008] Inputting the preprocessed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is formed by stacking a plurality of visual 3D convolutions, wherein the visual 3D convolutions include a spatial 2D convolution and a visual temporal 1D convolution;

[0009] Inputting the preprocessed video clip into an audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is formed by stacking a plurality of audio 2D convolutions, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio time series 1D convolution;

[0010] Inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data;

[0011] Inputting the global feature data into a lip synchronization classifier to perform lip synchronization recognition processing to obtain a lip synchronization recognition result;

[0012] The lip synchronization recognition result is outputted.

[0013] Furthermore, the step of preprocessing the original video clip to obtain the preprocessed video clip specifically includes the following steps:

[0014] Read the video frame images of the original video clip frame by frame according to the video processing library;

[0015] Performing face detection processing on all the video frame images according to the face detection algorithm to obtain face detection results;

[0016] If the face detection result of the current video frame image detects a face area, key point detection processing is performed on the face area to obtain mouth key point data;

[0017] Acquire a mouth area image corresponding to the mouth key point data according to an image cropping method;

[0018] Performing correction processing on the mouth area image to obtain a standardized mouth image;

[0019] After all video frame images with detected face areas are rectified, the pre-processed video clips are obtained.

[0020] Furthermore, the step of inputting the pre-processed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data specifically includes the following steps:

[0021] Performing preliminary visual feature extraction processing on the preprocessed video clip according to the first visual 3D convolution of the visual feature extraction network to obtain preliminary visual feature extraction data;

[0022] Inputting the preliminary visual feature extraction data into the next visual 3D convolutional layer of the visual feature extraction network for further feature extraction processing to obtain intermediate visual feature extraction data;

[0023] Performing spatial feature extraction processing on the intermediate visual feature extraction data according to the spatial two-dimensional convolution to obtain intermediate spatial feature data;

[0024] Performing visual temporal feature extraction processing on the intermediate visual feature extraction data according to the visual temporal 1-dimensional convolution to obtain intermediate visual temporal feature data;

[0025] Determining whether there is a difference between the intermediate spatial feature data and the intermediate visual temporal feature data;

[0026] If there is a difference between the intermediate spatial feature data and the intermediate visual temporal feature data, the intermediate spatial feature data and the intermediate visual temporal feature data are fused to obtain intermediate visual fusion feature data, and the intermediate visual fusion feature data is input into the next visual 3D convolution layer of the visual feature extraction network for further feature extraction and difference judgment operation;

[0027] If there is no difference between the intermediate spatial feature data and the intermediate visual temporal feature data, the intermediate visual temporal feature data is used as the visual feature data.

[0028] Furthermore, after the step of performing preliminary visual feature extraction processing on the pre-processed video clip according to the first visual 3D convolution of the visual feature extraction network to obtain preliminary visual feature extraction data, the following steps are also included:

[0029] Determining whether the dimension of the preliminary visual feature extraction data meets a preset dimension condition;

[0030] If the dimension of the preliminary visual feature extraction data does not meet the preset dimension condition, the preliminary visual feature extraction data is dimensionally adjusted to obtain preliminary visual feature extraction data that meets the preset dimension condition.

[0031] Furthermore, the step of inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data specifically includes the following steps:

[0032] Performing vectorization processing on the visual feature data and the audio feature data respectively to obtain a visual feature vector and an audio feature vector;

[0033] Performing vector concatenation processing on the visual feature vector and the audio feature vector to obtain a preliminary fused feature vector;

[0034] Inputting the preliminary fused feature vector into a gated recurrent neural network for time series feature extraction processing to obtain time series fused feature data;

[0035] The time series fusion feature data is screened for importance according to a gating mechanism to obtain the global feature data.

[0036] Furthermore, after the step of inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data, the following steps are also included:

[0037] Inputting the global feature data into a deep neural network for deep feature extraction processing to obtain a high-level semantic feature vector;

[0038] Performing feature screening processing on the high-level semantic feature vector according to a feature selection method to obtain enhanced feature data;

[0039] The step of inputting the global feature data into a lip synchronization classifier for lip synchronization recognition processing to obtain a lip synchronization recognition result specifically includes the following steps:

[0040] The enhanced feature data is input into a lip synchronization classifier for lip synchronization recognition processing to obtain the lip synchronization recognition result.

[0041] In order to solve the above technical problems, the embodiment of the present application further provides a lip synchronization recognition device, which adopts the following technical solution:

[0042] The original video acquisition module is used to acquire the original video clip to be identified;

[0043] A preprocessing module, used for preprocessing the original video clip to obtain a preprocessed video clip;

[0044] A visual feature extraction module, used for inputting the pre-processed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is formed by stacking a plurality of visual 3D convolutions, wherein the visual 3D convolutions include a spatial 2D convolution and a visual temporal 1D convolution;

[0045] An audio feature extraction module, used for inputting the pre-processed video clip into an audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is formed by stacking a plurality of audio 2D convolutions, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio time series 1D convolution;

[0046] A feature fusion module, used for inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data;

[0047] A recognition module, used for inputting the global feature data into a lip synchronization classifier for lip synchronization recognition processing to obtain a lip synchronization recognition result;

[0048] The result output module is used to output the lip synchronization recognition result.

[0049] Furthermore, the preprocessing module includes:

[0050] A video frame acquisition submodule, used for reading the video frame images of the original video clip frame by frame according to a video processing library;

[0051] A face detection submodule is used to perform face detection processing on all the video frame images according to a face detection algorithm to obtain face detection results;

[0052] A key point detection submodule is used to perform key point detection processing on the face area to obtain mouth key point data if the face detection result of the current video frame image detects a face area;

[0053] A mouth region acquisition submodule, used for acquiring a mouth region image corresponding to the mouth key point data according to an image cropping method;

[0054] A correction submodule, used for performing correction processing on the mouth area image to obtain a standardized mouth image;

[0055] The preprocessing confirmation submodule is used to obtain the preprocessed video clip after all video frame images with detected face areas are corrected.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0057] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the lip synchronization recognition method as described above when executing the computer-readable instructions.

[0058] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0059] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the lip synchronization recognition method described above are implemented.

[0060] The present application provides a lip synchronization recognition method, comprising: obtaining an original video clip to be recognized; preprocessing the original video clip to obtain a preprocessed video clip; inputting the preprocessed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is formed by stacking a plurality of visual 3D convolutions, and the visual 3D convolution includes a spatial 2D convolution and a visual temporal 1D convolution; inputting the preprocessed video clip into an audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is formed by stacking a plurality of audio 2D convolutions, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio temporal 1D convolution; inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data; inputting the global feature data into a lip synchronization classifier for lip synchronization recognition processing to obtain a lip synchronization recognition result; and outputting the lip synchronization recognition result. Compared with the existing technology, this application addresses the problem of audio and lip shape being out of sync in videos by adopting an innovative multimodal feature fusion method, which can effectively capture the timing relationship between audio and lip shape, greatly improve the accuracy of lip synchronization recognition, and provide reliable technical support for video quality assessment and anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0063] Figure 2 is a flowchart of the implementation of the lip synchronization recognition method provided in Example 1 of the present application;

[0064] Figure 3 is a structural diagram of a lip synchronization recognition device provided in Embodiment 2 of the present application;

[0065] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0070] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0071] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0072] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0073] It should be noted that the lip synchronization recognition method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the lip synchronization recognition device is generally arranged in the server / terminal device.

[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0075] Embodiment 1

[0076] Continue to refer Figure 2 , showing a flow chart of an embodiment of a lip synchronization recognition method according to the present application. The lip synchronization recognition method comprises: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0077] In step S201, an original video segment to be identified is obtained.

[0078] In step S202, the original video segment is preprocessed to obtain a preprocessed video segment.

[0079] In an embodiment of the present application, preprocessing may be to extract all video frames of a video clip, use facial key points to locate the mouth area, crop the mouth area, and perform correction.

[0080] In step S203, the preprocessed video clip is input into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is composed of a plurality of visual 3D convolutions stacked together, and the visual 3D convolution includes a spatial 2D convolution and a visual temporal 1D convolution.

[0081] In the implementation of the traditional model, in order to speed up the extraction of visual features, only the last part of the features of each frame uses 3D convolution, and the previous part only extracts visual features of a single frame. Although this method speeds up the reasoning speed of the model, it cannot fully extract time-related features.

[0082] In the embodiment of the present application, in order to better extract timing-related information, all visual features are extracted using 3D convolution. Different from the traditional 3D convolution, this patent proposes a separated 3D convolution, which is composed of a spatial 2D convolution kernel and a temporal 1D convolution, so the first convolution extracts spatial information and the second convolution extracts temporal information. The speed of the decomposed 3D convolution is greatly improved, and a visual feature extraction network is formed by stacking multiple such 3D convolutions.

[0083] In step S204, the preprocessed video clip is input into the audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is composed of a plurality of audio 2D convolutions stacked together, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio timing 1D convolution.

[0084] In the embodiment of the present application, the audio timing feature is generally a two-dimensional feature (one dimension is time, and the other dimension is the MFCC dimension). In order to improve the extraction speed of audio features, the two-dimensional convolution of the audio is decomposed into two one-dimensional convolutions, one convolution is used to extract the features of the MFCC dimension, and the second convolution is used to extract timing-related information. Multiple such two-dimensional convolutions are stacked to form an audio feature extraction network.

[0085] In step S205, the visual feature data and the audio feature data are input into a gated recurrent neural network for feature fusion to obtain global feature data.

[0086] In the embodiment of the present application, it is necessary to fuse video features and audio features. At present, some methods of fusing these two features may adopt the method of transformer, which is based on the large amount of calculation. The present application adopts a lightweight gated recurrent neural network, first adding video features to audio features, and then sending them to the GRU network to model this multimodal feature.

[0087] In step S206, the global feature data is input into a lip synchronization classifier for lip synchronization recognition processing to obtain a lip synchronization recognition result.

[0088] In an embodiment of the present application, the completed global feature data is input into a lip synchronization classifier for lip synchronization recognition processing. The classifier adopts a support vector machine (SVM) model to finally obtain a lip synchronization recognition result.

[0089] In step S207, the lip synchronization recognition result is output.

[0090] In an embodiment of the present application, a lip synchronization recognition method is provided, including: obtaining an original video clip to be recognized; preprocessing the original video clip to obtain a preprocessed video clip; inputting the preprocessed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is composed of a plurality of visual 3D convolutions stacked, and the visual 3D convolution includes a spatial 2D convolution and a visual temporal 1D convolution; inputting the preprocessed video clip into an audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is composed of a plurality of audio 2D convolutions stacked, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio temporal 1D convolution; inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data; inputting the global feature data into a lip synchronization classifier for lip synchronization recognition processing to obtain a lip synchronization recognition result; and outputting the lip synchronization recognition result. Compared with the existing technology, this application addresses the problem of audio and lip shape being out of sync in videos by adopting an innovative multimodal feature fusion method, which can effectively capture the timing relationship between audio and lip shape, greatly improve the accuracy of lip synchronization recognition, and provide reliable technical support for video quality assessment and anomaly detection.

[0091] In some optional implementations of the embodiments of the present application, step S202 includes the following steps:

[0092] Read the video frame images of the original video clip frame by frame according to the video processing library;

[0093] Perform face detection processing on all video frame images according to the face detection algorithm to obtain face detection results;

[0094] If the face detection result of the current video frame image detects a face area, key point detection processing is performed on the face area to obtain mouth key point data;

[0095] Obtain a mouth region image corresponding to the mouth key point data according to an image cropping method;

[0096] Correct the mouth area image to obtain a standardized mouth image;

[0097] After all video frame images with detected face areas are rectified, preprocessed video clips are obtained.

[0098] In an embodiment of the present application, when the video processing library OpenCV is used to read the video frame by frame, the cvVideoCapture function can be used to read the video file; then, the video image is read frame by frame through cap.read(), and the image is converted into a grayscale image using cvcvtCol or to improve the efficiency of face detection; face detection is performed using OpenCV's pre-trained face detection classifier, and the coordinates and size of the face area in each frame of the image are determined through the detectMultiScale method; after a face is detected in a frame of the image, the Dlib library or other face key point detection algorithms can be used to further detect face key points.

[0099] In an embodiment of the present application, facial key points can be obtained through predictor(gray, face), where face is the detected face area; for the mouth area, the key points are generally the 49th to 68th points; by calculating the minimum circumscribed rectangle of these key points, the position and size of the mouth area can be determined; using the image cropping method of OpenCV, the located mouth area can be cropped out from the original frame image.

[0100] In the embodiment of the present application, if the cropped mouth area image is tilted or deformed, it can be corrected by calculating the tilt angle of the mouth key points.

[0101] Continue reading Figure 4 , showing Figure 2 A flowchart of a specific implementation of step S** in FIG. 1 , for ease of explanation, only the part related to the present application is shown.

[0102] In some optional implementations of the embodiments of the present application, step S203 may further perform the following steps:

[0103] Perform preliminary visual feature extraction processing on the preprocessed video clip according to the first visual 3D convolution of the visual feature extraction network to obtain preliminary visual feature extraction data;

[0104] The preliminary visual feature extraction data is input into the next visual 3D convolutional layer of the visual feature extraction network for further feature extraction processing to obtain intermediate visual feature extraction data;

[0105] Performing spatial feature extraction processing on the intermediate visual feature extraction data according to the spatial 2D convolution to obtain intermediate spatial feature data;

[0106] Perform visual temporal feature extraction processing on the intermediate visual feature extraction data according to the visual temporal 1-dimensional convolution to obtain the intermediate visual temporal feature data;

[0107] Determine whether there is a difference between the intermediate spatial feature data and the intermediate visual temporal feature data;

[0108] If there is a difference between the intermediate spatial feature data and the intermediate visual temporal feature data, the intermediate spatial feature data and the intermediate visual temporal feature data are fused to obtain intermediate visual fused feature data, and the intermediate visual fused feature data is input into the next visual 3D convolutional layer of the visual feature extraction network for further feature extraction and difference judgment operations;

[0109] If there is no difference between the intermediate spatial feature data and the intermediate visual temporal feature data, the intermediate visual temporal feature data is used as the visual feature data.

[0110] In an embodiment of the present application, it is assumed that the present application has a preprocessed video clip, which is first input into a visual feature extraction network; the size of the video clip is (16, 128, 128, 3), where 16 represents the number of frames, 128×128 is the resolution of each frame, and 3 represents the RGB channel; through the first visual 3D convolution layer in the visual feature extraction network, this layer uses 64 convolution kernels of size (3, 3, 3), with a step size of (1, 1, 1), and no padding; the output shape after convolution is calculated to be (16, 126, 126, 64).

[0111] In some optional implementations of the embodiments of the present application, after performing preliminary visual feature extraction processing on the preprocessed video clip according to the first visual 3D convolution of the visual feature extraction network to obtain preliminary visual feature extraction data, the following steps are also included:

[0112] Determine whether the dimension of the preliminary visual feature extraction data meets the preset dimension condition;

[0113] If the dimension of the preliminary visual feature extraction data does not meet the preset dimension condition, the preliminary visual feature extraction data is dimensionally adjusted to obtain preliminary visual feature extraction data that meets the preset dimension condition.

[0114] In an embodiment of the present application, the preliminary visual feature data is input into the next visual 3D convolution layer. Assume that the layer uses 128 convolution kernels of size (3, 3, 3), with a step size of (1, 1, 1), without padding, and obtains intermediate visual feature data with a dimension of (16, 124, 124, 128); next, spatial 2D convolution is used to extract spatial features from the intermediate visual feature data. The 2D convolution layer uses 128 convolution kernels of size (3, 3), with a step size of (1, 1), without padding, and an output dimension of (16, 122, 122, 128) to obtain spatial feature data; according to the intermediate visual feature data, visual temporal 1D convolution is used to extract temporal features. The 1D convolution layer uses 128 convolution kernels of size (3), with a step size of 1, without padding. The output dimension is (14, 124, 124, 128), and the temporal feature data is obtained; next, it is determined whether there is a significant difference between the spatial feature data and the temporal feature data, which can be determined by calculating their Euclidean distance or cosine similarity; if there is a large difference, a feature fusion method is used, such as weighted averaging or splicing, to obtain fused visual feature data. Assuming that weighted averaging is used, the output dimension is (16, 122, 122, 128); finally, through the last visual 3D convolution layer of the visual feature extraction network, which uses 256 convolution kernels of size (3, 3, 3), a step size of (1, 1, 1), and no padding, the fused visual feature data is processed again to obtain the final visual feature data with a dimension of (16, 120, 120, 256).

[0115] In an embodiment of the present application, the format of the final visual feature data is ensured to conform to the expected output format. If output to a specific shape is required, it can be processed through a fully connected layer or global average pooling and output as (256) visual feature data.

[0116] In some optional implementations of the embodiments of the present application, step S205 may further perform the following steps:

[0117] Performing vectorization processing on the visual feature data and the audio feature data respectively to obtain a visual feature vector and an audio feature vector;

[0118] Perform vector concatenation processing on the visual feature vector and the audio feature vector to obtain a preliminary fused feature vector;

[0119] The preliminary fused feature vector is input into the gated recurrent neural network for time series feature extraction processing to obtain time series fused feature data;

[0120] The time series fusion feature data is screened for importance according to the gating mechanism to obtain global feature data.

[0121] In an embodiment of the present application, in order to capture the changes and connections in the time dimension, a gated recurrent neural network (GRU) is used, the preliminary fused feature vector is input into the GRU network, the number of hidden units is set to 256, and after one or more layers of GRU processing, the output processing result is a 256-dimensional time series feature vector.

[0122] In the embodiment of the present application, through the gating mechanism of GRU, it is possible to judge and select the information that needs to be remembered and forgotten, extract important information from the time series feature vector, and obtain global feature data.

[0123] In some optional implementations of the embodiments of the present application, the following steps may be further performed after step S205:

[0124] The global feature data is input into the deep neural network for deep feature extraction to obtain a high-level semantic feature vector;

[0125] Perform feature screening on high-level semantic feature vectors according to feature selection method to obtain enhanced feature data;

[0126] The above step S206 specifically includes the following steps:

[0127] The enhanced feature data is input into the lip synchronization classifier for lip synchronization recognition processing to obtain the lip synchronization recognition result.

[0128] In an embodiment of the present application, the global feature data is then input into a deep neural network (DNN) comprising two fully connected layers with 512 neurons in each layer for further feature extraction to obtain a high-level semantic feature vector.

[0129] In an embodiment of the present application, if the high-level semantic feature vector contains redundant information, a principal component analysis (PCA) algorithm can be used to reduce the dimension of the feature vector to 128 dimensions, retain the most representative features, and enhance the distinctiveness of the features.

[0130] In an embodiment of the present application, the screened 128-dimensional feature vector is input into a support vector machine (SVM) classifier or a convolutional neural network detection module for category judgment or target detection.

[0131] In some optional implementations of the embodiments of the present application, the classification or detection results can be evaluated to calculate indicators such as accuracy and recall rate, and the model parameters can be further optimized to ensure the performance and accuracy of the model.

[0132] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0133] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0135] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0136] Embodiment 2

[0137] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a lip synchronization recognition device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0138] like Figure 3As shown, the lip synchronization recognition device 200 of the embodiment of the present application includes:

[0139] The original video acquisition module 210 is used to acquire the original video segment to be identified;

[0140] The preprocessing module 220 is used to preprocess the original video clip to obtain a preprocessed video clip;

[0141] A visual feature extraction module 230 is used to input the pre-processed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is formed by stacking a plurality of visual 3D convolutions, and the visual 3D convolution includes a spatial 2D convolution and a visual temporal 1D convolution;

[0142] The audio feature extraction module 240 is used to input the pre-processed video clip into the audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is stacked by a plurality of audio 2D convolutions, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio time series 1D convolution;

[0143] A feature fusion module 250 is used to input the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data;

[0144] The recognition module 260 is used to input the global feature data into the lip synchronization classifier to perform lip synchronization recognition processing to obtain a lip synchronization recognition result;

[0145] The result output module 270 is used to output the lip synchronization recognition result.

[0146] In an embodiment of the present application, a lip synchronization recognition device 200 is provided, including: an original video acquisition module 210, used to acquire an original video segment to be recognized; a preprocessing module 220, used to preprocess the original video segment to obtain a preprocessed video segment; a visual feature extraction module 230, used to input the preprocessed video segment into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is composed of a plurality of visual 3D convolutions stacked, and the visual 3D convolution includes a spatial 2D convolution and a visual temporal 1D convolution; an audio feature extraction module 240, used to input the preprocessed video segment into a visual feature extraction network for visual feature extraction processing to obtain visual feature data. The video clip is input into the audio feature network for audio feature extraction and processing to obtain audio feature data, wherein the audio feature network is composed of a plurality of audio 2D convolutions stacked together, and the audio 2D convolution includes a MFCC dimension 1D convolution and an audio timing 1D convolution; the feature fusion module 250 is used to input the visual feature data and the audio feature data into the gated recurrent neural network for feature fusion to obtain global feature data; the recognition module 260 is used to input the global feature data into the lip synchronization classifier for lip synchronization recognition processing to obtain the lip synchronization recognition result; the result output module 270 is used to output the lip synchronization recognition result. Compared with the prior art, the present application adopts an innovative multimodal feature fusion method to solve the problem of audio and lip shape being out of sync in the video, which can effectively capture the temporal relationship between audio and lip shape, greatly improve the accuracy of lip synchronization recognition, and provide reliable technical support for video quality assessment and anomaly detection.

[0147] In some optional implementations of the embodiments of the present application, the preprocessing module includes:

[0148] The video frame acquisition submodule is used to read the video frame images of the original video clip frame by frame according to the video processing library;

[0149] The face detection submodule is used to perform face detection processing on all video frame images according to the face detection algorithm to obtain face detection results;

[0150] The key point detection submodule is used to perform key point detection processing on the face area to obtain the mouth key point data if the face detection result of the current video frame image detects a face area;

[0151] A mouth region acquisition submodule, used for acquiring a mouth region image corresponding to the mouth key point data according to an image cropping method;

[0152] A correction submodule is used to correct the mouth area image to obtain a standardized mouth image;

[0153] The preprocessing confirmation submodule is used to obtain preprocessed video clips after all video frame images with detected face areas are corrected.

[0154] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0155] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0156] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0157] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the lip synchronization recognition method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0158] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the lip synchronization recognition method.

[0159] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0160] The computer device provided in this application adopts an innovative multimodal feature fusion method to address the problem of audio and lip shape being out of sync in videos. It can effectively capture the timing relationship between audio and lip shape, greatly improve the accuracy of lip synchronization recognition, and provide reliable technical support for video quality assessment and anomaly detection.

[0161] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the lip synchronization recognition method as described above.

[0162] The computer-readable storage medium provided in this application adopts an innovative multimodal feature fusion method to address the problem of audio and lip shape being out of sync in videos. It can effectively capture the timing relationship between audio and lip shape, greatly improve the accuracy of lip synchronization recognition, and provide reliable technical support for video quality assessment and anomaly detection.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0164] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A lip synchronization recognition method, characterized in that: The steps include: Obtain the original video clip to be identified; Preprocessing the original video clip to obtain a preprocessed video clip; Inputting the preprocessed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is formed by stacking a plurality of visual 3D convolutions, wherein the visual 3D convolutions include a spatial 2D convolution and a visual temporal 1D convolution; Inputting the preprocessed video clip into an audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is formed by stacking a plurality of audio 2D convolutions, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio time series 1D convolution; Inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data; Inputting the global feature data into a lip synchronization classifier to perform lip synchronization recognition processing to obtain a lip synchronization recognition result; The lip synchronization recognition result is outputted.

2. The lip synchronization recognition method according to claim 1, characterized in that: The step of preprocessing the original video clip to obtain the preprocessed video clip specifically includes the following steps: Read the video frame images of the original video clip frame by frame according to the video processing library; Performing face detection processing on all the video frame images according to the face detection algorithm to obtain face detection results; If the face detection result of the current video frame image detects a face area, key point detection processing is performed on the face area to obtain mouth key point data; Acquire a mouth area image corresponding to the mouth key point data according to an image cropping method; Performing correction processing on the mouth area image to obtain a standardized mouth image; After all video frame images with detected face areas are rectified, the pre-processed video clips are obtained.

3. The lip synchronization recognition method according to claim 1, characterized in that: The step of inputting the pre-processed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data specifically includes the following steps: Performing preliminary visual feature extraction processing on the preprocessed video clip according to the first visual 3D convolution of the visual feature extraction network to obtain preliminary visual feature extraction data; Inputting the preliminary visual feature extraction data into the next visual 3D convolutional layer of the visual feature extraction network for further feature extraction processing to obtain intermediate visual feature extraction data; Performing spatial feature extraction processing on the intermediate visual feature extraction data according to the spatial two-dimensional convolution to obtain intermediate spatial feature data; Performing visual temporal feature extraction processing on the intermediate visual feature extraction data according to the visual temporal 1-dimensional convolution to obtain intermediate visual temporal feature data; Determining whether there is a difference between the intermediate spatial feature data and the intermediate visual temporal feature data; If there is a difference between the intermediate spatial feature data and the intermediate visual temporal feature data, the intermediate spatial feature data and the intermediate visual temporal feature data are fused to obtain intermediate visual fusion feature data, and the intermediate visual fusion feature data is input into the next visual 3D convolution layer of the visual feature extraction network for further feature extraction and difference judgment operation; If there is no difference between the intermediate spatial feature data and the intermediate visual temporal feature data, the intermediate visual temporal feature data is used as the visual feature data.

4. The lip synchronization recognition method according to claim 3, characterized in that: After the step of performing preliminary visual feature extraction processing on the pre-processed video clip according to the first visual 3D convolution of the visual feature extraction network to obtain preliminary visual feature extraction data, the following steps are also included: Determining whether the dimension of the preliminary visual feature extraction data meets a preset dimension condition; If the dimension of the preliminary visual feature extraction data does not meet the preset dimension condition, the preliminary visual feature extraction data is dimensionally adjusted to obtain preliminary visual feature extraction data that meets the preset dimension condition.

5. The lip synchronization recognition method according to claim 1, characterized in that: The step of inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data specifically includes the following steps: Performing vectorization processing on the visual feature data and the audio feature data respectively to obtain a visual feature vector and an audio feature vector; Performing vector concatenation processing on the visual feature vector and the audio feature vector to obtain a preliminary fused feature vector; Inputting the preliminary fused feature vector into a gated recurrent neural network for time series feature extraction processing to obtain time series fused feature data; The time series fusion feature data is screened for importance according to a gating mechanism to obtain the global feature data.

6. The lip synchronization recognition method according to claim 1, characterized in that: After the step of inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data, the following step is also included: Inputting the global feature data into a deep neural network for deep feature extraction processing to obtain a high-level semantic feature vector; Performing feature screening processing on the high-level semantic feature vector according to a feature selection method to obtain enhanced feature data; The step of inputting the global feature data into a lip synchronization classifier for lip synchronization recognition processing to obtain a lip synchronization recognition result specifically includes the following steps: The enhanced feature data is input into a lip synchronization classifier for lip synchronization recognition processing to obtain the lip synchronization recognition result.

7. A lip synchronization recognition device, characterized in that: include: The original video acquisition module is used to acquire the original video clip to be identified; A preprocessing module, used for preprocessing the original video clip to obtain a preprocessed video clip; A visual feature extraction module, used for inputting the pre-processed video clip into a visual feature extraction network for visual feature extraction processing to obtain visual feature data, wherein the visual feature extraction network is formed by stacking a plurality of visual 3D convolutions, wherein the visual 3D convolutions include a spatial 2D convolution and a visual temporal 1D convolution; An audio feature extraction module, used for inputting the pre-processed video clip into an audio feature network for audio feature extraction processing to obtain audio feature data, wherein the audio feature network is formed by stacking a plurality of audio 2D convolutions, and the audio 2D convolution includes an MFCC dimension 1D convolution and an audio time series 1D convolution; A feature fusion module, used for inputting the visual feature data and the audio feature data into a gated recurrent neural network for feature fusion to obtain global feature data; A recognition module, used for inputting the global feature data into a lip synchronization classifier for lip synchronization recognition processing to obtain a lip synchronization recognition result; The result output module is used to output the lip synchronization recognition result.

8. The lip synchronization recognition device according to claim 7, characterized in that: The preprocessing module comprises: A video frame acquisition submodule, used for reading the video frame images of the original video clip frame by frame according to a video processing library; A face detection submodule is used to perform face detection processing on all the video frame images according to a face detection algorithm to obtain face detection results; A key point detection submodule is used to perform key point detection processing on the face area to obtain mouth key point data if the face detection result of the current video frame image detects a face area; A mouth region acquisition submodule, used for acquiring a mouth region image corresponding to the mouth key point data according to an image cropping method; A correction submodule, used for performing correction processing on the mouth area image to obtain a standardized mouth image; The preprocessing confirmation submodule is used to obtain the preprocessed video clip after all video frame images with detected face areas are corrected.

9. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the lip synchronization recognition method according to any one of claims 1 to 6 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the lip synchronization recognition method according to any one of claims 1 to 6 are implemented.

Images