Lip language recognition-based Chinese digital population lip matching degree evaluation method and device, and lip language recognition equipment

Abstract

The invention relates to the technical field of Chinese digital people, and discloses a method and device for evaluating the lip matching degree of a Chinese digital person based on lip language recognition and lip language recognition equipment, and the method comprises the steps: obtaining an image sequence of a lip region in an original video and a real text corresponding to the original video; performing text reasoning on the image sequence of the lip region according to an AVSR model and an RNN language model to obtain a prediction text corresponding to the image sequence; and performing Chinese syllable matching degree calculation on the real text and the predicted text in the sliding window to obtain an evaluation result of the lip matching degree. According to the invention, the matching degree of the lip language recognition result can be evaluated. The accuracy of evaluation is improved, and the problem of complexity caused by polyphones and homophones in a Chinese environment is solved.

Description

Technical Field

[0001] The present application relates to the technical field of Chinese digital humans, for example, to a method and device for evaluating the lip matching degree of a Chinese digital human based on lip reading recognition, and a lip reading recognition device. Background Art

[0002] At present, lip synchronization technology uses computer vision and speech recognition technology to synchronize the lip movements of digital humans with audio content, thereby achieving a natural and smooth visual effect. Lip matching refers to the degree of matching between the lip shape and pronunciation of a digital human, and is one of the important indicators for measuring the authenticity and naturalness of a digital human. However, current technicians need to find ways to improve the lip matching of Chinese digital humans.

[0003] In order to improve the lip matching degree of Chinese digital population, the related technology discloses a lip reading recognition method, including: standardizing the frame rate of the original video, and separating the obtained standard video to obtain a valid audio stream and a valid video stream; using a face recognition algorithm to track the face in the valid video stream, and extracting the mouth area in the face, obtaining a frame lip movement video, and thus obtaining a lip image sequence; using a sequence segmentation rule to segment the lip image sequence to obtain a segmented image sequence; inputting each segmented image sequence corresponding to the lip image sequence into a lip image recognition model in turn for recognition, and obtaining a lip image recognition result; inputting the valid audio stream into a speech recognition model to obtain a speech recognition result; when the similarity between the two reaches a preset value, the lip image recognition result is used as the lip reading recognition result of the original video.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0005] The relevant technology can perform speech recognition on lip images in videos, but the recognition results have not been evaluated, making it difficult to determine whether the recognition results are accurate.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a method and apparatus for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition, and a lip reading recognition device, which can evaluate the matching degree of the lip reading recognition results.

[0009] In some embodiments, the method for evaluating the lip matching of a Chinese digital population based on lip reading recognition includes: obtaining an image sequence of the lip area in an original video and a real text corresponding to the original video; performing text inference on the image sequence of the lip area according to an AVSR (Audio-Visual Speech Recognition) model and an RNN (Recurrent Neural Network) language model to obtain a predicted text corresponding to the image sequence; and calculating the Chinese syllable matching of the real text and the predicted text within a sliding window to obtain an evaluation result of the lip matching.

[0010] Optionally, text inference is performed on the image sequence of the lip area according to the audio-video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain predicted text corresponding to the image sequence, including: encoding the image sequence using the AVSR model to extract the spatiotemporal features of the image sequence as a token sequence; according to the RNN language model, decoding the token sequences of multiple paths through beam search to obtain predicted text corresponding to the image sequence.

[0011] Optionally, the image sequence is encoded using the AVSR model, including: extracting features from the image sequence through the residual network of the AVSR model; converting the image sequence after feature extraction into a low-dimensional feature sequence, and adding coding to the low-dimensional feature sequence; and inputting the low-dimensional feature sequence after adding the coding into the encoder layer for encoding.

[0012] Optionally, the encoded low-dimensional feature sequence is input into the encoder layer for encoding, including: calculating the attention distribution of the low-dimensional feature sequence to obtain context features; performing nonlinear mapping on the context features to obtain the encoded low-dimensional feature sequence.

[0013] Optionally, the method for evaluating lip matching of Chinese digital speakers based on lip reading recognition further includes: before adding coding to the low-dimensional feature sequence, performing residual connection and normalization operations on the low-dimensional feature sequence.

[0014] Optionally, according to the RNN language model, the token sequences of multiple paths are decoded through beam search to obtain predicted text corresponding to the image sequence, including: according to the context of the current path, the token sequences of multiple paths are input into the RNN language model for scoring to obtain probability scores; the decoder score and the CTC (Connectionist Temporal Classification) score are obtained; according to the decoder score, the CTC connectionist temporal classification score and the probability score, the comprehensive scores of the multiple paths are calculated respectively; the token sequence of the path with the highest comprehensive score is converted into a text string to obtain the predicted text corresponding to the image sequence.

[0015] Optionally, the Chinese syllable matching degree of the real text and the predicted text is calculated within the sliding window, including: converting the real text and the predicted text into pinyin; calculating the matching degree of each word of the real text and the predicted text according to multiple index numbers of the predicted text pinyin within the sliding window; and calculating the average matching degree of all words as the evaluation result of lip matching.

[0016] Optionally, the matching degree of each word between the real text and the predicted text is calculated based on multiple index numbers of the predicted text pinyin in the sliding window, including: based on the multiple index numbers of the predicted text pinyin in the sliding window, the predicted text pinyin corresponding to the index numbers are converted into the real text pinyin in turn to obtain multiple editing times; the minimum editing number is used as the matching degree of a single word between the real text and the predicted text; and the matching degree of each word between the real text and the predicted text is calculated respectively.

[0017] Optionally, multiple index numbers of the predicted text pinyin in the sliding window are calculated as follows: the index number of the real text is multiplied by the displacement coefficient; wherein the displacement coefficient is the ratio of the real text to the predicted text; and the multiplication result is added to multiple sliding window values ​​to obtain multiple index numbers of the predicted text.

[0018] Optionally, an image sequence of the lip area in the original video is obtained, including: decomposing the original video into multiple image frames and transcoding them; recording the lip position in each image frame according to the key point coordinates of the lips; cropping the lip position in the image frame and performing noise reduction processing as the input of the AVSR model; and using the output of the AVSR model as the image sequence of the lip area in the original video.

[0019] In some embodiments, the Chinese digital population lip matching evaluation device based on lip reading recognition includes: an input preprocessing module, configured to obtain an image sequence of the lip area in the original video and a real text corresponding to the original video; a model inference module, configured to perform text inference on the image sequence of the lip area according to the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model, and obtain a predicted text corresponding to the image sequence; an evaluation calculation module, configured to perform Chinese syllable matching calculation on the real text and the predicted text within a sliding window, and obtain an evaluation result of the lip matching.

[0020] In some embodiments, the device for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition includes: a processor and a memory storing program instructions, and the processor is configured to execute the method for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition as described above when running the program instructions.

[0021] In some embodiments, the lip reading recognition device includes: a lip reading recognition device body; and the above-mentioned Chinese digital population lip matching evaluation device based on lip reading recognition, which is installed in the lip reading recognition device body.

[0022] The method and device for evaluating the lip matching degree of a Chinese digital person based on lip reading recognition, and the lip reading recognition device provided in the embodiments of the present disclosure can achieve the following technical effects:

[0023] The disclosed embodiment realizes efficient evaluation of lip matching of Chinese digital population. First, the image sequence of the lip area and the corresponding real text are obtained from the original video, and the audio and video speech recognition (AVSR) model and the recurrent neural network (RNN) language model are used to perform text reasoning on the image sequence to generate predicted text. Secondly, by converting both the real text and the predicted text into pinyin, and performing matching calculations within the sliding window to reduce the speech alignment error, the evaluation result of the lip matching is finally obtained. This process not only improves the accuracy of the evaluation, but also solves the complexity problems caused by polyphones and homophones in the Chinese environment. The disclosed embodiment realizes the ability to convert visual indicators that require subjective judgment in digital population lip recognition into quantifiable objective evaluation indicators, which is particularly suitable for Chinese environments, and can more accurately and objectively evaluate lip matching, providing important technical support for the development of the digital human industry.

[0024] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0026] Figure 1 is a schematic diagram of an implementation environment of a method for evaluating lip matching of Chinese digital population based on lip reading recognition according to an embodiment of the present disclosure;

[0027] Figure 2 is a schematic diagram of a method for evaluating lip matching of Chinese digital population based on lip reading recognition provided by an embodiment of the present disclosure;

[0028] Figure 3 is a schematic diagram of another method for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition provided by an embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram of another method for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition provided by an embodiment of the present disclosure;

[0030] Figure 5 is a schematic diagram of another method for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition provided by an embodiment of the present disclosure;

[0031] Figure 6 is a schematic diagram of a Chinese digital population lip matching evaluation device based on lip reading recognition provided by an embodiment of the present disclosure;

[0032] Figure 7 It is a schematic diagram of another device for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0034] The terms "first", "second", etc. in the technical solutions described in the embodiments of the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0035] Unless otherwise stated, the term "plurality" means two or more.

[0036] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0037] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0038] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0039] Digital humans are digital human images that are close to human images and are created using core technologies such as computer graphics, motion capture, speech synthesis, image rendering, and deep learning. They have human appearance, behavior patterns, and language abilities, and can interact with humans in digital space. Currently, digital humans are widely used in media, entertainment, medical care, education, finance, and other fields.

[0040] The current Chinese digital human testing field lacks a solution for measuring lip matching, especially in the Chinese scenario, which involves complex problems such as polyphones, homophones, near-phonetic characters, voice alignment, and visual ambiguity. Directly using the lip reading recognition model to calculate the lip matching will lead to inaccurate results. In addition, artificial intelligence technology relies on a large amount of data for model training and testing, while the digital human field, especially data sets for lip matching, are extremely scarce. Therefore, the disclosed embodiment constructs an automated digital human lip reading recognition system that can extract the lip area from the video frame based on the input digital human video, accurately calculate the lip matching through the data preprocessing module, the model reasoning module, and the evaluation calculation module, and output the evaluation result.

[0041] Figure 1 Schematic diagram of the implementation environment of the method for evaluating the lip matching degree of Chinese digital population based on lip reading recognition according to the embodiment of the present disclosure. Figure 1 As shown, the implementation environment may include a front-end display interface 100 and a back-end processor 200 .

[0042] The disclosed embodiment has a front-end and back-end separation design. The back-end processor 200 continuously and cyclically captures the input of the front-end display interface 100, and sequentially executes the input preprocessing module, the model reasoning module, and the evaluation calculation module when the input changes. The results are summarized into a lip matching evaluation report, which is sent to the front-end display interface 100 and the visual display is updated.

[0043] Combination Figure 2 As shown, the embodiment of the present disclosure provides a method for evaluating the lip matching degree of a Chinese digital person based on lip reading recognition, comprising:

[0044] S201, a backend processor obtains an image sequence of a lip region in an original video and real text corresponding to the original video.

[0045] S202, the back-end processor performs text inference on the image sequence of the lip area according to the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain predicted text corresponding to the image sequence.

[0046] S203, the back-end processor calculates the Chinese syllable matching degree of the real text and the predicted text in the sliding window to obtain the evaluation result of the lip matching degree.

[0047] The lip matching evaluation method for Chinese digital population based on lip reading recognition provided by the embodiment of the present disclosure is adopted to realize the efficient evaluation of the lip matching of Chinese digital population. First, the image sequence of the lip area and the corresponding real text are obtained from the original video, and the audio and video speech recognition (AVSR) model and the recurrent neural network (RNN) language model are used to perform text reasoning on the image sequence to generate predicted text. Secondly, by converting both the real text and the predicted text into pinyin, and performing matching calculation in the sliding window to reduce the speech alignment error, the evaluation result of the lip matching is finally obtained. This process not only improves the accuracy of the evaluation, but also solves the complexity problem caused by polyphones and homophones in the Chinese environment. The embodiment of the present disclosure realizes the ability to convert the visual indicators that require subjective judgment in the lip recognition of the digital population into quantifiable objective evaluation indicators, which is particularly suitable for the Chinese environment, and can more accurately and objectively evaluate the lip matching, providing important technical support for the development of the digital human industry.

[0048] Optionally, text inference is performed on the image sequence of the lip area according to the audio-video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain predicted text corresponding to the image sequence, including: encoding the image sequence using the AVSR model to extract the spatiotemporal features of the image sequence as a token sequence; according to the RNN language model, decoding the token sequences of multiple paths through beam search to obtain predicted text corresponding to the image sequence.

[0049] Combination Figure 3 As shown, the embodiment of the present disclosure provides another method for evaluating the lip matching degree of Chinese digital population based on lip reading recognition, comprising:

[0050] S301, a backend processor obtains an image sequence of a lip region in an original video and real text corresponding to the original video.

[0051] S302: The backend processor encodes the image sequence using the AVSR model to extract the spatiotemporal features of the image sequence as a token sequence.

[0052] S303, the backend processor decodes the token sequences of multiple paths through beam search according to the RNN language model to obtain predicted text corresponding to the image sequence.

[0053] S304, the backend processor calculates the Chinese syllable matching degree of the real text and the predicted text in the sliding window to obtain the evaluation result of the lip matching degree.

[0054] In the disclosed embodiment, the acquired lip area image sequence is first encoded using the AVSR model to extract its spatiotemporal features and form a series of token sequences. This process is implemented through 3D convolution and attention mechanisms, ensuring the effective capture and processing of long sequence information. The extracted token sequence is input into the recurrent neural network RNN ​​language model, and multiple possible decoding paths are evaluated through the beam search algorithm. During the beam search process, the RNN language model generates a probability score for each token sequence based on the contextual information, and combines other scoring modules (such as decoders and CTC) for comprehensive scoring, thereby selecting the optimal path and finally obtaining the predicted text corresponding to the image sequence.

[0055] By combining the advantages of the AVSR model and the RNN language model, the disclosed embodiments can efficiently and accurately match lip movements with corresponding text content, significantly improving the lip reading recognition accuracy of Chinese digital population and overcoming the limitations of traditional methods in the recognition of polyphones and homophones.

[0056] Optionally, the image sequence is encoded using the AVSR model, including: extracting features from the image sequence through the residual network of the AVSR model; converting the image sequence after feature extraction into a low-dimensional feature sequence, and adding coding to the low-dimensional feature sequence; and inputting the low-dimensional feature sequence after adding the coding into the encoder layer for encoding.

[0057] In the disclosed embodiment, first, the input lip area image sequence is feature extracted through the residual network in the AVSR model, which can capture the key visual information in the image, including the lip shape and motion features. The extracted features are processed through a convolution layer and a linear transformation to a low-dimensional feature sequence suitable for processing by the Transformer architecture, which not only reduces the dimension of the data, but also retains the most important information, allowing the model to process data more efficiently. Then, position encoding is added to the low-dimensional feature sequence to provide the model with the position information of each element in the sequence, so that the model can understand the time order and structural relationship in the sequence. Finally, the low-dimensional feature sequence with position encoding added is input to the encoder layer for encoding. The encoder layer contains a multi-head self-attention mechanism and a feedforward network, which can further extract the contextual information in the sequence and generate a richer and more abstract feature representation.

[0058] In this way, the complex image sequence information can be converted into a numerical form that the model can understand and process, so that the model can accurately capture the correspondence between lip movements and pronunciations, and improve the accuracy and efficiency of digital population lip matching evaluation. Through this encoding method, the model can not only process a single image frame, but also understand the temporal dynamic changes in the image sequence, which is crucial to improving the performance of digital population lip synchronization technology, especially in a language environment such as Chinese where the complexity of speech and visual information is high.

[0059] Optionally, the encoded low-dimensional feature sequence is input into the encoder layer for encoding, including: calculating the attention distribution of the low-dimensional feature sequence to obtain context features; performing nonlinear mapping on the context features to obtain the encoded low-dimensional feature sequence.

[0060] In the disclosed embodiment, the encoder layer calculates the attention distribution of the low-dimensional feature sequence through a multi-head self-attention mechanism. The model can simultaneously focus on different parts of the sequence and capture the dependencies between different positions, thereby obtaining rich contextual features. The contextual features are nonlinearly mapped through the position feedforward layer, which enhances the expressive power of the model and enables the model to learn more complex feature representations. The low-dimensional feature sequence processed by the encoder layer not only contains the spatiotemporal information of the original image sequence, but also incorporates the contextual information of the sequence, providing a more accurate and comprehensive feature representation for the subsequent decoding process.

[0061] In this way, the model's ability to capture lip movement details is greatly improved, allowing the model to more accurately predict the pronunciation corresponding to the lip movement, thereby improving the accuracy of the digital population lip matching assessment. Especially when dealing with complex Chinese pronunciation, it does not rely solely on a single syllable or visual feature, but integrates multiple information in the sequence, and can better handle the problems of polyphones and homophones.

[0062] Optionally, the method for evaluating lip matching of Chinese digital speakers based on lip reading recognition further includes: before adding coding to the low-dimensional feature sequence, performing residual connection and normalization operations on the low-dimensional feature sequence.

[0063] In the disclosed embodiment, before the low-dimensional feature sequence is input into the encoder layer, the input feature sequence is first added to the processed feature sequence through a residual connection structure, which can alleviate the gradient vanishing problem in deep network training and enable the network to learn deep features more effectively.

[0064] Perform normalization operations, usually layer normalization, to independently standardize each sample in the feature sequence to reduce internal covariate shift, accelerate model convergence, and improve the generalization ability of the model.

[0065] By performing residual connections and normalization operations before adding encoding to the low-dimensional feature sequence, firstly, the model's ability to learn deep features is enhanced, so that the model can more effectively learn deep, abstract feature representations when facing complex lip reading recognition tasks. Secondly, the normalization operation improves the stability and efficiency of model training, reduces training time, and helps the model maintain good performance when facing new, unseen data.

[0066] Optionally, according to the RNN language model, the token sequences of multiple paths are decoded through beam search to obtain predicted text corresponding to the image sequence, including: according to the context of the current path, the token sequences of multiple paths are input into the RNN language model for scoring to obtain probability scores; the decoder score and the CTC score are obtained; according to the decoder score, the CTC score and the probability score, the comprehensive scores of the multiple paths are calculated respectively; the token sequence of the path with the highest comprehensive score is converted into a text string to obtain the predicted text corresponding to the image sequence.

[0067] In the disclosed embodiment, first, according to the context information of the current path, the token sequence of each path is input into the RNN language model for scoring to obtain a probability score for each path, and the score reflects the model's preference for the current path. Obtain the decoder score and CTC score. The decoder score reflects the model's decoding ability for the token sequence, while the CTC score provides the alignment information of the sequence in the time dimension. Then, the decoder score, CTC score and probability score obtained by the RNN language model are combined, and the comprehensive score of each path is calculated by weighted summation. The comprehensive score takes into account decoding accuracy, time alignment and language model preference. Finally, select the path with the highest comprehensive score, convert the path's token sequence into a text string, and obtain the predicted text corresponding to the image sequence.

[0068] The probability score of the RNN language model enables the model to prefer paths that are more linguistically reasonable and smoother. By integrating the probability score of the RNN language model, the model can more accurately capture the correspondence between lip movements and pronunciation, especially in a language environment such as Chinese where the complexity of speech and visual information is high. This method not only improves the accuracy of the digital human lip matching assessment, but also enhances the model's adaptability to complex language environments, allowing digital humans to provide a more natural and realistic interactive experience in a variety of application scenarios.

[0069] Optionally, the Chinese syllable matching degree of the real text and the predicted text is calculated within the sliding window, including: converting the real text and the predicted text into pinyin; calculating the matching degree of each word of the real text and the predicted text according to multiple index numbers of the predicted text pinyin within the sliding window; and calculating the average matching degree of all words as the evaluation result of lip matching.

[0070] Combination Figure 4 As shown, the embodiment of the present disclosure provides another method for evaluating the lip matching degree of Chinese digital population based on lip reading recognition, comprising:

[0071] S401, the back-end processor obtains an image sequence of the lip area in the original video and the real text corresponding to the original video.

[0072] S402, the back-end processor performs text inference on the image sequence of the lip area according to the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain predicted text corresponding to the image sequence.

[0073] S403, the backend processor converts the real text and the predicted text into pinyin.

[0074] S404, the backend processor calculates the matching degree of each word of the real text and the predicted text according to the multiple index numbers of the predicted text pinyin in the sliding window.

[0075] S405: The backend processor calculates the average value of the matching degree of all words as the evaluation result of the lip matching degree.

[0076] In the disclosed embodiment, because there are polyphones and homophones in Chinese, directly comparing the text may not accurately reflect the matching degree of pronunciation. Converting both the real text and the predicted text into corresponding pinyin forms can unify the evaluation criteria. Calculating the index number of the pinyin of each word in the sliding window can solve the problem of voice alignment. The length of the predicted text and the real text may be different. The matching range can be dynamically adjusted through the sliding window. Finally, the average value of the matching degree of all words is calculated. The average value is the evaluation result of the lip matching degree, which is used to reflect the matching degree of the predicted text and the real text in pronunciation.

[0077] The disclosed embodiments can quantify the similarity between the predicted text and the real text at the pronunciation level. By converting the text into pinyin, the problem of polyphones and homophones unique to Chinese is effectively solved, and the accuracy of the evaluation is improved. The use of sliding windows enables the evaluation to adapt to texts of different lengths, enhancing the flexibility and robustness of the evaluation method.

[0078] Optionally, the matching degree of each word between the real text and the predicted text is calculated based on multiple index numbers of the predicted text pinyin in the sliding window, including: based on the multiple index numbers of the predicted text pinyin in the sliding window, the predicted text pinyin corresponding to the index numbers are converted into the real text pinyin in turn to obtain multiple editing times; the minimum editing number is used as the matching degree of a single word between the real text and the predicted text; and the matching degree of each word between the real text and the predicted text is calculated respectively.

[0079] In the disclosed embodiment, first, based on the pinyin of the predicted text, multiple possible serial numbers are determined in the sliding window, and the index serial number corresponds to the pinyin position that may be matched in the real text. Then, for each index serial number, the predicted text pinyin is converted into the real text pinyin, and multiple possible editing times are obtained by calculating the minimum number of editing operations (including addition, deletion, and replacement operations) required to convert the predicted text pinyin into the real text pinyin. The minimum number of editing times is selected as a quantitative indicator of the matching degree of a single word of the real text and the predicted text. Finally, the above process is repeated for each word of the real text and the predicted text, and the matching degree is calculated respectively.

[0080] In this way, the similarity between the predicted text and the real text at the pronunciation level can be effectively evaluated, especially in a language environment such as Chinese where the complexity of speech and visual information is high. By considering multiple possible index numbers in the sliding window, it can adapt to the situation where the length of the predicted text and the real text are inconsistent, and improve the flexibility and accuracy of the matching. Selecting the minimum number of edits as a quantitative indicator of the matching degree directly reflects the minimum change required to convert the predicted pinyin into the real pinyin, making the evaluation result more objective and accurate.

[0081] Optionally, multiple index numbers of the predicted text pinyin in the sliding window are calculated as follows: the index number of the real text is multiplied by the displacement coefficient; wherein the displacement coefficient is the ratio of the real text to the predicted text; and the multiplication result is added to multiple sliding window values ​​to obtain multiple index numbers of the predicted text.

[0082] In the disclosed embodiment, the ratio reflects the relationship between the lengths of the two texts and is used to adjust the correspondence between the predicted text pinyin and the real text pinyin. Each index number of the real text is multiplied by the displacement coefficient to obtain the adjusted index number, which can ensure the accuracy of the pinyin matching.

[0083] Optionally, an image sequence of the lip area in the original video is obtained, including: decomposing the original video into multiple image frames and transcoding them; recording the lip position in each image frame according to the key point coordinates of the lips; cropping the lip position in the image frame and performing noise reduction processing as the input of the AVSR model; and using the output of the AVSR model as the image sequence of the lip area in the original video.

[0084] Combination Figure 5 As shown, the embodiment of the present disclosure provides a method for evaluating the lip matching degree of a Chinese digital person based on lip reading recognition, comprising:

[0085] S501, the back-end processor decomposes the original video into multiple image frames and performs transcoding processing.

[0086] S502: The backend processor records the lip position in each image frame according to the key point coordinates of the lips.

[0087] S503: The back-end processor crops the lip position in the image frame and performs noise reduction processing as input of the AVSR model.

[0088] S504, the back-end processor uses the output of the AVSR model as an image sequence of the lip region in the original video.

[0089] S505, the backend processor obtains the real text corresponding to the original video.

[0090] S506, the back-end processor performs text inference on the image sequence of the lip area according to the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain predicted text corresponding to the image sequence.

[0091] S507, the back-end processor calculates the Chinese syllable matching degree of the real text and the predicted text in the sliding window to obtain the evaluation result of the lip matching degree.

[0092] In the disclosed embodiment, the video frame decomposition and transcoding process is first performed to decompose the continuous video stream into discrete image frames, and the video format is unified through transcoding to ensure the consistency and compatibility of subsequent processing. The key point detection technology of the face is used to identify and record the key point coordinates of the lips in each image frame to locate the lip area, thereby ensuring the accurate extraction of the lip area. Then, according to the key point coordinates, the lip position in the image frame is cropped to extract the lip area, providing accurate input for subsequent lip reading recognition. The cropped lip image may have noise due to factors such as video quality and lighting conditions, so a noise reduction process is performed to improve the image quality by reducing image noise, thereby providing a clearer lip image input for the AVSR model. Finally, the cropped and denoised image sequence is input into the AVSR model, and the model analyzes and processes the lip image sequence, and the output result is used as the image sequence of the lip area in the original video.

[0093] The disclosed embodiment can accurately and efficiently extract the image sequence of the lip area from the original video, providing high-quality input data for lip reading recognition. Through transcoding and key point detection, the accurate positioning and extraction of the lip area are ensured, while cropping and noise reduction processing further improve the quality of the image, allowing the AVSR model to more accurately recognize and analyze lip movements. Not only does it improve the accuracy of lip reading recognition, but it also enhances the model's adaptability to complex video environments, allowing digital humans to provide a more natural and realistic interactive experience in a variety of application scenarios.

[0094] Combination Figure 6 As shown, the embodiment of the present disclosure provides a Chinese digital population lip matching evaluation device 60 based on lip reading recognition, including an input preprocessing module 601, a model reasoning module 602 and an evaluation calculation module 603. The input preprocessing module 601 is configured to obtain an image sequence of the lip area in the original video and a real text corresponding to the original video. The model reasoning module 602 is configured to perform text reasoning on the image sequence of the lip area according to the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain a predicted text corresponding to the image sequence. The evaluation calculation module 603 is configured to perform Chinese syllable matching calculation on the real text and the predicted text in a sliding window to obtain an evaluation result of the lip matching.

[0095] The Chinese digital population lip matching evaluation device 60 based on lip reading recognition provided by the embodiment of the present disclosure is used to achieve efficient evaluation of the lip matching of the Chinese digital population. First, the image sequence of the lip area and the corresponding real text are obtained from the original video, and the audio and video speech recognition (AVSR) model and the recurrent neural network (RNN) language model are used to perform text reasoning on the image sequence to generate predicted text. Secondly, by converting both the real text and the predicted text into pinyin, and performing matching calculation in the sliding window to reduce the speech alignment error, the evaluation result of the lip matching is finally obtained. This process not only improves the accuracy of the evaluation, but also solves the complexity problem caused by polyphones and homophones in the Chinese environment. The embodiment of the present disclosure can convert the visual indicators that require subjective judgment in the lip recognition of the digital population into quantifiable objective evaluation indicators, which is particularly suitable for the Chinese environment, and can more accurately and objectively evaluate the lip matching, providing important technical support for the development of the digital human industry.

[0096] Combination Figure 7 As shown, the embodiment of the present disclosure provides a device 70 for evaluating the lip matching degree of a Chinese digital population based on lip reading recognition, including a processor (processor) 700 and a memory (memory) 701. Optionally, the device 70 may also include a communication interface (Communication Interface) 702 and a bus 703. Among them, the processor 700, the communication interface 702, and the memory 701 can communicate with each other through the bus 703. The communication interface 702 can be used for information transmission. The processor 700 can call the logic instructions in the memory 701 to execute the Chinese digital population lip matching degree evaluation method based on lip reading recognition of the above embodiment.

[0097] In addition, the logic instructions in the memory 701 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0098] The memory 701 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 700 executes the function application and data processing by running the program instructions / modules stored in the memory 701, that is, the Chinese digital population lip matching evaluation method based on lip reading recognition in the above embodiment is implemented.

[0099] The memory 701 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 701 may include a high-speed random access memory and may also include a non-volatile memory.

[0100] The disclosed embodiments provide a lip reading recognition device, including: a lip reading recognition device body, and the above-mentioned Chinese digital population lip matching evaluation device based on lip reading recognition. The Chinese digital population lip matching evaluation device based on lip reading recognition is installed on the lip reading recognition device body. The installation relationship described here is not limited to the internal placement of the lip reading recognition device body, but also includes the installation connection with other components of the lip reading recognition device, including but not limited to physical connection, electrical connection or signal transmission connection, etc. Those skilled in the art can understand that the Chinese digital population lip matching evaluation device based on lip reading recognition can be adapted to a feasible lip reading recognition device body, thereby realizing other feasible embodiments.

[0101] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0102] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0103] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0104] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

Claims

1. A method for evaluating the lip matching degree of Chinese digital population based on lip reading recognition, characterized in that: include: Obtain an image sequence of the lip region in the original video and a ground-truth text corresponding to the original video; Perform text inference on the image sequence of the lip area based on the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain the predicted text corresponding to the image sequence; The Chinese syllable matching degree of the real text and the predicted text is calculated within the sliding window to obtain the evaluation result of the lip matching degree.

2. The method according to claim 1, characterized in that According to the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model, text inference is performed on the image sequence of the lip area to obtain the predicted text corresponding to the image sequence, including: The AVSR model is used to encode the image sequence to extract the spatiotemporal features of the image sequence as a token sequence; According to the RNN language model, the token sequences of multiple paths are decoded through beam search to obtain the predicted text corresponding to the image sequence.

3. The method according to claim 2, characterized in that The AVSR model is used to encode the image sequence, including: Feature extraction of image sequences is performed through the residual network of the AVSR model; Convert the image sequence after feature extraction into a low-dimensional feature sequence, and add encoding to the low-dimensional feature sequence; The encoded low-dimensional feature sequence is input into the encoder layer for encoding.

4. The method according to claim 3, characterized in that The encoded low-dimensional feature sequence is input into the encoder layer for encoding, including: Calculate the attention distribution of low-dimensional feature sequences to obtain contextual features; The context features are nonlinearly mapped to obtain a low-dimensional feature sequence after adding encoding.

5. The method according to claim 3, characterized in that: Also includes: Before adding the encoding to the low-dimensional feature sequence, the low-dimensional feature sequence is subjected to residual connection and normalization operations.

6. The method according to claim 2, characterized in that According to the RNN language model, the token sequences of multiple paths are decoded through beam search to obtain the predicted text corresponding to the image sequence, including: According to the context of the current path, the token sequences of multiple paths are input into the RNN language model for scoring to obtain probability scores. Get the decoder score and the connection temporal classification CTC score; According to the decoder score, CTC connection timing classification score and probability score, the comprehensive scores of multiple paths are calculated respectively; The token sequence of the path with the highest comprehensive score is converted into a text string to obtain the predicted text corresponding to the image sequence.

7. The method according to claim 1, characterized in that Calculate the Chinese syllable matching degree of the real text and the predicted text in the sliding window, including: Convert real text and predicted text into pinyin; According to multiple index numbers of the predicted text pinyin in the sliding window, the matching degree of each word of the real text and the predicted text is calculated; The average matching degree of all words is calculated as the evaluation result of lip matching.

8. The method according to claim 7, characterized in that According to the multiple index numbers of the predicted text pinyin in the sliding window, the matching degree of each word between the real text and the predicted text is calculated, including: According to multiple index numbers of the predicted text pinyin in the sliding window, the predicted text pinyin corresponding to the index numbers is converted into the real text pinyin in sequence to obtain multiple editing times; The minimum number of edits is taken as the matching degree of a single word between the real text and the predicted text; Calculate the matching degree of each word of the real text and the predicted text respectively.

9. The method according to claim 7, characterized in that: The multiple index numbers of the predicted text pinyin in the sliding window are calculated as follows: Multiply the index number of the real text by the displacement coefficient; where the displacement coefficient is the ratio of the real text to the predicted text; The multiplication result is added to multiple sliding window values ​​to obtain multiple index numbers of the predicted text.

10. The method according to any one of claims 1 to 9, characterized in that: Get the image sequence of the lip area in the original video, including: Decompose the original video into multiple image frames and perform transcoding; According to the key point coordinates of the lips, the lip position in each image frame is recorded; The lip position in the image frame is cropped and denoised as the input of the AVSR model; The output of the AVSR model is taken as an image sequence of the lip region in the original video.

11. A device for evaluating the lip matching degree of Chinese digital population based on lip reading recognition, characterized in that: include: An input preprocessing module is configured to obtain an image sequence of a lip region in an original video and a ground truth text corresponding to the original video; A model inference module is configured to perform text inference on the image sequence of the lip region according to the audio and video speech recognition AVSR model and the recurrent neural network RNN ​​language model to obtain a predicted text corresponding to the image sequence; The evaluation calculation module is configured to calculate the Chinese syllable matching degree of the real text and the predicted text in the sliding window to obtain the evaluation result of the lip matching degree.

12. A device for evaluating the lip matching degree of Chinese digital people based on lip reading recognition, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the method for evaluating lip matching of Chinese digital persons based on lip reading recognition as described in any one of claims 1 to 10 when running the program instructions.

13. A lip reading recognition device, characterized in that: include: Lip reading recognition device body; The Chinese digital population lip matching evaluation device based on lip reading recognition as described in claim 11 or 12 is installed on the lip reading recognition device body.

Images