Lip language recognition system and method based on improved TSM-ResNet50 space-time network model

Abstract

The invention relates to the technical field of artificial intelligence, computer vision and voice processing, in particular to a lip language recognition system and method based on an improved TSM-ResNet50 space-time network model for the voice recognition requirements of deaf-mute people and in a noisy environment. According to the system, efficient and accurate recognition of lip language video data is achieved by fusing deep learning technologies such as a YOLOv7 target detection algorithm, a TSM-ResNet50 space-time network model, a GRU network and an attention mechanism, visual display of lip language recognition is achieved by adopting a B / S framework and a lightweight Flask framework, and user experience is greatly improved. The invention aims to provide a more convenient and efficient communication mode for the deaf-mute, and promote the development and application of the lip language recognition technology.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, computer vision and speech processing technology, and in particular to a lip reading recognition system and method based on an improved TSM-ResNet50 spatiotemporal network model for speech recognition needs of deaf-mute people and in noisy environments. Background Art

[0002] As a large special group in society, the communication barriers of the deaf-mute people have always been the focus of social attention. This group faces many difficulties in learning, employment, medical treatment, etc., and often feels lonely and anxious due to communication barriers. Although traditional hearing aids and implants provide certain hearing assistance for the deaf-mute people, their high cost, frequent maintenance requirements, and electronic waste generated during use not only increase the economic burden of users, but also put considerable pressure on the environment.

[0003] In order to solve the communication difficulties of the deaf and mute people, lip reading recognition technology came into being. Lip reading recognition technology captures the speaker's lip movements in real time and converts them into text information, allowing the deaf and mute people to intuitively see what others are saying, thereby effectively alleviating their difficulties in daily interpersonal communication. However, traditional lip reading recognition technology often fails to fully consider the time and space information in the lip reading video, resulting in pain points such as poor real-time performance, low accuracy, and poor visualization of lip reading recognition.

[0004] Specifically, lip reading recognition technology needs to accurately extract the characteristics of lip movement from video data, which requires not only paying attention to the lip shape and configuration in a single video frame (spatial information), but also understanding the movement and changes of the lips over time (temporal information). Traditional lip reading recognition methods often focus on the processing of single-frame images, ignoring the temporal continuity of lip movement, which affects the accuracy and real-time performance of recognition.

[0005] In addition, with the rapid development of artificial intelligence, computer vision and deep learning technology, new solutions have been provided for lip reading recognition technology. In particular, the advantages of deep learning technology in feature extraction and time series modeling make it possible to build an efficient and accurate lip reading recognition system. However, how to effectively apply these advanced technologies to the field of lip reading recognition is still a problem that needs to be solved. Summary of the invention

[0006] In view of the problems and challenges existing in the above-mentioned background technology, the present invention proposes a lip-reading recognition system based on an improved TSM-ResNet50 spatio-temporal network model. By integrating deep learning technologies such as the YOLOv7 object detection algorithm, the TSM-ResNet50 spatio-temporal network model, the GRU network, and the attention mechanism, the system realizes efficient and accurate recognition of lip-reading video data. It adopts a B / S architecture and a lightweight Flask framework to achieve visual display of lip-reading recognition, greatly improving the user experience. The purpose of the present invention is to provide a more convenient and efficient communication method for deaf-mute people, and at the same time promote the development and application of lip-reading recognition technology.

[0007] The technical solution adopted by the present invention to solve the technical problems is as follows: A lip-reading recognition system based on an improved TSM-ResNet50 spatio-temporal network model, including:

[0008] A video preprocessing module for frame cutting, lip detection, and localization of the input video data;

[0009] A feature extraction module that uses an improved TSM-ResNet50 spatio-temporal network model. The improved TSM-ResNet50 spatio-temporal network model includes a time shift module and a ResNet-50 architecture of the residual network, and is used to extract spatio-temporal features of lip movement;

[0010] A recognition module that performs lip-reading recognition based on the extracted spatio-temporal features and outputs the recognition result;

[0011] A visualization module for displaying the recognition result in the form of real-time text subtitles on the Web front-end page.

[0012] The video preprocessing module uses the YOLOv7 algorithm for lip detection and localization.

[0013] The improved TSM-ResNet50 spatio-temporal network model also introduces a gated recurrent unit network and an attention mechanism to enhance the network's long-term temporal modeling ability and key feature extraction ability.

[0014] It also includes a real-time speech recognition module for converting the speech of healthy people into text information in real time and making an auxiliary comparison with the text generated by lip-reading recognition.

[0015] Adopting a B / S architecture, using a lightweight Flask framework to realize the visualization of lip-reading recognition.

[0016] An operation method of a lip-reading recognition system based on an improved TSM-ResNet50 spatio-temporal network model includes the following steps:

[0017] S1. Preprocess the input video data, and use the YOLOv7 algorithm for lip detection and localization;

[0018] S2. Input the preprocessed video data into the improved TSM-ResNet50 spatio-temporal network model to extract the spatio-temporal features of lip movement;

[0019] S3. Based on the extracted spatio-temporal features, perform lip reading recognition through the recognition module and output the recognition result;

[0020] S4. Display the recognition result in the form of real-time text subtitles on the Web front-end page.

[0021] When the improved TSM-ResNet50 spatio-temporal network model extracts spatio-temporal features, it combines the time shift module and the ResNet-50 architecture of the residual network, and introduces the gated recurrent unit network and the attention mechanism.

[0022] An operation method of a lip reading recognition system based on the improved TSM-ResNet50 spatio-temporal network model further includes the step of: converting the speech of healthy people into text information in real time and making an auxiliary comparison with the text generated by lip reading recognition to improve the feasibility and practicality of recognition.

[0023] The visualization of the lip reading recognition system is realized through the B / S architecture and the lightweight Flask framework.

[0024] An operation method of a lip reading recognition system based on the improved TSM-ResNet50 spatio-temporal network model further includes the step of: the user submits registration and login requests through the front end, and the system verifies the information through the MySQL database. After successful verification, the user can perform lip reading recognition operations.

[0025] The beneficial effects of the present invention are: the lip reading recognition system and its operation method based on the improved TSM-ResNet50 spatio-temporal network model proposed by the present invention have significant beneficial effects, which are specifically reflected in the following aspects:

[0026] Improve the lip reading recognition accuracy: By combining the time shift module (TSM) and the ResNet-50 architecture of the residual network, the TSM-ResNet50 spatio-temporal network model designed by the present invention can utilize both the spatial and temporal information of video data, significantly improving the lip reading recognition accuracy. Experimental results show that compared with the commonly used visual feature extraction network, the recognition accuracy of the improved TSM-ResNet50 model is increased by at least 2.124% and at most 25.491%.

[0027] Improving the real-time performance of lip language recognition: By using the YOLOv7 algorithm for lip detection and leveraging its lightweight design and high inference speed, the present invention achieves fast and accurate lip localization. Meanwhile, without increasing the computational cost and the number of parameters, the TSM-ResNet50 model enhances the learning ability of temporal features through the temporal shift module, further improving the real-time performance of lip language recognition.

[0028] Realizing the visualization of lip language recognition: The present invention adopts the B / S architecture and the lightweight Flask framework, and displays the lip language recognition results in the form of real-time text captions on the Web front-end page, realizing the visualization of lip language recognition. This not only enhances the user experience but also makes the lip language recognition technology more intuitive and easy to use.

[0029] Reducing the usage cost and environmental burden: Compared with traditional hearing aids and implant devices, the lip language recognition system provided by the present invention does not require high hardware costs and frequent maintenance, and does not generate a large amount of electronic waste during use, thus reducing the economic burden on users and the pressure on the environment.

[0030] Improving the quality of life of deaf-mute people: Through real-time and accurate lip language recognition, the present invention provides a more convenient and efficient communication method for deaf-mute people, significantly improving their social skills and self-confidence, promoting integration into society, and thus improving their quality of life.

[0031] Assisting special education and classroom learning: In the field of special education, the present invention can assist hearing-impaired students in understanding and participating in classroom activities, improving learning effects. By converting the teacher's lip movements into text or speech in real time, students can obtain the teaching content more clearly, reduce information omission, and improve learning efficiency and grades.

[0032] Enhancing the speech recognition ability in noisy environments: In noisy environments, lip language recognition technology is not affected by noise interference and can accurately recognize and analyze lip movement information. By combining lip language recognition and speech recognition technologies, the present invention improves the recognition accuracy in complex environments and provides users with a more stable speech interaction experience.

[0033] Expanding application fields and scenarios: The lip language recognition technology provided by the present invention is not only applicable to the communication needs of deaf-mute people, but also shows broad application prospects in multiple fields such as public security monitoring and multimedia game interaction, bringing substantial benefits to people in different fields.

[0034] Improving the reliability and practicality of the system: By introducing deep learning technologies such as the gated recurrent unit (GRU) network and the attention mechanism, the present invention further improves the long-term temporal modeling ability and key feature extraction ability of the lip language recognition system, enhancing the reliability and practicality of the system.

[0035] Promote technology development and application: The proposal of this invention has promoted the development and application of lip-reading recognition technology, provided new ideas and methods for research in related fields, and contributed to improving the technical level and innovation ability of the entire industry. Brief Description of the Drawings

[0036] Figure 1 is a schematic diagram of the overall system architecture of the present invention;

[0037] Figure 2 is the TSM time displacement module in the embodiment;

[0038] Figure 3 is the ResNet-50 network structure diagram in the embodiment;

[0039] Figure 4 is the improved TSM-ResNet50 spatio-temporal network structure diagram in the embodiment;

[0040] Figure 5 is the system recognition flow chart in the embodiment;

[0041] Figure 6 is the comparison table of recognition accuracies of different feature extraction networks in Embodiment 1;

[0042] Figure 7 is the performance comparison of object detection algorithms in Embodiment 1. Detailed Implementation Modes

[0043] Embodiment 1

[0044] This embodiment details the specific implementation modes of a lip-reading recognition system based on an improved TSM-ResNet50 spatio-temporal network model, including the overall architecture of the system, the functional implementation of each module, the data processing flow, and the operation method of the system, etc.

[0045] As Figure 1-7 shown, the overall architecture of this system adopts a B / S architecture, mainly consisting of three parts: the front end, the back end, and the database. The front end is responsible for user interaction and result display, the back end is responsible for data processing and algorithm implementation, and the database is used to store user information and recognition results.

[0046] The front-end module has a user interaction interface, provides user login and registration interfaces, and users submit usernames and passwords through HTML forms for login or registration. It provides video upload and real-time video capture functions, and users can choose to upload local videos or turn on the camera for real-time video capture. It displays the lip-reading recognition results, and shows the recognition results in the form of real-time text subtitles on the Web front-end page. The front-end module also realizes data interaction, and conducts data interaction with the back end through AJAX or Fetch API, sending video data and receiving recognition results, etc.

[0047] The backend module described above includes a video preprocessing module, a feature extraction module, an identification module, a real-time speech recognition module, and a result output module. Among them, the video preprocessing module uses the YOLOv7 algorithm to quickly cut frames, detect and locate lips in the input video data. The YOLOv7 algorithm extracts features from video frames through a convolutional neural network, and uses a pre-trained model to quickly and accurately detect the positions of the face and lips. The detected lip positions are cropped to ensure that the lips are located in the center of the image, facilitating subsequent feature extraction and identification.

[0048] The feature extraction module described above uses an improved TSM-ResNet50 spatio-temporal network model to extract features from the preprocessed video data. The improved TSM-ResNet50 spatio-temporal network model includes a Time Shift Module (TSM) and a ResNet-50 architecture. The TSM module realizes the exchange of temporal information between video frames by slightly shifting the outputs of some channels along the temporal dimension in specific network layers, enhancing the network's ability to learn temporal features. The ResNet-50 architecture solves the problems of gradient vanishing and gradient explosion in the training of deep networks by introducing the concept of "residual learning", improving the training efficiency and accuracy of the network. A Gated Recurrent Unit (GRU) network and an attention mechanism are introduced to further enhance the network's long-term temporal modeling ability and key feature extraction ability. The GRU network can effectively capture dependencies in long sequences, and the attention mechanism allows the model to dynamically focus on the most important parts when processing sequence data.

[0049] The identification module described above performs lip language recognition based on the extracted spatio-temporal features through a fully connected layer or a classifier, and outputs the recognition results. The recognition results include the recognized words or sentences, as well as corresponding confidence information, etc. The real-time speech recognition module described above uses a speech recognition algorithm (such as the Google Speech-to-Text API or a self-trained speech recognition model) to convert the speech of healthy people into text information in real time. The speech recognition results are used for auxiliary comparison with the lip language recognition results to improve the feasibility and practicality of recognition.

[0050] The result output module described above outputs the lip language recognition results to the front end in JSON format, and the front end displays the results on the Web page through HTML and JS. At the same time, the recognition results are stored in the MySQL database for subsequent query and analysis.

[0051] The database module described above stores user registration information, login information, recognition results, etc. It supports operations such as adding, deleting, modifying, and querying user information, as well as querying and statistics of recognition results.

[0052] The operation method and process of this embodiment are specifically as follows:

[0053] S1. Video Input: The user uploads a local video or turns on the camera for real-time video capture, and the video data is sent to the backend.

[0054] S2. Video Preprocessing: The backend video preprocessing module uses the YOLOv7 algorithm to perform frame cutting, lip detection, and localization on the video data, and crops out the lip image.

[0055] S3. Feature Extraction: The lip image is input into an improved TSM-ResNet50 spatio-temporal network model for feature extraction to extract the spatio-temporal features of lip movement.

[0056] S4. Lip Reading Recognition: Based on the extracted spatio-temporal features, lip reading recognition is performed through the recognition module, and the recognition result is output.

[0057] S5. Result Output and Display: The recognition result is displayed on the Web front-end page in the form of real-time text subtitles, and optionally, the result is stored in the database.

[0058] S6. Real-time Speech Recognition: For healthy people, the system simultaneously collects speech data, converts the speech into text information in real time through the real-time speech recognition module, and makes an auxiliary comparison with the lip reading recognition result.

[0059] In this embodiment, the specific operation method of the system is as follows:

[0060] User Registration and Login: The user submits a registration or login request through the front end and fills in the username and password. The front end sends the request to the backend, and the backend verifies the user information through the MySQL database. After successful verification, the user jumps to the main function interface.

[0061] Video Upload and Real-time Capture: The user selects to upload a local video or turns on the camera for real-time video capture. The front end sends the video data to the backend for processing.

[0062] Video Preprocessing: The backend video preprocessing module uses the YOLOv7 algorithm to perform frame cutting, lip detection, and localization on the video data. The cropped lip image is sent to the feature extraction module. Feature Extraction and Lip Reading Recognition: The feature extraction module uses an improved TSM-ResNet50 spatio-temporal network model to extract features from the lip image. The recognition module performs lip reading recognition based on the extracted spatio-temporal features and outputs the recognition result.

[0063] Result Display and Storage: The recognition result is displayed on the Web front-end page in the form of real-time text subtitles. Optionally, the recognition result is stored in the MySQL database.

[0064] Real-time speech recognition (optional): For healthy people, the system simultaneously collects speech data. The speech is real-time converted into text information through the real-time speech recognition module. The speech recognition result is used for auxiliary comparison with the lip-reading recognition result to improve the feasibility and practicality of recognition.

[0065] In this embodiment, the front end communicates with the back end through the HTTP protocol, and uses AJAX or Fetch API for data interaction. Each module in the back end is interconnected and controlled through function calls and data transmission to ensure the coherence and accuracy of data processing. The database module interacts with the back end through SQL statements, supporting the storage and query of user information, as well as the storage and statistics of recognition results. This embodiment details the specific implementation of the lip-reading recognition system based on the improved TSM-ResNet50 spatio-temporal network model, including the overall architecture of the system, the function implementation of each module, the data processing flow, and the operation method of the system, etc. By introducing the gated recurrent unit network and the attention mechanism, the improved TSM-ResNet50 spatio-temporal network model significantly improves the accuracy and real-time performance of lip-reading recognition. The system adopts the B / S architecture and the lightweight Flask framework, realizes the visualization of lip-reading recognition, and provides additional functions such as real-time speech recognition and user registration and login, providing users with a convenient and efficient communication method.

[0066] Embodiment 2

[0067] This embodiment is optimized based on Embodiment 1. The optimization content mainly includes: introducing more advanced deep learning technologies to enhance the model performance, optimizing the data processing flow to improve real-time performance and accuracy, and enhancing the scalability and integration ability of the system to meet wider application requirements; mainly including the following aspects:

[0068] Modular design: The system is divided into finer-grained modules, including video preprocessing module, feature extraction module, recognition module, visualization module, real-time speech recognition module, and user management module, etc., for the convenience of system maintenance and upgrade.

[0069] Microservices architecture: Adopting the microservices architecture, each module is independently deployed and run to improve the scalability and fault tolerance of the system.

[0070] Cloud deployment: The system is deployed in the cloud, leveraging the elastic computing and storage resources of the cloud platform to improve the performance and availability of the system.

[0071] In the optimization of the video preprocessing module, multi-scale feature fusion is introduced. Based on the YOLOv7 algorithm, multi-scale feature fusion technology is introduced to improve the accuracy and robustness of lip detection. By fusing feature maps of different scales, richer lip detail information can be captured, improving the detection accuracy.

[0072] Improve real-time performance, further optimize the YOLOv7 algorithm, reduce unnecessary computational complexity, and increase the inference speed. For example, reduce model parameters and computational complexity through techniques such as pruning and quantization, while maintaining high detection accuracy.

[0073] Introduce the Transformer architecture. Based on the improved TSM-ResNet50 spatio-temporal network model, introduce the Transformer architecture to enhance the model's global feature extraction ability. The Transformer architecture can capture long-range dependencies between video frames through self-attention mechanisms, further improving the accuracy of lip reading recognition.

[0074] Achieve multimodal fusion. Combine audio information (if any) in the video data to further improve the accuracy of lip reading recognition through multimodal fusion techniques. Audio information can serve as auxiliary information for lip movement, helping the model better understand the speaker's intention.

[0075] Introduce a language model. Introduce a language model (such as BERT, GPT, etc.) in the recognition module to post-process the recognition results and improve the fluency and accuracy of the recognition results. The language model can capture syntactic and semantic information in natural language, correcting and optimizing the recognition results.

[0076] Dynamically adjust the recognition strategy: Dynamically adjust the recognition strategy according to the quality and complexity of the input video. For example, in the case of poor video quality or complex lip movement, use a more complex model or increase computational complexity to improve recognition accuracy; in the case of good video quality or simple lip movement, use a simpler model or reduce computational complexity to improve real-time performance.

[0077] Enhance the visualization effect, optimize the front-end page design, and improve the intuitiveness and usability of the visualization effect. For example, make the recognition results more prominent and easy to understand by adding animation effects, adjusting font sizes and colors, etc.

[0078] Adaptation for multiple terminals. By optimizing the front-end code, enable the system to run well on different terminals (such as PCs, mobile phones, tablets, etc.), improving the popularity and user experience of the system.

[0079] Introduce noise suppression technology. Introduce noise suppression technology in the real-time speech recognition module to reduce the interference of environmental noise on the speech recognition results. For example, improve the signal-to-noise ratio of the speech signal through techniques such as adaptive filtering and beamforming. Add a multilingual support module to expand the functions of the real-time speech recognition module, supporting speech recognition and conversion in multiple languages to meet the needs of different users.

[0080] This embodiment adopts more secure encryption technologies and authentication mechanisms to protect user information and system security. For example, the HTTPS protocol is used for data transmission, and multi-factor authentication technology is used for user login verification. A user personalization setting function is provided, such as adjusting the recognition speed, recognition accuracy, interface layout, etc., to meet the usage habits and needs of different users. This embodiment can optimize the data processing flow, perform parallel processing on modules such as video preprocessing, feature extraction, and recognition, and improve the real-time performance of the system. For example, multiple models or algorithms are run in parallel on the GPU to reduce the data processing time. A caching mechanism is introduced to cache frequently accessed data, reducing the number of database accesses and the amount of computation. For example, commonly used lip detection models and feature extraction models are cached to improve the system's response speed. Incremental learning technology is adopted to continuously train and optimize the model. By continuously collecting new lip-reading video data, the model is updated and iterated to improve the adaptability and accuracy of the model.

[0081] This embodiment provides rich API interfaces to facilitate other systems or applications to integrate the lip-reading recognition function of this system. The API interfaces adopt the RESTful style and support multiple data formats (such as JSON, XML, etc.), improving the scalability and integration ability of the system. Message queue technologies (such as RabbitMQ, Kafka, etc.) are introduced to achieve asynchronous communication and data exchange between the front end and the back end. Through the message queue, the dependency relationship between the front end and the back end can be decoupled, improving the flexibility and scalability of the system. A log monitoring system is established to monitor and record the running status and error information of the system in real time. Through the log monitoring system, problems can be discovered and solved in a timely manner, improving the stability and reliability of the system.

[0082] Based on Embodiment 1, this embodiment makes the optimized system have higher accuracy, real-time performance, and scalability and can meet a wider range of application requirements by introducing more advanced deep learning technologies, optimizing the data processing flow, and enhancing the scalability and integration ability of the system.

Claims

1. A lip reading recognition system based on an improved TSM-ResNet50 spatiotemporal network model, characterized by: include: Video preprocessing module, used for frame cutting, lip detection and positioning of input video data; A feature extraction module, which uses an improved TSM-ResNet50 spatiotemporal network model, wherein the improved TSM-ResNet50 spatiotemporal network model includes a time shift module and a residual network ResNet-50 architecture, and is used to extract spatiotemporal features of lip movements; The recognition module performs lip reading recognition based on the extracted spatiotemporal features and outputs the recognition results; The visualization module is used to display the recognition results in the form of real-time text subtitles on the Web front-end page.

2. The lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 1, characterized in that: The video preprocessing module uses the YOLOv7 algorithm to perform lip detection and positioning.

3. The lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 1, characterized in that: The improved TSM-ResNet50 spatiotemporal network model also introduces a gated recurrent unit network and an attention mechanism to enhance the network's long-term temporal modeling capabilities and key feature extraction capabilities.

4. The lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 1, characterized in that: It also includes a real-time speech recognition module, which is used to convert the speech of a healthy person into text information in real time, and to assist in comparing it with the text generated by lip reading recognition.

5. The lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 1, characterized in that: The B / S architecture is adopted and the lightweight Flask framework is used to realize the visualization of lip reading recognition.

6. An operating method of a lip reading recognition system based on an improved TSM-ResNet50 spatiotemporal network model, characterized in that: The following steps are involved: S1, pre-process the input video data and use the YOLOv7 algorithm to detect and locate the lips; S2, input the preprocessed video data into the improved TSM-ResNet50 spatiotemporal network model to extract the spatiotemporal features of lip movement; S3, based on the extracted spatiotemporal features, lip reading recognition is performed through the recognition module, and the recognition result is output; S4. Display the recognition results in the form of real-time text subtitles on the Web front-end page.

7. The operating method of the lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 6, characterized in that: The improved TSM-ResNet50 spatiotemporal network model combines the time shift module and the residual network ResNet-50 architecture when extracting spatiotemporal features, and introduces a gated recurrent unit network and an attention mechanism.

8. The method for operating the lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 6, characterized in that: The method also includes the steps of converting the speech of a healthy person into text information in real time, and performing auxiliary comparison with the text generated by lip reading recognition to improve the feasibility and practicality of recognition.

9. The method for operating the lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 6, characterized in that: The visualization of the lip reading recognition system is realized through B / S architecture and lightweight Flask framework.

10. The method for operating the lip reading recognition system based on the improved TSM-ResNet50 spatiotemporal network model according to claim 6, characterized in that: It also includes the following steps: the user submits registration and login requests through the front end, the system verifies the information through the MySQL database, and after successful verification, the user can perform lip reading recognition operations.

Images