Language auxiliary system based on airflow sensor and deep learning model

Abstract

The invention relates to a language assistance technology, and discloses a language assistance system based on an airflow sensor and a deep learning model, which is used for helping a person who cannot make a sound to realize language communication with others. The system comprises a language auxiliary device and a cloud. The language auxiliary device comprises an airflow sensor, an MCU, a data processing module, a communication module and a voice playing module. The airflow sensor collects airflow data at the mouth and nose when a user makes a sound; the data processing module preprocesses the airflow data and extracts airflow characteristics; the airflow features extracted by the MCU are uploaded to the cloud through the communication module, and the cloud identifies the pronunciation content corresponding to the airflow features by using a deep learning model, and replies the pronunciation content as an identification result to the language auxiliary device. And the MCU processes the received result replied by the cloud, converts the result into voice data, and performs voice playing through the voice playing module. The invention is suitable for the disabled who cannot produce sound.

Description

Technical Field

[0001] The present invention relates to language assistance technology, and in particular to a language assistance system based on an airflow sensor and a deep learning model, which is used to help people who cannot speak to achieve language communication. Background Art

[0002] The human vocalization mechanism is complex and is the result of a series of organs working in coordination. It has certain regular and repeatable movements. Articulation disorders refer to difficulty in pronunciation, abnormal pronunciation, unclear pronunciation but correct words. They are caused by muscle paralysis, ataxia or increased muscle tension of the vocal organs. They manifest as speech disorders, articulation disorders, weak and stiff pronunciation, speech fluency disorders, etc., which greatly affect language communication.

[0003] At present, the way for people who cannot speak (people with dysphonia) to communicate with others is to use sign language or writing to communicate with others. However, there are few people in society who can use sign language, and the efficiency of writing communication is relatively low, which limits communication. As we all know, language communication is the most direct and convenient way of communication between people. Therefore, there is an urgent need for a language assistance system that can help people who cannot speak to achieve language communication with others. Summary of the invention

[0004] The technical problem to be solved by the present invention is to propose a language assistance system based on airflow sensors and deep learning models to help people who cannot speak to achieve language communication with others.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] A language assistance system based on airflow sensors and deep learning models, comprising: a language assistance device and a cloud;

[0007] The language assisting device comprises: an airflow sensor, an MCU, a data processing module, a communication module and a voice playing module;

[0008] The airflow sensor is electrically connected to the data processing module, and the data processing module, the communication module and the voice playing module are all electrically connected to the MCU;

[0009] The airflow sensor is used to collect airflow data at the mouth and nose when the user pronounces, and transmit it to the data processing module;

[0010] The data processing module is used to pre-process the airflow data and extract airflow features, and transmit them to the MCU;

[0011] The MCU is used to upload the extracted airflow characteristics to the cloud through the communication module, and to process the results of the cloud reply when receiving them, convert them into voice data, and then play them through the voice playing module;

[0012] The communication module is responsible for data interaction between the language assistance device and the cloud;

[0013] The voice playing module is responsible for voice playing;

[0014] The cloud is deployed with a deep learning model, which is used to use the deep learning model to identify the pronunciation content corresponding to the corresponding airflow features after receiving the airflow features uploaded by the language assistance device, and reply the pronunciation content to the language assistance device as the recognition result.

[0015] Furthermore, the communication module adopts 5GRedCap.

[0016] Furthermore, the airflow data is preprocessed, including denoising and standardization.

[0017] Furthermore, the extracted airflow features include airflow velocity, direction and change rate features.

[0018] Furthermore, the deep learning model adopts a fusion network model of a convolutional neural network (CNN) and a long short-term memory network (LSTM).

[0019] Furthermore, the result of the cloud reply is a voice expression or a text expression of the pronunciation content.

[0020] The beneficial effects of the present invention are:

[0021] The above-mentioned system provided by the present invention can detect the airflow data when the user pronounces, and extract the airflow features after processing and upload them to the cloud through the communication module, and recognize the user's intention based on the trained deep learning model deployed in the cloud, that is, recognize the pronunciation content corresponding to the airflow features, and then feed it back to the terminal for voice playback, thereby realizing real-time translation of the user's pronunciation intention, helping people who cannot speak to communicate with others in language. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural diagram of a language assistance system based on an airflow sensor and a deep learning model in an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of deep learning model training in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] According to research, for each pronunciation, the airflow data at the mouth and nose will be different when pronouncing. Therefore, the airflow speed, direction, and change rate contained in the airflow data can be used to distinguish different pronunciations, so as to train a deep learning model that can learn the mapping relationship between different airflow features and pronunciations. In actual use, the airflow data when the user pronounces is collected through an airflow sensor, and the airflow features are extracted after processing and uploaded to the cloud through a communication module. The trained deep learning model deployed on the cloud is used to identify the user's intention, that is, the pronunciation content corresponding to the airflow feature is identified, and then fed back to the terminal for voice playback, so as to achieve real-time translation of the user's pronunciation intention, and help realize language communication between people who cannot speak and others. Based on the above conception, the present invention provides a language assistance system based on an airflow sensor and a deep learning model, which uses the airflow data during pronunciation and uses a deep learning module to translate the pronunciation intention, helping people who cannot speak to achieve language communication with others, and is suitable for people with disabilities who cannot speak.

[0025] Example

[0026] See also Figure 1 ,The language assistance system based on airflow sensor and deep learning model provided in this embodiment includes: a language assistance device and a cloud;

[0027] Among them, the language assistance device includes: an airflow sensor, an MCU, a data processing module, a communication module and a voice playing module; the airflow sensor is electrically connected to the data processing module, and the data processing module, the communication module and the voice playing module are all electrically connected to the MCU.

[0028] In specific implementation, the above-mentioned language assistance device can be designed as a wearable device, such as a helmet or a mask, etc., which is not specifically limited here.

[0029] The functions of each part are introduced as follows:

[0030] Airflow sensor: used to collect airflow data at the mouth and nose when the user pronounces, and transmit it to the data processing module.

[0031] Data processing module: used to pre-process the airflow data and extract airflow features, and transmit them to the MCU; among them, pre-processing includes denoising and standardization processing; denoising processing can remove noise as much as possible and retain useful signals; standardization processing can eliminate the influence of differences in dimensions and value ranges between feature indicators; the extracted airflow features can include airflow speed, direction and rate of change features, which can comprehensively reflect the differences in pronunciation.

[0032] MCU: It is the core processor of the device and is responsible for the coordination of various functional parts, including uploading the extracted airflow characteristics to the cloud through the communication module, and processing the results of the cloud reply when received, converting them into voice data and playing them through the voice playback module.

[0033] Communication module: responsible for data interaction between the language assistance device and the cloud. In an exemplary solution, the communication module can use 5G RedCap, which is a lightweight 5G technology with large bandwidth and low latency to meet the needs of real-time communication. The communication protocol can use HTTP / HTTPS or MQTT protocol.

[0034] Voice playback module: responsible for voice playback;

[0035] Cloud: A deep learning model is deployed to identify the pronunciation content corresponding to the airflow features after receiving the airflow features uploaded by the language assistance device, and reply the pronunciation content to the language assistance device as the recognition result.

[0036] For the deep learning model, in an exemplary solution, a fusion network model of a convolutional neural network (CNN) and a long short-term memory network (LSTM) can be selected. Of course, other applicable neural network models can also be selected.

[0037] Before model training, sample data needs to be prepared, for example, a large number of samples of airflow data at the mouth and nose when the subjects pronounce the words need to be recorded one by one with the corresponding words, so as to make labels, thus obtaining a large amount of sample data. Then the sample data is divided into a training set and a validation set, where the training set is used to train the model and the validation set is used to test the performance of the model.

[0038] Then, train the model on the training set and continuously optimize the model parameters. After a certain number of training iterations, use the validation set to evaluate the performance of the model and calculate indicators such as accuracy and recall. If the indicators meet the set requirements, the model can be put into use. If not, consider adjusting the model-related hyperparameters or directly selecting other models and continue training until a qualified model is finally obtained. For the training process, see Figure 2 .

[0039] Finally, the trained model is deployed in the cloud, which can be used in actual scenarios. In actual scenarios, the airflow data of the user's pronunciation is collected in real time (users who cannot speak have airflow data when pronouncing words, but no sound), and the airflow features are extracted after processing and uploaded to the cloud through the communication module. The pronunciation content corresponding to the airflow features is identified based on the trained deep learning model deployed in the cloud, and then fed back to the terminal for voice playback, so as to achieve normal language communication.

[0040] It is understandable that the sample data used by the deep learning model trained above is the airflow sample data of a large number of different users. The trained model has a certain degree of versatility. In order to accurately identify the user's pronunciation intention, a deep learning model can be trained separately for a specific user. For example, a character library is pre-set, and each time the operator points to a character, the user is asked to pronounce the character, thereby collecting the airflow sample of the user when pronouncing the character, and the airflow sample is recorded one by one with the corresponding character, so as to make a label. The deep learning model trained in this way is a model for the user, which can accurately translate the user's pronunciation intention.

[0041] Of course, in order to further improve the accuracy of the model in recognizing pronunciation intentions, it is also possible to consider adding facial muscle detection to the device, such as a flexible strain sensor, to detect muscle deformation during pronunciation. The characteristics of different muscle deformations can also be associated with different pronunciations. Therefore, the model jointly trained based on airflow detection signals and muscle deformation detection signals can more accurately recognize the user's pronunciation intentions. In actual application scenarios, the corresponding real-time signals collected include both airflow signals and muscle deformation signals.

[0042] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.

Claims

1. A language assistance system based on airflow sensor and deep learning model ， It is characterized in that Includes: language assistance devices and cloud; The language assisting device comprises: an airflow sensor, an MCU, a data processing module, a communication module and a voice playing module; The airflow sensor is electrically connected to the data processing module, and the data processing module, the communication module and the voice playing module are all electrically connected to the MCU; The airflow sensor is used to collect airflow data at the mouth and nose when the user pronounces, and transmit it to the data processing module; The data processing module is used to pre-process the airflow data and extract airflow features, and transmit them to the MCU; The MCU is used to upload the extracted airflow characteristics to the cloud through the communication module, and to process the results of the cloud reply when receiving them, convert them into voice data, and then play them through the voice playing module; The communication module is responsible for data interaction between the language assistance device and the cloud; The voice playing module is responsible for voice playing; The cloud is deployed with a deep learning model, which is used to use the deep learning model to identify the pronunciation content corresponding to the corresponding airflow features after receiving the airflow features uploaded by the language assistance device, and reply the pronunciation content to the language assistance device as the recognition result.

2. A language assistance system based on airflow sensor and deep learning model as claimed in claim 1, It is characterized in that The communication module adopts 5GRedCap.

3. A language assistance system based on airflow sensor and deep learning model as claimed in claim 1, Features The airflow data was preprocessed, including denoising and standardization.

4. A language assistance system based on airflow sensor and deep learning model as claimed in claim 1, Features The extracted airflow features include airflow velocity, direction and rate of change.

5. A language assistance system based on airflow sensor and deep learning model as claimed in claim 1, Features The deep learning model adopts a fusion network model of a convolutional neural network and a long short-term memory network.

6. A language assistance system based on an airflow sensor and a deep learning model as described in any one of claims 1 to 5, It is characterized in that The result of the cloud reply is a voice expression or a text expression of the pronunciation content.

Images