Data processing method and system for multi-modal face moving point data and vocal cord motion data

Through the processing methods of multimodal facial moving point data and vocal cord movement data, a facial and neck movement model for Chinese vocalization was established to help deaf and dumb people imitate and make vocalization, solving the problem of high threshold and inconvenience in assisting deaf and dumb people's communication methods, and realizing Intuitive voice training and improve communication efficiency.

CN119025825AActive Publication Date: 2024-11-26ZHEJIANG UNIV
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
CN202411114777.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-26
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

The existing communication methods and devices for deaf and dumb people have problems such as high thresholds and inconvenience in learning and popularization. In particular, the regionality and logic of sign language and the difference between normal language limits its universality and the formation of written language.

Method used

Using multimodal facial moving point data and vocal cord motion data, the data processing method is adopted to collect continuous facial vocalization images and throat vibration data of normal people, and the space-time characteristics are extracted, and the facial neck movement model for Chinese vocalization is established to help deaf and mute people. Imitate voice and provide immediate feedback to lower learning thresholds and improve communication efficiency.

Benefits of technology

It provides a new, intuitive and convenient method of voice vocal training, lowers the learning threshold, improves communication efficiency and quality, and is suitable for deaf and dumb groups around the world and promotes their social integration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025825A_ABST
    Figure CN119025825A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method and system for multi-modal face moving point data and vocal cord motion data. The method comprises the steps of providing a text, collecting face continuous images or videos and throat vibration data when a normal person makes a sound, preprocessing the data, extracting time features and space features, establishing a face and neck movement model of Chinese sound, and enabling the deaf-mute to make a sound in a simulated mode according to the model and obtain feedback. The system comprises a depth camera, a throat vibration sensor and a microphone, provides instant feedback for the deaf-mute, reduces the learning threshold, improves the communication efficiency through comprehensive utilization of multi-modal data, and is suitable for the deaf-mute group in the global range. According to the invention, voice production training is promoted, and the method has a wide application prospect and social significance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of biological signal processing, and in particular relates to a data processing method and system for multimodal facial motion point data and vocal cord motion data. Background Art

[0002] Speech is the main way of communication between humans and a basic survival skill. Speech recognition is a voice technology in daily life that can help computers understand the speaker's intention. Its development has greatly promoted the relationship between humans and computers and facilitated natural and efficient human-computer interaction. However, for the deaf and mute who cannot speak, simple speech recognition technology is of little significance.

[0003] People often say that "ten deaf people are mute". The "mute" here does not mean "cannot speak", but "cannot speak". When normal people speak, they are always monitored by their ears, and can get timely feedback in the process of learning to speak; but people with bilateral deafness cannot hear their own pronunciation, cannot judge the effect of speaking, and have no timely hearing feedback, so they cannot express their intentions in words. Over time, they go from simple deafness to deaf-muteness.

[0004] Sign language uses hand gestures to measure movements, and simulates images or syllables based on changes in gestures to form certain meanings or words. It is a language of hands for people with hearing impairments or who cannot speak to communicate and exchange ideas with each other.

[0005] However, in actual application scenarios, sign language itself still has problems and limitations. As a language with low universality, sign language itself requires special training and learning, and has certain barriers, which limits the universality of sign language communication. In addition, there are many local sign languages ​​around the world, which makes sign language communication even more inconvenient; in addition, the language logic of sign language itself is different from that of normal language, and the written language of hearing-impaired people is also restricted by sign language and cannot form correct written language;

[0006] Existing methods, devices and systems for assisting the deaf-mute in communication usually collect other physiological signals, such as eye movement signals or physical buttons, which require the addition of additional devices and logic; or they are extensions of traditional sign language translation and still fall into the rut of sign language.

[0007] Therefore, it is necessary to provide a device for assisting the deaf-mute to learn and train normal vocalization to solve the above problems.

[0008] Invention patent content

[0009] The present invention provides a data processing method and system for multi-modal facial motion point data and vocal cord motion data in order to solve the technical problems existing in the known technology.

[0010] A data processing method for multimodal facial motion point data and vocal cord motion data comprises the following steps:

[0011] Step 1, providing a single-word text, collecting continuous images or videos of normal people's facial vocalization and laryngeal vibration data;

[0012] Step 2, preprocessing the collected offline facial continuous images or videos and laryngeal vibration data to obtain preprocessed facial motion point data and vocal cord movement data;

[0013] Step 3: Input the preprocessed vocal cord motion data into the vocal cord motion data recognition model to extract the temporal and spatial features of the vocal cord motion data. Input the preprocessed facial motion point data into the facial motion point data recognition model to extract the temporal and spatial features of the facial motion point data points.

[0014] Step 4, processing the obtained facial motion point data and vocal cord motion data according to their temporal and spatial features, obtaining the facial feature point motion trajectory and vocal cord motion trajectory of Chinese speech based on normal people, and finally establishing a face and neck motion model for Chinese speech;

[0015] Step 5: The deaf-mute imitates the Chinese pronunciation face and neck movement model and collects the sound signal during the pronunciation. The facial motion point data, vocal cord movement data and sound signal of the deaf-mute are input into the Chinese pronunciation face and neck movement model to obtain the evaluation result and feedback to the deaf-mute;

[0016] In the present invention, compared with auxiliary communication devices for the deaf-mute based on other physiological signals, the data collected in step 1 is generated during normal vocalization, which is simple and easy to obtain. The decoding method in steps 3 and 4 and the constructed face and neck movement model of Chinese vocalization follow the physiological changes of Chinese vocalization, conform to the normal logic of written language, and have no additional mapping relationship. In step 5, the deaf-mute person imitates the vocalization based on the face and neck movement model of Chinese vocalization, which is expected to get rid of the assistance of additional systems and realize autonomous Chinese vocalization.

[0017] In step 1, the laryngeal vibration signal includes the vibration generated by the friction between the vocal cords and the air, the vibration generated by the opening and closing movement of the vocal cords, and the vibration signal transmitted through the neck muscles;

[0018] The facial motion point data include: peripheral facial motion data of the face, lips and mandible when speaking.

[0019] In step 2, the preprocessing includes signal filtering, amplification, dimensionality reduction and Fourier transform.

[0020] In step 3, the method for extracting spatiotemporal features is Conformer;

[0021] In step 5, the evaluation result is calculated by the Multimodal DBM method.

[0022] A system for assisting deaf-mute voice training using multimodal facial motion point data and vocal cord motion data, comprising:

[0023] A depth camera for capturing continuous images or video, a sensor for recording throat vibrations, and a microphone for recording sound;

[0024] A depth camera for capturing continuous images or video, a sensor for recording throat vibrations, and a microphone for recording sound;

[0025] The module for extracting time features and space features is used to obtain vocal facial motion point data after preprocessing the continuous images or videos taken by the depth camera, and extract time features and space features from the facial motion point data; it is used to obtain vocal cord motion data after preprocessing the laryngeal vibration signal collected by the laryngeal vibration sensor, and extract time features and space features from the vocal cord motion data;

[0026] The feature reconstruction module is used to use the temporal and spatial characteristics of normal people's facial motion point data and vocal cord motion data to form the facial feature point motion trajectory and vocal cord motion trajectory of Chinese pronunciation, and to construct the face and neck motion model of Chinese pronunciation;

[0027] The correction module is used to read the prompts and guidance of the face and neck movement model of the deaf-mute according to the Chinese pronunciation, imitate the face and neck movement of normal people in Chinese pronunciation, perform offline data of pronunciation, facial dynamic point data, vocal cord movement data and sound signals of the deaf-mute, and perform multimodal recognition to output the correction results.

[0028] The deaf-mute person imitates the facial and neck movements of normal people in Chinese pronunciation according to the prompts and guidance of the facial and neck movement model for Chinese pronunciation, and makes sounds. At the same time, continuous images or videos, laryngeal vibration signals and actual sounds are recorded, and after further processing, they are sent to the facial and neck movement model for Chinese pronunciation to obtain evaluation results, which are fed back to the deaf-mute person.

[0029] The advantages and positive effects of the present invention are:

[0030] Promoting speech training: This invention combines facial motion point data with vocal cord movement data to provide a new speech training method for the deaf-mute. Through the comprehensive use of multimodal data, the deaf-mute can more intuitively understand the physiological process of vocalization, which helps to improve vocal skills.

[0031] Providing instant feedback: Traditional vocal training methods often lack instant auditory feedback. The present invention can provide more intuitive and accurate instant feedback to the deaf-mute by collecting facial motion point data and vocal cord movement data, helping them to adjust and improve their vocal effects in a timely manner.

[0032] Lowering the learning threshold: Compared with traditional sign language training, the method provided by the present invention is more intuitive and convenient, and does not require additional language learning, thereby lowering the learning threshold and enabling more deaf-mute people to easily learn and master vocal skills.

[0033] Improve communication efficiency: By adopting the method provided by the present invention for voice training, the deaf-mute can express their intentions more accurately, thereby improving the efficiency and quality of communication and promoting their integration and communication with the surrounding society.

[0034] Wide applicability: The present invention is not limited by region and language and is applicable to deaf-mute groups around the world, providing them with a universal speech vocalization training method with broad application prospects and social significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a structural schematic diagram of a system for assisting deaf-mute voice training using multimodal facial motion point data and vocal cord movement data of the present invention.

[0036] Figure 2 It is a schematic diagram of a workflow for extracting, analyzing and modeling vocal cord movement data of the present invention.

[0037] Figure 3 It is a schematic diagram of the workflow of facial dynamic point data extraction, analysis and modeling of the present invention.

[0038] Figure 4 It is a workflow diagram of a data processing method for multimodal facial motion point data and vocal cord motion data of the present invention. DETAILED DESCRIPTION

[0039] In order to further understand the content, features and effects of the present invention, the following embodiments are listed and described in detail with reference to the accompanying drawings:

[0040] The Chinese meanings of the English words and abbreviations in this article are as follows:

[0041] Conformer: The Conformer algorithm is a speech recognition model that combines the advantages of Transformer and Convolutional Neural Network (CNN). It is designed to capture both global and local features of speech signals, thereby improving the performance and efficiency of speech recognition. This paper uses it to extract temporal and spatial features of facial motion data and vocal cord motion data.

[0042] Multimodal DBM: Multimodal Deep Boltzmann Machine (MDBM) is a deep generative model for processing multiple input modes. The model can capture the complex relationship between different modes by jointly learning the latent representations of multiple input modes. Specifically, MDBM fuses data from different modalities through a multi-layer neural network structure to generate a unified latent representation. This paper uses Multimodal DBM to calculate the multimodal data input to the face and neck movement model of Chinese pronunciation and obtain the final evaluation results

[0043] Figure 1 The display shows a schematic diagram of a multi-modal facial motion point data and vocal cord movement data-assisted voice training system for the deaf-mute, including a camera for collecting continuous facial images or videos during voice production, a microphone for collecting laryngeal vibration signals during voice production, and actual sound. The deaf-mute imitates the Chinese voice of normal people based on the facial and neck movement model of Chinese voice production displayed on the screen for training.

[0044] Figure 2 What is demonstrated is the extraction, analysis and modeling process of vocal cord movement data. After the laryngeal vibration signal collected by the sensor is input, it is first preprocessed and then the time and space characteristics are extracted. According to the extracted time and space characteristics, the vocal cord movement trajectory of the voice is reconstructed, and together with other information, it constitutes the face and neck movement model of Chinese voice.

[0045] Figure 3 What is demonstrated is the extraction, analysis and modeling process of facial motion point data. The continuous images or videos taken by the depth camera are taken as input. They are first preprocessed and then the time-space features are extracted. According to the extracted time-space features, the motion trajectory of the facial feature points during vocalization is reconstructed, and together with other information, a face and neck motion model for Chinese vocalization is constructed.

[0046] Figure 4 What is shown is the implementation process of a preferred embodiment of the present invention.

[0047] Example 1

[0048] The present invention provides a data processing method for multimodal facial motion point data and vocal cord motion data, and the specific implementation steps are as follows:

[0049] The following is a preferred embodiment of the present invention to further illustrate the working process and working principle of the present invention:

[0050] Step 1: Data Collection

[0051] Provide text containing a single Chinese character, use a depth camera to collect continuous facial images or videos of normal people speaking, and use a laryngeal vibration sensor to record laryngeal vibration data. The facial image captured by the depth camera includes the movement data of the face, lips, and mandible, and the laryngeal vibration sensor records the vibration generated by the friction between the vocal cords and the air, the vibration generated by the opening and closing movement of the vocal cords, and the vibration signal transmitted through the neck muscles.

[0052] Step 2: Data Preprocessing

[0053] The collected facial continuous images or videos and laryngeal vibration data are preprocessed. The preprocessing steps include signal filtering, amplification, dimensionality reduction and Fourier transform, the purpose of which is to remove noise, enhance the effective part of the signal, simplify the data dimension, and retain the main feature information.

[0054] Step 3: Feature Extraction

[0055] The preprocessed vocal cord motion data is input into the vocal cord motion data recognition model, and the temporal and spatial features of the vocal cord motion data are extracted using the Conformer method. The preprocessed facial motion point data is input into the facial motion point data recognition model, and the temporal and spatial features of the facial motion point data are also extracted using the Conformer method.

[0056] Step 4: Model building

[0057] The temporal and spatial features of the extracted facial motion point data and vocal cord motion data are processed to obtain the facial feature point motion trajectory and vocal cord motion trajectory based on normal Chinese speech, and finally establish a face and neck motion model for Chinese speech. This model can accurately reflect the facial and vocal cord motion state of normal people when speaking.

[0058] Step 5: Voice training for the deaf

[0059] The deaf-mute person performs imitation vocalization training based on the established Chinese vocalization face and neck movement model. At the same time, a depth camera and a laryngeal vibration sensor are used to collect facial motion point data and vocal cord movement data of the deaf-mute person when speaking, and the actual sound signals emitted are recorded. These data are input into the Chinese vocalization face and neck movement model, the evaluation results are calculated using the Multimodal DBM method, and the feedback results are provided to the deaf-mute person to help him adjust and improve his vocalization. Through the above embodiments, the deaf-mute person can perform more accurate Chinese vocalization training and improve his vocalization ability under the guidance of visual and auditory multimodality.

[0060] The embodiments described above are only used to illustrate the technical ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The patent scope of the present invention cannot be limited only by this embodiment, that is, any equivalent changes or modifications made to the spirit disclosed by the present invention still fall within the patent scope of the present invention.

Claims

1. A data processing method based on multimodal facial motion point data and vocal cord motion data, It is characterized in that The steps include: Step 1, providing a single-word text, collecting continuous facial vocalization images and laryngeal vibration data of a normal person; Step 2, preprocessing the collected facial vocalization continuous images and laryngeal vibration data to obtain preprocessed facial motion point data and vocal cord movement data; Step 3, inputting the preprocessed vocal cord motion data into a vocal cord motion data recognition model to extract the temporal and spatial features of the vocal cord motion data; The pre-processed facial motion point data is input into the facial motion point data recognition model to extract the temporal and spatial features of the facial motion point data points; Step 4, processing the temporal and spatial features of the vocal cord motion data obtained in step 3 to obtain a vocal cord motion trajectory based on a normal person's Chinese pronunciation; Processing the temporal and spatial features of the facial moving point data points obtained in step 3 to obtain a motion trajectory of the facial feature points based on the Chinese pronunciation of a normal person; A face and neck movement model for Chinese pronunciation is established based on the vocal cord movement trajectory of normal Chinese pronunciation and the facial feature point movement trajectory of normal Chinese pronunciation; Step 5, input the data of the deaf-mute person imitating the Chinese pronunciation into the face and neck movement model of Chinese pronunciation obtained in step 4. The model extracts the facial motion point data, vocal cord movement data and sound signals of the deaf-mute person and performs multimodal recognition to output the correction result.

2. The data processing method of multimodal facial motion point data and vocal cord motion data according to claim 1, It is characterized in that The laryngeal vibration data includes: vibration data generated by the friction between the vocal cords and the air, vibration data generated by the opening and closing movement of the vocal cords, and vibration signals transmitted through the neck muscles; The facial vocalization continuous images include: peripheral facial movement data of the face, lips and mandible during vocalization.

3. The data processing method of multimodal facial motion point data and vocal cord motion data according to claim 1, It is characterized in that In step 2, the preprocessing includes: signal filtering, amplification, dimensionality reduction and Fourier transform.

4. The method for processing multimodal facial motion point data and vocal cord motion data according to claim 1, It is characterized in that In step 3, Conformer is used to extract temporal and spatial features.

5. The data processing method of multimodal facial motion point data and vocal cord motion data according to claim 1, It is characterized in that In step 5, multimodal recognition adopts the MultimodalDBM method.

6. A system for assisting the deaf-mute in Chinese pronunciation training using multimodal facial motion point data and vocal cord motion data to implement the data processing method according to any one of claims 1 to 5, It is characterized in that include: A depth camera for capturing continuous images or video, a sensor for recording throat vibrations, and a microphone for recording sound; A module for extracting temporal and spatial features, which is used to obtain facial motion point data of vocalization after preprocessing the continuous images or videos taken by the depth camera, and extract temporal and spatial features from the facial motion point data; Used to pre-process the laryngeal vibration signal collected by the laryngeal vibration sensor to obtain vocal cord movement data, and extract time features and spatial features from the vocal cord movement data; The feature reconstruction module is used to use the temporal and spatial characteristics of normal people's facial motion point data and vocal cord motion data to form the facial feature point motion trajectory and vocal cord motion trajectory of Chinese pronunciation, and to construct the face and neck motion model of Chinese pronunciation; The correction module is used to read the prompts and guidance of the face and neck movement model of the deaf-mute according to the Chinese pronunciation, imitate the face and neck movement of normal people in Chinese pronunciation, perform offline data of pronunciation, facial dynamic point data, vocal cord movement data and sound signals of the deaf-mute, and perform multimodal recognition to output the correction results.

Citation Information

Patent Citations

  • Speech interactive training system and speech interactive training method

    CN102063903A

  • A system and method for assisting dialogues between a deaf person and a normal person, and a smart mobile phone

    CN106686223A

  • Vibration feedback system and device for speech rehabilitation

    CN108320625A

  • Language recognition system and method used for deaf and dumb

    CN108510988A

  • Early-stage autism screening system based on joint attention ability test and audio-video behavior analysis

    CN110313923A