Deep learning based deafness assistant communication glasses

Abstract

The application provides a deep learning-based deafness auxiliary communication glasses, comprising: a glasses main body; an image acquisition device suspended on the glasses main body; the image acquisition device is used for acquiring a video image containing the lip shape and facial expression of a person communicating with the deaf person; a miniature projection device arranged on one lens of the glasses main body; a data processing device arranged on the frame of the glasses main body, wherein a data storage and transmission module is arranged in the data processing device, and the data storage and transmission module is used for storing and outputting data; the data processing device is used for calling the video image acquired by the image acquisition device, loading a pre-set lip reading recognition model, recognizing the text according to the video image and assembling the text into a sentence, and displaying the sentence on the miniature projection device for the deaf person to view. In the conversation process of the deaf person, the video recorded by the glasses is transmitted to the lip shape recognition model as an input parameter, and the model recognizes the sentence according to the training result and displays the sentence on the lens.

Description

Technical Field

[0001] This invention relates to the field of smart glasses technology, specifically to a deep learning-based assistive communication glasses for the deaf. Background Technology

[0002] Lip reading plays a crucial role in human communication and language comprehension. It's the task of decoding text from the movement of a speaker's mouth. Traditional methods divide the problem into two phases: designing or learning visual features, and prediction. Lip reading can be categorized into three methods: automatic lip reading, deep learning-based classification methods, and sequence prediction in speech recognition. Automatic lip reading: Most existing lip reading work does not require deep learning. These works require extensive preprocessing of frames to extract image features, and temporary preprocessing of frames to extract video features (e.g., optical flow or motion detection). Deep learning-based classification methods: perform word or phoneme classification, such as methods that learn multimodal audiovisual representations. For deaf individuals in everyday environments, due to the complexity of dialogue scenarios and numerous interfering factors, recognizing the other party's intent through speech is difficult to effectively help them improve their understanding of the conversation. Therefore, algorithms with high lip reading recognition rates are key to enabling normal conversation for deaf individuals.

[0003] The Chinese language is profound and nuanced; the same sentence can convey different meanings depending on tone and pauses. Existing lip-reading models cannot fully capture effective keyframes, resulting in inaccurate word boundary delineation. Video samples under the same word label often exhibit variations. Accurately capturing lip movements is also challenging (words with similar pronunciations have similar lip shapes, and homonym recognition further complicates the process). These factors lead to inaccurate purity, hindering effective inference of video content. In fact, ensuring good computer lip-reading performance largely depends on two factors: the ability to effectively capture lip movements and the ability to effectively resist noise caused by changes in posture, lighting, speaker appearance, etc.

[0004] Traditional approaches address this problem in two steps: designing or learning visual features, and prediction. This invention employs a deep lip-reading method with end-to-end training, using sentence-level sequence prediction and incorporating emotional factors into a model that maps variable-length video sequences to text to decode text based on lip movements. Summary of the Invention

[0005] To address the technical problems existing in the background art described above, the present invention provides a deep learning-based assistive communication glasses for the deaf, comprising: The main body of the glasses; An image acquisition device suspended on the main body of the glasses; the image acquisition device is used to acquire video images containing the lip movements and facial expressions of a person communicating with a deaf person; A miniature projection device is installed on one of the lenses of the main body of the glasses; A data processing device is installed on the frame of the glasses body. The data processing device contains a data storage and transmission module for storing and outputting data. The data processing device is used to call up video images acquired by the image acquisition device, load a pre-set lip reading recognition model, recognize text based on the video images, assemble them into sentences, and display the sentences on the micro-projection device for the deaf to view.

[0006] As a further explanation of the present invention, the construction process of the lip reading recognition model includes: Constructing a video database of lip shapes for lip reading recognition; For each video sample, a 100×50 pixel region centered on the mouth is extracted from each frame to construct a training set. The entire training set data is then standardized to the RGB channel to have zero mean and unit variance. Koopman extracts invariant and variable features from the input sequence of images, and divides these features into two groups. Each group has two branches: variable features and invariant features, which are used for facial expression recognition and to generate feature F. f Another set of two branches, one with variable features and the other with invariant features, is fed into a 3-layer spatiotemporal convolutional neural network to construct a Conv-LSTM as the spatiotemporal convolutional neural network for each layer; the feature F output by fusing the variable and invariant features of the 3-layer spatiotemporal convolutional neural network is then used. c and F f The fusion is then passed to two softmax processors: one softmax is used to recognize lip-reading text, and the other softmax is used to recognize tone of voice. When integrating sentences, it assists in the judgment of sentence segmentation and symbols. Generate a lip-reading recognition model.

[0007] As a further explanation of the present invention, the invariant features are the inherent features of mouth shape and facial expression, and the variable features are the features exhibited by different people under different lighting, angles, and conditions.

[0008] As a further explanation of the present invention, the construction process of the lip reading recognition model also includes: performing a simple transformation on the training set to enhance the dataset to reduce overfitting, thereby obtaining 20 times the training data.

[0009] As a further explanation of the present invention, a sensor is provided inside the nose pad of the main body of the glasses. The sensor is used to sense whether the glasses are being worn and to send the sensing result to the data processing device.

[0010] As a further explanation of the present invention, the main body of the glasses is also provided with a power supply device for supplying power to the image acquisition device, the micro projection device and the data processing device.

[0011] Compared with the prior art, the present invention has the following beneficial technical effects: This invention utilizes a OneToAll model based on a Koopman spatiotemporal convolutional neural network, trained end-to-end to obtain a lip-reading recognition model. OneToAll can perform multiple tasks with a single model. In lip-reading recognition, it first identifies facial expressions and then fuses the results with variable and invariant features extracted by Koopman, mapping the speaker's lip image frame sequence onto the entire sentence. Since lip shape changes continuously and sequentially during the expression of a word or sentence, compared to traditional recognition methods, this invention's end-to-end model no longer needs to segment the video into words before predicting the sentence. Instead, it introduces features of temporal lip shape changes to identify sentences, resulting in higher accuracy.

[0012] This invention combines assistive communication glasses for the deaf. During conversations, the glasses record the other person's lip movements in real time. A lip-reading recognition model analyzes the lip movements into sentences, which are then projected onto the lenses for the deaf person to view and understand the speaker's intent. Compared to commercially available projection-enabled smart glasses, this device, besides the system's storage for the model, has no other storage for the recorded content. The intention is to analyze spoken content in real time without saving the video. Storing video would require a large amount of storage space, increasing device cost and glasses weight, causing discomfort for the wearer. All video content is transmitted as a data stream without any saving.

[0013] Other features and advantages of this technical solution will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the technical solution. The objectives and other advantages of this technical solution can be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0014] The technical solution of this invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the present technical solution and form part of the specification. They are used together with the embodiments of the present technical solution to explain the present technical solution, but do not constitute a limitation thereof. In the accompanying drawings: Figure 1 A schematic diagram of the structure of the deep learning-based assistive communication glasses for the deaf provided by the present invention. Detailed Implementation

[0016] The preferred embodiments of this technical solution are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this technical solution and are not intended to limit this technical solution.

[0017] like Figure 1 As shown, this invention provides a deep learning-based assistive communication glasses for the deaf, comprising: Glasses body 1; An image acquisition device 2 is suspended on the main body of the glasses 1; the image acquisition device 2 is used to acquire video images containing the lip movements and facial expressions of a person communicating with a deaf person. A miniature projection device 4 is installed on one of the lenses of the main body of the glasses 1; A data processing device 3 is installed on the frame of the glasses body 1. The data processing device 3 is equipped with a data storage and transmission module, which is used to store and output data. The data processing device 3 is used to call the video image acquired by the image acquisition device 2, load the pre-set lip reading recognition model, recognize the text according to the video image and assemble it into a sentence, and display the sentence on the micro projection device 4 for the deaf person to view.

[0018] The image acquisition device 2 can be a camera suspended in front of the glasses or in front of the right eye lens, with a 5-megapixel resolution capable of shooting 720p video. The data processing device 3 can be a wide, strip-shaped computer processor located on the right side of the frame. The micro-projection device 4 can be a head-mounted micro-display mounted on the lens, projecting data onto a small screen above the user's right eye; the display effect is similar to a 25-inch high-definition screen 2.4 meters away, utilizing the principle of head-up display (HUD). The micro-projector first projects light onto a reflective screen, which is then refracted through a convex lens onto the user's eyeball, achieving a so-called "first-order magnification," forming a sufficiently large virtual screen in front of the viewer to display simple text information and various data.

[0019] A sensor is installed inside the nose pad 5 of the main body of the glasses 1. The sensor is used to detect whether the glasses are being worn and sends the sensing result to the data processing device 3. The nose pad is adjustable to fit different face shapes. The sensor is a capacitive sensor implanted in the nose pad. When an object approaches the sensor, it changes the electric field distribution of the sensor, thereby causing a change in the capacitance on the sensor. This change can detect whether the glasses are being worn. When the capacitive sensor detects that the glasses are being worn, it transmits a signal to the data processing device 3. At this time, the data processing device 3 controls the image acquisition device 2 to start acquiring images.

[0020] The main body of the glasses 1 is also equipped with a power supply device to power all components such as the image acquisition device 2, the micro-projection device 4, the data processing device 3, and sensors. For example, a power supply battery can be installed, and the battery's capacity can support the glasses' normal use for a day. Charging can be done via a Micro USB interface or a specially designed charger. The operating system used by the hearing-impaired assistive communication glasses can be Android 4.0.4, version number Ice Cream Sandwich, for developing Android applications.

[0021] The aforementioned assistive communication glasses for the deaf are actually a combination of a miniature projector, camera, sensor, storage and transmission device, and control device.

[0022] The construction process of the lip-reading recognition model includes the following specific steps: Step C011: Construct a lip-shape video database for lip reading recognition.

[0023] Step C012: Extract a 100×50 pixel region centered on the mouth from each frame of each video sample to construct a training set. Standardize the entire training set data to the RGB channel to have zero mean and unit variance.

[0024] Step C013: Perform a simple transformation on the training set to augment the dataset and reduce overfitting, resulting in 20 times more training data.

[0025] Step C014: Koopman extracts invariant and variable features from the input image sequence, and divides the two features into two groups. Each group has two branches: variable features and invariant features, which are used for expression recognition and to generate feature F. f Another set of two branches, one with variable features and the other with invariant features, is fed into a 3-layer spatiotemporal convolutional neural network to construct a Conv-LSTM as the spatiotemporal convolutional neural network for each layer; the feature F output by fusing the variable and invariant features of the 3-layer spatiotemporal convolutional neural network is then used. c and F f The fusion is then passed to two softmax processors: one for recognizing lip-reading text and the other for recognizing tone. When integrating sentences, it assists in punctuation and punctuation judgment. By building an end-to-end model, the recognition efficiency and accuracy are accelerated.

[0026] Let T be the time step of the input x, and m be the feature dimension at each time step, representing m-dimensional features.

[0027] x =( x 1, x 2,..., xT (1) xt =( xt 1, xt 2,..., xtm (2) Koopman extracts variable features f from x. v and immutable feature f iv , will f v and f iv The input is a spatial flow convolutional neural network, which is essentially an image classification structure that takes consecutive single video frames as input and extracts facial expression representations F from static images. f ;f v and f iv Then, the image is fed into a spatiotemporal convolutional neural network to extract lip movement representations from static images, and fused with F... f and F c The recognition process is performed; the spatiotemporal network takes lip-sync video data as input, and the RGB values ​​along the time frame are assigned to an independent channel; features are extracted through a dual-path structure and passed as input parameters to the Long Short-Term Memory (LSTM) network to train the model. Let w represent the parameters of the LSTM network, then the LSTM can be represented as a function: y = Nw ( x (3) The output time step is also T, which corresponds one-to-one with the input. The output dimension of each time step is n, which represents n-dimensional output, and is actually n probabilities.

[0028] y =( y 1, y 2,..., yT (4) yt =( yt 1, yt 2,..., (ytn) (5) For recognizing 4000 commonly used Chinese characters, the output dimension n of an LSTM at each time step is 4000, representing the probability of each of the 4000 characters being output at that time step. If we select one element at each time step based on these probabilities, we can obtain the output sequence, whose output space can be denoted as... L T A B transform is defined to transform the output sequence of the LSTM into the true output. Unlike foreign languages, the recognition order in Chinese is the same as the word order; there's no need to reorder the sentences. Only the symbols and phrases following the current word need to be determined. Therefore, an emotion factor is introduced to enhance the judgment of phrases and symbols. The network uses an expression loss value. flip-sync loss value c , tone loss value s Three loss functions are combined to enhance the network's convergence speed and recognition accuracy.

[0029] Step C015: Generate a lip-reading recognition model.

[0030] In the above model, the invariant features are the inherent features of lip shape and facial expression, while the variable features are the features exhibited by different lighting, angles, and different people.

[0031] During conversations among deaf individuals, the video recorded by the assistive communication glasses is passed as input parameters to the lip-reading recognition model. The model then identifies sentences based on the training results and displays them on the lenses.

[0032] The purpose of this invention is to provide deaf individuals with a communication channel to interact with hearing people in complex environments. Because deaf individuals cannot hear the voices of hearing people, they are unable to effectively understand the intentions of others. This invention employs a Koopman-based spatiotemporal convolutional neural network, trained end-to-end to obtain a OneToAll lip-reading model. The recognition model is a sentence-level lip-reading model that uses a single-end, speaker-independent deep model to simultaneously learn spatiotemporal visual features and a sequence model. The OneToAll model simultaneously recognizes lip movements and facial expressions through sequential images, introducing emotional factors (facial expression recognition) during image-to-text conversion, which enhances the accuracy of sentence phrases and symbols, better helping deaf individuals understand the current context in Chinese-speaking contexts. Koopman can extract invariant and variable features from sequences. Invariant features are the inherent features of lip movements and expressions, while variable features are the features exhibited under different lighting, angles, and by different people. Effectively extracting both types of features through Koopman allows for effective learning of recognition features, increasing recognition accuracy. The deaf wear glasses equipped with a camera and a miniature projector to record the other person's lip movements in real time during conversations. This method makes it easy to intervene in the conversation between the deaf and the speaker. The lip-reading recognition model identifies the speaker's language and displays it on the glasses for the deaf to observe in real time, thereby facilitating effective communication between the speakers.

[0033] Obviously, those skilled in the art can make various modifications and variations to this technical solution without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this technical solution and their equivalents, this technical solution also intends to include these modifications and variations.

Claims

1. A deep learning-based assistive communication glasses for the deaf, characterized in that, include: The main body of the glasses (1); An image acquisition device (2) is suspended on the main body of the glasses (1); the image acquisition device (2) is used to acquire video images containing the lip movements and facial expressions of a person communicating with a deaf person; A micro-projection device (4) is disposed on one of the lenses of the main body of the glasses (1). The data processing device (3) is installed on the frame of the main body of the glasses (1). The data processing device (3) is equipped with a data storage and transmission module, which is used to store and output data. The data processing device (3) is used to call the video image obtained by the image acquisition device (2), load the pre-set lip reading recognition model, recognize the text according to the video image and assemble it into a sentence, and display the sentence on the micro projection device (4) for the deaf to view. The construction process of the lip reading recognition model includes: Constructing a video database of lip shapes for lip reading recognition; For each video sample, a 100×50 pixel region centered on the mouth is extracted from each frame to construct a training set. The entire training set data is then standardized to the RGB channel to have zero mean and unit variance. Koopman extracts invariant and variable features from the input sequence of images, and divides these features into two groups. Each group has two branches: variable features and invariant features, which are used for facial expression recognition and to generate feature F. f Another set of two branches, one with variable features and the other with invariant features, is fed into a 3-layer spatiotemporal convolutional neural network to construct a Conv-LSTM as the spatiotemporal convolutional neural network for each layer; the variable and invariant features of the 3-layer spatiotemporal convolutional neural network are then fused to output feature F. c Variable and invariant features are fed into a spatiotemporal convolutional neural network to extract lip movement representations from static images. c and F f The lip-reading text is fused and passed to two softmax processors. One softmax processor is used to recognize the lip-reading text, and the other softmax processor is used to recognize the tone of voice. When integrating sentences, it assists in the judgment of sentence segmentation and symbols. The invariant features are the inherent features of lip shape and facial expression, and the variable features are different lighting conditions and angles. Generate a lip-reading recognition model.

2. The deep learning-based assistive communication glasses for the deaf as described in claim 1, characterized in that, The process of constructing the lip reading recognition model also includes: performing simple transformations on the training set to enhance the dataset and reduce overfitting, resulting in 20 times the amount of training data.

3. The deep learning-based assistive communication glasses for the deaf as described in claim 1, characterized in that, A sensor is provided inside the nose pad (5) of the main body of the glasses (1). The sensor is used to sense whether the glasses are being worn and to send the sensing result to the data processing device (3).

4. The deep learning-based assistive communication glasses for the deaf as described in claim 1, characterized in that, The main body (1) of the glasses is also provided with a power supply device for supplying power to the image acquisition device (2), the micro projection device (4) and the data processing device (3).

Images