Flexible sensing recognition device and lip language recognition method based on same

Abstract

The invention relates to the technical field of artificial intelligence and lip language recognition, and provides a flexible sensing device and a lip language recognition method based on the same. According to the flexible sensing device and the lip language recognition method based on the flexible sensing device, a data acquisition unit is used for acquiring differential electromyographic signals; the transmission cable is used for transmitting the collected differential electromyographic signals; the analog noise filtering module is used for twice amplification and low-pass and band-pass filtering; the acquisition mainboard is used for controlling analog-to-digital conversion, data preprocessing and wireless transmission; a data preprocessing module; the software denoising module is used for software denoising and normalization; and the upper computer is used for the feature extraction and screening unit, machine learning classification and visualization. According to the device and the method, the fixing and contact problems of the rigid electrode in the lip language muscle electric signal acquisition process are solved, and more effective denoising and more accurate human body lip language signal identification can be carried out.

Description

Technical Field

[0001] The invention relates to the technical field of artificial intelligence and lip reading recognition, and proposes a flexible sensing device and a lip reading recognition method based on the device. Background Art

[0002] Nowadays, more and more people are suffering from hearing loss. Hearing-impaired people either communicate with others through inefficient sign language or expensive hearing aids. The difficulty of direct language communication increases sharply, and lip reading requires a lot of learning and completion. Therefore, if there is a device that can achieve the function of "lip reading recognition" and convert it into effective communication signals, it will greatly improve people's quality of life.

[0003] Existing communication aids for the hearing impaired are expensive, and current lip reading recognition technology is mainly implemented through image recognition. The method that relies on camera capture faces many challenges in practical applications, such as inconsistent lighting conditions and irregular body movements of the communicator, which leads to reduced recognition accuracy. In addition, the user must remain within the field of view of the camera, which limits the user's freedom and privacy protection. Summary of the invention

[0004] The purpose of the present invention is to address the defects of existing hearing-impaired communication auxiliary devices, which are high in cost and cannot meet the supply needs of ordinary hearing-impaired people suffering from hearing impairments in general economic conditions, and propose a flexible sensing recognition device and a lip reading recognition method based on the device.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] As one aspect of the present invention, a flexible sensor recognition device is proposed, including an external power supply and a host computer, and also including: a data acquisition unit, a transmission cable, an analog noise filtering module and an acquisition mainboard; the data acquisition unit is used to collect differential electromyographic signals; the transmission cable is used to transmit the collected differential electromyographic signals; the analog noise filtering module is used for double amplification, low-pass and band-pass filtering; the acquisition mainboard includes an analog-to-digital converter, a data preprocessing module, an input interface, a control module, a switch and a Bluetooth module; used to control analog-to-digital conversion, data preprocessing and wireless transmission; the data preprocessing module includes a channel amplitude The channel amplitude adjustment unit, the high-pass filtering unit, the low-pass filtering unit and the normalization unit; the channel amplitude adjustment unit is connected to the high-pass filtering unit and the low-pass filtering unit, and the high-pass filtering unit and the low-pass filtering unit are connected to the normalization unit; used for software denoising; the host computer includes a feature extraction and screening unit, a machine learning classification module and a visualization module; the feature extraction and screening unit is connected to the machine learning classification module; used for data processing and visualization classification output; the machine learning classification module receives the extracted and screened features output by the feature extraction and screening unit, classifies the screened features, and outputs the classification results.

[0007] In specific implementation, the external power supply is connected to the input interface and switch of the acquisition mainboard to power the mainboard. The analog noise filtering module includes a first amplification unit, a low-pass filter, a second amplification unit and a high-pass filter; the first amplification unit is connected to the low-pass filter, and the low-pass filter is connected to the second amplification unit; used for hardware amplification and filtering.

[0008] The data acquisition unit includes several groups of electrode patches; the electrode patches are pre-gelled mesh electrode patches. In specific implementation, each of the several groups of electrode patches includes a positive electrode patch, a negative electrode patch and a reference electrode patch;

[0009] The electrode patch is prepared from Ag / AgCl, AgCl, Ag, Au, carbon, paper base, ceramic or nano metal.

[0010] The electrode patches are distributed on the orbicularis oris muscle, masseter muscle, labii inferioris muscle, mylohyoid muscle, sternohyoid muscle, sternocleidomastoid muscle, levator alaris muscle, and risorius muscle / cheek muscle.

[0011] When the data acquisition unit is collecting data, the reference electrode sheet is attached to the corresponding position to obtain the reference voltage at different collection points, specifically:

[0012] (1) The reference electrode is attached to the zygomatic bone to obtain the reference voltage of the buccinator muscle;

[0013] (2) The reference electrode is attached to the temporal bone to obtain the reference voltage of the depressor labii inferioris muscle;

[0014] (3) The reference electrode is attached to the clavicle to obtain the reference voltage of the mylohyoid muscle;

[0015] The reference voltage is a standard voltage that does not change with the muscle force at the acquisition point.

[0016] As another aspect of the present invention, a lip reading recognition method based on a flexible sensing recognition device is proposed, comprising the following steps:

[0017] S1: Deploy differential electrodes on several channels on the skin surface;

[0018] In specific implementation, the differential electrode arrangement described in S1 is specifically as follows: the two electrodes should be attached to different positions on the same muscle, and the reference electrode is attached to the clavicle.

[0019] S2: Collect data through the deployed differential electrodes to obtain muscle electrical signals;

[0020] S3: preprocessing the muscle electrical signal by analog filtering, amplification, analog-to-digital conversion and bandpass filtering to obtain preprocessed data;

[0021] S4: normalize the preprocessed data and send the normalized data to the host computer;

[0022] S5: Extract and filter the features of the data received by the host computer to obtain the features to be classified;

[0023] The features to be classified include but are not limited to time domain, frequency domain, transform domain, entropy features and statistical features;

[0024] S6: Based on the features to be classified, a variety of machine learning classification algorithms are used to perform recognition and obtain recognition results;

[0025] S7: Compare and evaluate the performance of different algorithms and present them visually.

[0026] The number of the plurality of channels in S1 is greater than or equal to 4; the plurality of channels correspond to at least four of the orbicularis oris, masseter, depressor labii inferioris, mylohyoid, sternohyoid, sternocleidomastoid, levator alaris, and risorius / cheek muscles.

[0027] As a specific embodiment, the muscle distribution of the differential electrodes of several channels is specifically: selecting the levator alaris, zygomaticus minor, mylohyoid muscle, buccinator muscle, depressor labii inferioris muscle and platysma muscle.

[0028] The preprocessing in S3 specifically includes the following sub-steps:

[0029] S31: performing a first-stage amplification on the collected muscle electrical signal to obtain a first-stage amplified signal;

[0030] S32: performing low-pass filtering on the first-stage amplified signal to obtain a low-pass filtered signal;

[0031] S33: performing a second-stage amplification on the low-pass filtered signal to obtain a second-stage amplified signal;

[0032] S34: performing high-pass filtering on the secondary amplified signal to obtain an analog preprocessed signal;

[0033] S35: performing analog-to-digital conversion on the analog preprocessed signal to obtain a digital signal and subtracting the respective offset amplitudes from the digital signals of each channel;

[0034] S35 The offset amplitude is the average value of the digital signal amplitude of the corresponding channel;

[0035] S36: performing band-pass filtering on the digital signals of each channel after the bias amplitude is removed to obtain pre-processed data; the low-pass cut-off frequency of the band-pass filtering is greater than or equal to 55; and the high-pass cut-off frequency is greater than or equal to 40;

[0036] S4: normalization, including but not limited to Min-Max normalization, Log function conversion and Z-Score method; sending the normalized data to the host computer is achieved through a Bluetooth module;

[0037] S31: the magnification of the first stage amplification is 5 to 50; S33: the magnification of the second stage amplification is 20 to 100; the low-pass filter and the high-pass filter are exchanged.

[0038] The high-pass filter is a 4th-order Butterworth filter, a Chebyshev filter or a Bessel filter.

[0039] The feature extraction and screening described in S5 specifically includes the following steps:

[0040] S51: extracting various features from the data received from the host computer;

[0041] The various features include, but are not limited to, time domain features, frequency domain features, transform domain features, entropy features and statistical features;

[0042] Time domain features include but are not limited to: average absolute value, number of zero crossings, and average waveform length;

[0043] The average absolute value is the average amplitude of muscle electricity in a short period of time, reflecting the activity level and strength of the muscle; the average waveform length is a characteristic parameter used to characterize the smoothness of the signal and the degree of waveform change; the number of zero crossings is the number of times the signal crosses the zero point within a period of time;

[0044] The statistical characteristics include but are not limited to mean, root mean square value and variance;

[0045] The mean is the average value of muscle electricity;

[0046] Variance represents the degree of deviation of the signal;

[0047] The RMS value reflects the average power of muscle electricity generated during muscle movement during speech;

[0048] The frequency domain features include but are not limited to: average power frequency and median frequency;

[0049] The average power frequency is the centroid frequency of the power spectrum; the median frequency is the frequency that divides the power spectrum into two parts of equal power;

[0050] The transform domain features include but are not limited to: wavelet transform, discrete cosine transform and Hilbert-Huang transform; the wavelet transform uses db7, bior2.4 or Haar wavelet basis;

[0051] The entropy characteristics include but are not limited to: differential entropy, relative entropy, joint entropy, and Rayleigh entropy;

[0052] The differential entropy is used to measure signal uncertainty;

[0053] The relative entropy is used to calculate the asymmetry between signal channels;

[0054] The joint entropy is a measure of uncertainty in the joint distribution of two random variables;

[0055] The Rayleigh entropy is used to measure signal complexity;

[0056] S52: Filter important features for lip reading classification based on various features to obtain features to be classified.

[0057] The screening described in S52 screens the various features extracted in S51 through the KPCA algorithm, the mutual information method based on MIC, the distance correlation coefficient method or the random forest contribution method to obtain the features to be classified.

[0058] S7 The visual display includes visual display of the data received by the host computer, the features to be classified, and the recognition results;

[0059] The data received by the host computer is visualized, specifically: drawing the data received by the host computer, including but not limited to drawing a time domain diagram, a frequency domain diagram, and a time-frequency diagram;

[0060] Visualize the features to be classified, specifically: draw a graph of the results of feature extraction and screening, i.e., the features to be classified, and draw a bar chart of the order of feature importance;

[0061] The recognition results are visualized, specifically: the classification results of S6 using multiple machine learning classification algorithms are plotted, including but not limited to plotting accuracy comparison charts and confusion matrices of each classifier; specifically in this embodiment, the recognition results are visualized to display the recognized words.

[0062] The multiple machine learning classification algorithms described in S6 include, but are not limited to: SVM, KNN, linear classification, MLP, ensemble learning and CNN.

[0063] The data acquisition unit is connected to the transmission cable; the transmission cable is connected to the analog noise filtering module; the analog noise filtering module is connected to the acquisition mainboard; the external power supply is connected to the acquisition mainboard; the acquisition mainboard is connected to the host computer;

[0064] The analog-to-digital converter is connected to the channel amplitude adjustment unit of the data preprocessing module; the normalization unit of the data preprocessing module is connected to the input interface, the input interface is connected to the control module, the control module is connected to the switch, and the switch is connected to the Bluetooth module;

[0065] The data preprocessing module receives the digital signal converted by the analog-to-digital converter, performs digital filtering and normalization, and outputs the sample signal to the input interface of the acquisition mainboard;

[0066] The acquisition mainboard receives the signal output by the analog noise filtering module, performs analog-to-digital conversion, software filtering, normalization, Bluetooth transmission and power supply control, and outputs the data to the host computer; the host computer receives the data transmitted by the acquisition mainboard, performs feature extraction and screening, machine learning classification and visualization;

[0067] The machine learning classification module includes several classification models, receives the data after feature extraction and screening, performs result prediction, and outputs classification results;

[0068] The amplification factor of the first amplifying unit is 5 to 50; the amplification factor of the second amplifying unit is 20 to 100; in specific implementation: the amplification factor of the first amplifying unit is 10 times, and the low-pass filter uses a cutoff frequency of 500Hz, so that signals below 500Hz pass through; the amplification factor of the second amplifying unit is 68; the high-pass filter uses a cutoff frequency of 20Hz, and the sample signal obtained after the high-pass filter is used for subsequent processing; the low-pass filter can also exchange positions with the high-pass filter; or a band-pass filter is used, with a high-pass cutoff frequency of 20Hz and a low-pass cutoff frequency of 500Hz; and signals from 20 to 500Hz are retained.

[0069] Beneficial Effects

[0070] The present invention proposes a flexible sensing device and a lip reading recognition method based on the device, which has the following beneficial effects compared with the prior art:

[0071] 1. The lip reading recognition device proposed in the present invention solves the problem of fixation and contact of rigid electrodes in the process of collecting human facial muscle electrical signals, and can more accurately recognize human muscle electrical signals;

[0072] 2. The muscle electrical signal preprocessing method proposed in the present invention can perform denoising filtering more effectively, extracting the original signal from a chaotic state to obtain a nearly pure muscle electrical signal;

[0073] 3. The feature extraction and classification learning recognition method proposed in the present invention can greatly improve the accuracy of target recognition and more effectively recognize the characteristics of muscle electrical signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a block diagram of the flexible sensing device of the present invention and a schematic diagram of its connection;

[0075] Figure 2 It is a schematic diagram of electrode arrangement of a flexible sensing device of the present invention;

[0076] Figure 3 This is the appearance of the electrode patch of the data acquisition unit of the present invention.

[0077] Illustration Description:

[0078] 1- orbicularis oris, 2- masseter, 3- depressor labii inferioris, 4- mylohyoid, 5- sternohyoid, 6- sternocleidomastoid, 7- levator alaris, 8- risorius, 9- platysma. DETAILED DESCRIPTION

[0079] The following is a detailed description of a flexible sensing device and a lip reading recognition method based on the device according to the present invention in conjunction with the accompanying drawings and specific embodiments.

[0080] Example 1

[0081] In principle, the method proposed in the present invention can be applied to lip reading recognition in any daily scenario.

[0082] This embodiment describes the specific implementation of a flexible sensing device and a lip reading recognition method based on the device according to the present invention. Figure 1This is the composition of a flexible sensing device provided by an embodiment of the present invention and the lip reading recognition process based on the device. The flexible sensing recognition device includes an external power supply and a host computer, and also includes: a data acquisition unit, a transmission cable, an analog noise filtering module and an acquisition mainboard; the data acquisition unit is used to collect differential electromyographic signals; the transmission cable is used to transmit the collected differential electromyographic signals; the analog noise filtering module is used for double amplification, low-pass and band-pass filtering; the acquisition mainboard includes an analog-to-digital converter, a data preprocessing module, an input interface, a control module, a switch and a Bluetooth module; used to control analog-to-digital conversion, data preprocessing and wireless transmission; the data preprocessing module includes a channel amplitude adjustment unit, A high-pass filtering unit, a low-pass filtering unit and a normalization unit; the channel amplitude adjustment unit is connected to the high-pass filtering unit and the low-pass filtering unit, and the high-pass filtering unit and the low-pass filtering unit are connected to the normalization unit; used for software denoising; the host computer includes a feature extraction and screening unit, a machine learning classification module and a visualization module; the feature extraction and screening unit is connected to the machine learning classification module; used for data processing and visualization classification output; the machine learning classification module receives the extracted and screened features output by the feature extraction and screening unit, classifies the screened features, and outputs the classification results.

[0083] In specific implementation, the external power supply is connected to the input interface and switch of the acquisition mainboard to power the mainboard. The analog noise filtering module includes a first amplification unit, a low-pass filter, a second amplification unit and a high-pass filter; the first amplification unit is connected to the low-pass filter, and the low-pass filter is connected to the second amplification unit; used for hardware amplification and filtering.

[0084] The data acquisition unit includes several groups of electrode patches; the electrode patches are pre-gelled mesh electrode patches. In specific implementation, each of the several groups of electrode patches includes a positive electrode patch, a negative electrode patch and a reference electrode patch;

[0085] During specific implementation, the electrode patch is used to collect and acquire lip electromyographic signals. The collection is achieved through a number of electrodes fixed inside the mask. After collection, bioelectric signals with high signal-to-noise ratio and high conductivity are obtained. The bioelectric signals may also include facial muscle electrical signals and eye electrical signals during specific implementation. The high signal-to-noise ratio is greater than or equal to 25dB. During specific implementation, the signal-to-noise ratio is 27dB. The high conductivity is greater than or equal to 0.9×10-2S·cm-1. During specific implementation, the conductivity is 1×10-2S·cm-1.

[0086] The data acquisition unit includes several groups of electrode patches; the electrode patches are pre-gelled mesh electrode patches, and the patches are made of Ag / AgCl, AgCl, Ag, Au, carbon, paper-based, ceramic or nanometal; each of the several groups of electrode patches includes a positive electrode patch, a negative electrode patch and a reference electrode patch. The electrode patches are distributed on the orbicularis oris, masseter, labii inferioris, mylohyoid, sternohyoid, sternocleidomastoid, levator alar, and risorius / cheek muscles.

[0087] Preferably, the electrode patch is a MXene-based hydrogel electrode, which is prepared by MXene-based hydrogel, and the preparation process has the advantages of simple process, uniform MXene distribution and biocompatibility; it solves the fixation and contact problems of rigid electrodes in the process of collecting electrical signals of human facial muscles, and can more accurately identify electrical signals of human muscles;

[0088] When the data acquisition unit is collecting data, the reference electrode sheet is attached to the corresponding position to obtain the reference voltage at different collection points, specifically:

[0089] (1) The reference electrode is attached to the zygomatic bone to obtain the reference voltage of the buccinator muscle;

[0090] (2) The reference electrode is attached to the temporal bone to obtain the reference voltage of the depressor labii inferioris muscle;

[0091] (3) The reference electrode is attached to the clavicle to obtain the reference voltage of the mylohyoid muscle;

[0092] The reference voltage is a standard voltage that does not change with the muscle force at the acquisition point.

[0093] As another aspect of the present invention, a lip reading recognition method based on a flexible sensing recognition device is proposed, comprising the following steps:

[0094] S1: Deploy differential electrodes on several channels on the skin surface;

[0095] In specific implementation, the differential electrode arrangement described in S1 is specifically as follows: the two electrodes should be attached to different positions on the same muscle, and the reference electrode is attached to the clavicle.

[0096] The number of the plurality of channels in S1 is greater than or equal to 4; the plurality of channels correspond to at least 4 of the orbicularis oris, masseter, depressor labii inferioris, mylohyoid, sternohyoid, sternocleidomastoid, levator alaris, and risorius / cheek muscles.

[0097] S2: Collect data through the deployed differential electrodes to obtain muscle electrical signals;

[0098] S3: preprocessing the muscle electrical signal by analog filtering, amplification, analog-to-digital conversion and bandpass filtering to obtain preprocessed data;

[0099] The preprocessing can perform denoising filtering more effectively, extracting nearly pure muscle electrical signals from the chaotic state of the original signal; it includes two parts, analog preprocessing and digital preprocessing, the analog preprocessing includes low-pass filtering, high-pass filtering and two-stage amplification; the low-pass filtering and high-pass filtering can be exchanged; the multiples of the two-stage amplification can be flexibly adjusted; after the analog preprocessing, the analog preprocessed signal is converted into a digital signal, and then digital preprocessing including at least band-pass digital filtering and normalization is performed.

[0100] S4: normalize the preprocessed data and send the normalized data to the host computer;

[0101] S5: Extract and filter the features of the data received by the host computer to obtain the features to be classified;

[0102] The features to be classified include, but are not limited to, time domain, frequency domain, transform domain, entropy features and statistical features;

[0103] S6: Based on the features to be classified, a variety of machine learning classification algorithms are used to perform recognition and obtain recognition results;

[0104] S7: Compare and evaluate the performance of different algorithms and present them visually.

[0105] The feature extraction and classification learning algorithms in S5 and S6 can greatly improve the accuracy of target recognition and more effectively identify the features of muscle electrical signals.

[0106] The number of the plurality of channels in S1 is greater than or equal to 4; specifically in this embodiment, the number of the plurality of channels in S1 is 4; it may also be 3 channels;

[0107] The plurality of channels correspond to at least four of the orbicularis oris, masseter, depressor labii inferioris, mylohyoid, sternohyoid, sternocleidomastoid, levator alaris, and risorius / cheek muscles.

[0108] As a specific embodiment, the muscle distribution of the differential electrodes of several channels is specifically: selecting the levator alaris, zygomaticus minor, mylohyoid muscle, buccinator muscle, depressor labii inferioris muscle and platysma muscle.

[0109] The preprocessing in S3 specifically includes the following sub-steps:

[0110] S31: performing a first-stage amplification on the collected muscle electrical signal to obtain a first-stage amplified signal;

[0111] S32: performing low-pass filtering on the first-stage amplified signal to obtain a low-pass filtered signal;

[0112] S33: performing a second-stage amplification on the low-pass filtered signal to obtain a second-stage amplified signal;

[0113] S34: performing high-pass filtering on the secondary amplified signal to obtain an analog preprocessed signal;

[0114] S35: performing analog-to-digital conversion on the analog preprocessed signal to obtain a digital signal and subtracting the respective offset amplitudes from the digital signals of each channel;

[0115] S35 The offset amplitude is the average value of the digital signal amplitude of the corresponding channel;

[0116] S36: performing band-pass filtering on the digital signals of each channel after the bias amplitude is removed to obtain pre-processed data;

[0117] The low-pass cutoff frequency of the band-pass filter is greater than or equal to 55; the high-pass cutoff frequency is greater than or equal to 40;

[0118] S4: normalization, including but not limited to Min-Max normalization, Log function conversion and Z-Score method; sending the normalized data to the host computer is achieved through a Bluetooth module;

[0119] S31 The magnification of the first stage amplification is 5 to 50; S33 The magnification of the second stage amplification is 20 to 100; The low-pass filter and the high-pass filter are exchanged. In specific implementation, the first stage amplifies 30 or 50 times, and the second stage amplifies 20 times; in another embodiment, the first stage amplifies 10 times, and the second stage amplifies 100 times;

[0120] The high-pass filter is a 4th-order Butterworth filter, a Chebyshev filter or a Bessel filter.

[0121] The feature extraction and screening described in S5 specifically includes the following steps:

[0122] S51: extracting various features from the data received from the host computer;

[0123] The various features include, but are not limited to, time domain features, frequency domain features, transform domain features, entropy features and statistical features;

[0124] In specific implementation, the time domain features include: average absolute value, number of zero crossings and average waveform length; the average absolute value is the average amplitude of muscle electricity in a short period of time, reflecting the activity level and strength of the muscle; the average waveform length is a characteristic parameter used to characterize the smoothness of the signal and the degree of waveform change; the number of zero crossings is:

[0125] The number of times a signal crosses zero over a period of time;

[0126] The statistical characteristics include but are not limited to mean, root mean square value and variance;

[0127] The mean is the average value of muscle electricity;

[0128] Variance represents the degree of deviation of the signal;

[0129] The RMS value reflects the average power of muscle electricity generated during muscle movement during speech;

[0130] The frequency domain features include but are not limited to: average power frequency and median frequency;

[0131] The average power frequency is the centroid frequency of the power spectrum; the median frequency is the frequency that divides the power spectrum into two parts of equal power;

[0132] The transform domain features include but are not limited to: wavelet transform, discrete cosine transform and Hilbert-Huang transform; the wavelet transform uses db7, bior2.4 or Haar wavelet basis;

[0133] In the specific implementation, bior2.4 wavelet basis is selected;

[0134] The entropy features include but are not limited to: differential entropy, relative entropy, joint entropy and Rayleigh entropy; in specific implementation, differential entropy and joint entropy are selected; the differential entropy is used to measure signal uncertainty;

[0135] The relative entropy is used to calculate the asymmetry between signal channels;

[0136] The joint entropy is a measure of uncertainty in the joint distribution of two random variables;

[0137] The Rayleigh entropy is used to measure signal complexity;

[0138] S52: Filter important features for lip reading classification based on various features to obtain features to be classified.

[0139] The screening described in S52 screens the various features extracted in S51 through the KPCA algorithm, the mutual information method based on MIC, the distance correlation coefficient method or the random forest contribution method to obtain the features to be classified.

[0140] S7 The visual display includes visual display of the data received by the host computer, the features to be classified, and the recognition results;

[0141] The data received by the host computer is visualized, specifically: drawing the data received by the host computer, including but not limited to drawing a time domain diagram, a frequency domain diagram, and a time-frequency diagram;

[0142] Visualize the features to be classified, specifically: draw a graph of the results of feature extraction and screening, i.e., the features to be classified, and draw a bar chart of the order of feature importance;

[0143] The recognition results are visualized, specifically: the classification results of S6 using multiple machine learning classification algorithms are plotted, including but not limited to plotting accuracy comparison charts and confusion matrices of each classifier; specifically in this embodiment, the recognition results are visualized to display the recognized words.

[0144] The multiple machine learning classification algorithms described in S6 include, but are not limited to: SVM, KNN, linear classification, MLP, ensemble learning and CNN. In specific implementation, the machine learning classification algorithm selects MLP.

[0145] The data acquisition unit is connected to the transmission cable; the transmission cable is connected to the analog noise filtering module; the analog noise filtering module is connected to the acquisition mainboard; the external power supply is connected to the acquisition mainboard; the acquisition mainboard is connected to the host computer;

[0146] The analog-to-digital converter is connected to the channel amplitude adjustment unit of the data preprocessing module; the normalization unit of the data preprocessing module is connected to the input interface, the input interface is connected to the control module, the control module is connected to the switch, and the switch is connected to the Bluetooth module;

[0147] The data preprocessing module receives the digital signal converted by the analog-to-digital converter, performs digital filtering and normalization, and outputs the sample signal to the input interface of the acquisition mainboard;

[0148] The acquisition mainboard receives the signal output by the analog noise filtering module, performs analog-to-digital conversion, software filtering, normalization, Bluetooth transmission and power supply control, and outputs the data to the host computer; the host computer receives the data transmitted by the acquisition mainboard, performs feature extraction and screening, machine learning classification and visualization;

[0149] The machine learning classification module includes several classification models, receives the data after feature extraction and screening, performs result prediction, and outputs classification results;

[0150] The amplification factor of the first amplifying unit is 5 to 50; the amplification factor of the second amplifying unit is 20 to 100; in specific implementation: the amplification factor of the first amplifying unit is 10 times, and the low-pass filter uses a cutoff frequency of 500Hz, so that signals below 500Hz pass through; the amplification factor of the second amplifying unit is 68; the high-pass filter uses a cutoff frequency of 20Hz, and the sample signal obtained after the high-pass filter is used for subsequent processing; the low-pass filter can also exchange positions with the high-pass filter; or a band-pass filter is used, with a high-pass cutoff frequency of 20Hz and a low-pass cutoff frequency of 500Hz; and signals from 20 to 500Hz are retained.

[0151] The lip reading recognition method is specifically implemented as follows: before collecting, differential electrodes are arranged on the skin surface, specifically at the following positions: Figure 2 3, 7 and 9 shown in the figure, collect the muscle electrical signals of the depressor labii inferioris, levator alaris and platysma, and the reference electrode is arranged at the clavicle; the transmission cable is 3 3.5mm wires connected to electrode buckles; the electrodes are self-designed buckle-type flexible electrode patches such as Figure 3 As shown; it is assembled using a custom circuit board. A total of 15 words were read during the collection: go to the bathroom, I want to eat, cover myself with a blanket, I want to drink water, help me, I'm so happy, thank you, you're welcome, I'll do it myself, go forward, step back, turn on the air conditioner, open the window, call him, bring it over;

[0152] The preprocessing in S3 specifically includes the following sub-steps:

[0153] S31: performing a first-stage amplification on the collected muscle electrical signal to obtain a first-stage amplified signal;

[0154] S32: performing low-pass filtering on the first-stage amplified signal to obtain a low-pass filtered signal;

[0155] S33: performing a second-stage amplification on the low-pass filtered signal to obtain a second-stage amplified signal;

[0156] S34: performing high-pass filtering on the secondary amplified signal to obtain an analog preprocessed signal;

[0157] S35: performing analog-to-digital conversion on the analog preprocessed signal to obtain a digital signal and subtracting the respective offset amplitudes from the digital signals of each channel;

[0158] S35 The offset amplitude is the average value of the digital signal amplitude of the corresponding channel;

[0159] S36: Perform bandpass filtering on the digital signals of each channel after the bias amplitude is removed to obtain preprocessed data. The low-pass cutoff frequency of the bandpass filtering in S36 is greater than or equal to 55; the high-pass cutoff frequency is greater than or equal to 40;

[0160] The normalization in S4 includes but is not limited to Min-Max normalization, Log function conversion and Z-Score method; the sending of normalized data to the host computer is achieved through a Bluetooth module.

[0161] The magnification of the first stage amplification is 5 to 50, and the magnification of the second stage amplification is 20 to 100. The high-pass filter is a 4th-order Butterworth filter, a Chebyshev filter or a Bessel filter, and the low-pass filter and the high-pass filter are exchanged.

[0162] In the specific implementation, the analog preprocessing from S31 to S34 is as follows: there is obvious noise in the acquisition environment, so the collected muscle electrical signals need to be preprocessed, including analog preprocessing (analog preprocessing is implemented by analog noise filtering module): the collected muscle electrical signals are amplified in the first stage, with an amplification factor of 10 times; the first-stage amplified signal is filtered through a 500Hz low-pass filter; the low-pass filtered signal is amplified in the second stage, with an amplification factor of 68 times; the second-stage amplified signal is filtered through a 10Hz high-pass filter; the two amplifier models are M258 operational amplifiers; the high-pass filter model is LPGE-592R; the low-pass filter model is LTC-1563-2;

[0163] The digital preprocessing comprises the following steps:

[0164] S35: performing analog-to-digital conversion on the analog preprocessed signal to obtain a digital signal and subtracting the respective offset amplitudes from the digital signals of each channel; the offset amplitude in S35 is the average amplitude of the digital signal of the corresponding channel;

[0165] In specific implementation: the analog-to-digital conversion of the analog preprocessed signal is performed through the domestic low-power analog-to-digital converter HXT9643 to obtain a digital signal; the DC bias caused by the potential difference between the differential electrode and the reference electrode is subtracted from the digital signal of each channel;

[0166] S36: performing band-pass filtering on the digital signals of each channel after the bias amplitude is removed to obtain pre-processed data;

[0167] In the specific implementation, the digital signal is subjected to 20Hz high-pass filtering through a 4th-order Butterworth filter; a band-pass filter with a center frequency of 45Hz-55Hz is used to remove the signal power frequency noise and power frequency interference; the filtered signal is Z-Score normalized; the pre-processed data is output to the input interface of the acquisition board;

[0168] Connect the external power supply and turn on the acquisition mainboard switch, control the input interface to transfer data through the STM32F407VET6 control module and send data to the host computer through the HC-06 Bluetooth module; the external power supply uses Yuanyang FLY.103450.2000 lithium battery, and is equipped with a wireless charging transmitter and receiver for wireless charging;

[0169] Extract the average amplitude, root mean square value, average waveform length, variance of the time domain features of each channel and the average power frequency and median frequency of the frequency domain features;

[0170] Based on the KPCA algorithm, the important features of lip reading classification are selected as the input of the machine learning algorithm; the Gaussian kernel SVM, KNN and CNN algorithms are used to identify the selected features, and the performance of different algorithms is compared and evaluated; the data analysis results are visualized. The specific steps are as follows:

[0171] Draw a time domain graph for the data received by the feature extraction and screening unit; draw a feature importance ranking bar graph for the screening results output by the feature extraction and screening unit; draw an accuracy comparison graph and confusion matrix of each classifier for the machine learning classification results;

[0172] Finally, the classifier with the highest accuracy is selected as the classifier for lip reading recognition application, and the words recognized by lip reading are visualized by the host computer.

[0173] The comparison of the highest recognition rates of the three classifiers is shown in Table 1:

[0174] Table 1 Comparison of classifier performance

[0175] Classifier CNN KNN SVM Accuracy 82.3％ 87.5％ 84.0％

[0176] By using three algorithms for identification, the KNN algorithm has a high accuracy rate of 87.5%;

[0177] At this point, the specific implementation of the flexible sensing device and the lip reading recognition method based on the device described in the present invention has been completed.

[0178] The above is only a preferred embodiment of the present invention, and the present invention should not be limited to the contents disclosed in the embodiment and the drawings. Any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A flexible sensing and recognition device, comprising an external power supply and a host computer, characterized in that: Also includes: Data acquisition unit, transmission cable, analog noise filtering module and acquisition mainboard; The data acquisition unit is used to collect differential electromyographic signals; The transmission cable is used to transmit the collected differential electromyographic signal; The analog noise filtering module is used for double amplification, low-pass and band-pass filtering; the acquisition mainboard includes an analog-to-digital converter, a data preprocessing module, an input interface, a control module, a switch and a Bluetooth module; it is used to control analog-to-digital conversion, data preprocessing and wireless transmission; the data preprocessing module includes a channel amplitude adjustment unit, a high-pass filtering unit, a low-pass filtering unit and a normalization unit; the channel amplitude adjustment unit is connected to the high-pass filtering unit and the low-pass filtering unit, and the high-pass filtering unit and the low-pass filtering unit are connected to the normalization unit; it is used for software denoising; the host computer includes a feature extraction and screening unit, a machine learning classification module and a visualization module; the feature extraction and screening unit is connected to the machine learning classification module; it is used for data processing and visualization classification output; the machine learning classification module receives the extracted and screened features output by the feature extraction and screening unit, classifies the screened features, and outputs the classification results.

2. A flexible sensing and identification device according to claim 1, characterized in that: The analog noise filtering module includes: a first amplifying unit, a low-pass filter, a second amplifying unit and a high-pass filter; the first amplifying unit is connected to the low-pass filter, and the low-pass filter is connected to the second amplifying unit; and is used for hardware amplification and filtering.

3. A flexible sensing and identification device according to claim 1, characterized in that: The data acquisition unit comprises: a plurality of groups of electrode patches; the electrode patches are pre-gelled mesh electrode patches; each of the plurality of groups of electrode patches comprises a positive electrode patch, a negative electrode patch and a reference electrode patch.

4. A flexible sensing and identification device according to claim 3, characterized in that: The electrode patches are distributed on the orbicularis oris muscle, masseter muscle, labii inferioris muscle, mylohyoid muscle, sternohyoid muscle, sternocleidomastoid muscle, levator alaris muscle, and risorius muscle / cheek muscle.

5. A lip reading recognition method based on a flexible sensing recognition device, characterized in that: The steps include: S1: Deploy differential electrodes on several channels on the skin surface; S2: Collect data through the deployed differential electrodes to obtain muscle electrical signals; S3: preprocessing the muscle electrical signal by analog filtering, amplification, analog-to-digital conversion and bandpass filtering to obtain preprocessed data; S4: normalizing the preprocessed data and sending the normalized data to a host computer; S5: Extract and filter the features of the data received by the host computer to obtain the features to be classified; The features to be classified include, but are not limited to, time domain, frequency domain, transform domain, entropy features and statistical features; S6: Based on the features to be classified, use a variety of machine learning classification algorithms to perform recognition and obtain recognition results; S7: Compare and evaluate the performance of different algorithms and present them visually.

6. The lip reading recognition method based on the flexible sensing recognition device according to claim 5, characterized in that: The number of the plurality of channels in S1 is greater than or equal to 4; the plurality of channels correspond to at least four of the orbicularis oris, masseter, depressor labii inferioris, mylohyoid, sternohyoid, sternocleidomastoid, levator alaris, and risorius / cheek muscles.

7. The lip reading recognition method based on the flexible sensing recognition device according to claim 5, characterized in that: The preprocessing in S3 specifically includes the following sub-steps: S31: performing a first-stage amplification on the collected muscle electrical signal to obtain a first-stage amplified signal; S32: performing low-pass filtering on the first-stage amplified signal to obtain a low-pass filtered signal; S33: performing a second-stage amplification on the low-pass filtered signal to obtain a second-stage amplified signal; S34: performing high-pass filtering on the secondary amplified signal to obtain an analog preprocessed signal; S35: performing analog-to-digital conversion on the analog preprocessed signal to obtain a digital signal and subtracting the respective offset amplitudes from the digital signals of each channel; S35 The offset amplitude is the average value of the digital signal amplitude of the corresponding channel; S36: performing band-pass filtering on the digital signals of each channel after the bias amplitude is removed to obtain pre-processed data. S36: the low pass cut-off frequency of the band pass filter is greater than or equal to 55; The high-pass cutoff frequency is greater than or equal to 40.

8. The lip reading recognition method based on the flexible sensing recognition device according to claim 5, characterized in that: The normalization in S4 includes but is not limited to Min-Max normalization, Log function conversion and Z-Score method; the sending of normalized data to the host computer is achieved through a Bluetooth module.

9. The lip reading recognition method based on the flexible sensing recognition device according to claim 7, characterized in that: The magnification of the first level amplification is 5 to 50; the magnification of the second level amplification is 20 to 100.

10. The lip reading recognition method based on the flexible sensing recognition device according to claim 7, characterized in that: The high-pass filter is a 4th-order Butterworth filter, a Chebyshev filter or a Bessel filter, and the low-pass filter and the high-pass filter are exchanged.

Images