Lip reading recognition method, device and computer readable storage medium

Abstract

The present invention discloses a lip reading recognition method, device and computer-readable storage medium, wherein the lip reading recognition method comprises: acquiring detection data of an ultrasonic detection array; determining the relative distance between each detection point according to the detection data; generating an ultrasonic detection distance matrix according to the relative distance; acquiring lip reading data associated with a pre-stored matrix matching the ultrasonic detection distance matrix as target lip reading data. The present invention aims to increase the diversity of the ways in which a device acquires user expression information.

Description

Technical Field

[0001] The present invention relates to the field of information recognition, and in particular to a lip reading recognition method, device and computer-readable storage medium. Background Art

[0002] With the development of the Internet era, people can communicate with each other without meeting each other, for example, by collecting the user's voice through MIC (microphone) or by typing. However, due to the unfortunate physical defects of some disabled people, the voice of people with speech impairments cannot be collected through MIC. People with hearing impairments cannot know the voice collected by others with MIC, and related equipment of MIC cannot be used normally by disabled people. Therefore, the way to obtain user expression information is too simple.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a lip reading recognition method, device and computer-readable storage medium, aiming to achieve the effect of increasing the diversity of the way in which the device obtains user expression information.

[0005] To achieve the above object, the present invention provides a lip reading recognition method, the lip reading recognition method comprising:

[0006] Acquiring detection data of an ultrasonic detection array;

[0007] Determine the relative distance between each detection point according to the detection data;

[0008] generating an ultrasonic detection distance matrix according to the relative distance;

[0009] The lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix is ​​obtained as the target lip reading data.

[0010] Optionally, before the step of acquiring the lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix, the step further includes:

[0011] Determine the identity information of the user to be identified;

[0012] Acquire a pre-stored matrix library corresponding to the user to be identified according to the identity information;

[0013] A pre-stored matrix matching the ultrasonic detection distance matrix is ​​determined in the pre-stored matrix library.

[0014] Optionally, the step of acquiring a pre-stored matrix library corresponding to the user to be identified according to the identity information includes:

[0015] If the pre-stored matrix library corresponding to the user to be identified is not obtained, instructing the user to be identified to perform a preset action;

[0016] When the user to be identified performs the preset action, confirming the pre-stored matrix corresponding to the preset action through the ultrasonic detection array;

[0017] The lip reading data corresponding to the preset action is associated with the pre-stored matrix and saved as a pre-stored matrix library, and the pre-stored matrix library is associated with the identity information of the user to be identified and saved.

[0018] Optionally, the step of determining the identity information of the user to be identified includes:

[0019] Obtaining biometric features of the user to be identified;

[0020] The identity information of the user to be identified is determined according to the biometric feature.

[0021] Optionally, the step of acquiring detection data of the ultrasonic detection array includes:

[0022] Controlling the ultrasonic detection unit of the ultrasonic detection array to transmit an ultrasonic signal to the detection point and receiving an echo signal of the ultrasonic signal;

[0023] The position information of each detection point is determined according to the echo signal.

[0024] Optionally, the step of determining the relative distance between each detection point according to the detection data includes:

[0025] The relative distances between the detection points are determined according to the position information.

[0026] Optionally, the step of acquiring lip reading data associated with a pre-stored matrix matching the ultrasonic detection distance matrix comprises:

[0027] Determining a pre-stored matrix that matches the ultrasonic detection distance matrix;

[0028] Obtaining pre-stored data corresponding to the pre-stored matrix;

[0029] The lip reading data is determined according to the pre-stored data.

[0030] Optionally, before the step of acquiring the detection data of the ultrasonic detection array, the step further includes:

[0031] When receiving the lip reading recognition instruction, starting the lip reading recognition mode and executing the step of acquiring the detection data of the ultrasonic detection array;

[0032] When receiving the voice collection instruction, switch the lip reading recognition mode to the voice mode and call the microphone to obtain the voice.

[0033] In addition, to achieve the above-mentioned purpose, the present invention also provides a lip reading recognition device, which includes a memory, a processor, and a lip reading recognition program stored in the memory and executable on the processor, and the lip reading recognition program implements the steps of the lip reading recognition method described above when executed by the processor.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a lip reading recognition program is stored, and when the lip reading recognition program is executed by a processor, the steps of the lip reading recognition method described above are implemented.

[0035] The embodiment of the present invention proposes a lip reading recognition method, device and computer-readable storage medium, which first obtains the detection data of the ultrasonic detection array; determines the relative distance between each detection point according to the detection data; generates an ultrasonic detection distance matrix according to the relative distance; and obtains the lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix as the target lip reading data. In this way, the relative distance between each detection point is determined by the detection data of the ultrasonic detection array, and the ultrasonic detection distance matrix is ​​generated, and the lip reading data managed by the pre-stored matrix matching the ultrasonic detection distance matrix is ​​obtained as the target lip reading data for processing. In this way, the information that the user wants to express can be determined by the target lip reading data, which is a completely different way from obtaining the user's expression information by collecting voice, and increases the diversity of the way the device obtains the user's expression information. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a flow chart of an embodiment of a lip reading recognition method of the present invention;

[0038] Figure 3 FIG. 4 is a flow chart of another embodiment of the lip reading recognition method of the present invention.

[0039] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0041] Because in related technologies, the method of obtaining voice through MIC is not suitable for people with hearing and speech disabilities, people with disabilities cannot use MIC to collect sound and obtain the sound collected by MIC, and cannot normally obtain the expression information of people with disabilities. The method of obtaining user expression information in traditional technologies is too single.

[0042] In order to increase the diversity of the ways in which a device obtains user expression information, the embodiment of the present invention proposes a lip reading recognition method, device and computer-readable storage medium, wherein the main steps of the method include:

[0043] Acquiring detection data of an ultrasonic detection array;

[0044] Determine the relative distance between each detection point according to the detection data;

[0045] generating an ultrasonic detection distance matrix according to the relative distance;

[0046] The lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix is ​​obtained as the target lip reading data.

[0047] In this way, the relative distance between each detection point is determined by the detection data of the ultrasonic detection array, and an ultrasonic detection distance matrix is ​​generated. The lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix is ​​obtained as the target lip reading data. The target lip reading data can obtain the user's expression information without using sound. Therefore, the diversity of the way in which the device obtains the user's expression information can be increased.

[0048] The content to be protected by the claims of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0050] The terminal in the embodiment of the present invention may be a lip reading recognition device.

[0051] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a memory 1003, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The memory 1003 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1003 may also be a storage device independent of the aforementioned processor 1001.

[0052] Those skilled in the art will understand that Figure 1The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0053] like Figure 1 As shown, the memory 1003 as a computer storage medium may include an operating system and a lip reading recognition program.

[0054] exist Figure 1 In the terminal shown, the processor 1001 can be used to call the lip reading recognition program stored in the memory 1003 and perform the following operations:

[0055] Acquiring detection data of an ultrasonic detection array;

[0056] Determine the relative distance between each detection point according to the detection data;

[0057] generating an ultrasonic detection distance matrix according to the relative distance;

[0058] The lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix is ​​obtained as the target lip reading data.

[0059] Furthermore, the processor 1001 may call the lip reading recognition program stored in the memory 1003, and further perform the following operations:

[0060] Determine the identity information of the user to be identified;

[0061] Acquire a pre-stored matrix library corresponding to the user to be identified according to the identity information;

[0062] A pre-stored matrix matching the ultrasonic detection distance matrix is ​​determined in the pre-stored matrix library.

[0063] Furthermore, the processor 1001 may call the lip reading recognition program stored in the memory 1003, and further perform the following operations:

[0064] If the pre-stored matrix library corresponding to the user to be identified is not obtained, instructing the user to be identified to perform a preset action;

[0065] When the user to be identified performs the preset action, confirming the pre-stored matrix corresponding to the preset action through the ultrasonic detection array;

[0066] The lip reading data corresponding to the preset action is associated with the pre-stored matrix and saved as a pre-stored matrix library, and the pre-stored matrix library is associated with the identity information of the user to be identified and saved.

[0067] Furthermore, the processor 1001 may call the lip reading recognition program stored in the memory 1003, and further perform the following operations:

[0068] Obtaining biometric features of the user to be identified;

[0069] The identity information of the user to be identified is determined according to the biometric feature.

[0070] Furthermore, the processor 1001 may call the lip reading recognition program stored in the memory 1003, and further perform the following operations:

[0071] Controlling the ultrasonic detection unit of the ultrasonic detection array to transmit an ultrasonic signal to the detection point and receiving an echo signal of the ultrasonic signal;

[0072] The position information of each detection point is determined according to the echo signal.

[0073] Furthermore, the processor 1001 may call the lip reading recognition program stored in the memory 1003, and further perform the following operations:

[0074] The relative distances between the detection points are determined according to the position information.

[0075] Furthermore, the processor 1001 may call the lip reading recognition program stored in the memory 1003, and further perform the following operations:

[0076] Determining a pre-stored matrix that matches the ultrasonic detection distance matrix;

[0077] Obtaining pre-stored data corresponding to the pre-stored matrix;

[0078] The lip reading data is determined according to the pre-stored data.

[0079] Furthermore, the processor 1001 may call the lip reading recognition program stored in the memory 1003, and further perform the following operations:

[0080] When receiving the lip reading recognition instruction, starting the lip reading recognition mode and executing the step of acquiring the detection data of the ultrasonic detection array;

[0081] When receiving the voice collection instruction, switch the lip reading recognition mode to the voice mode and call the microphone to obtain the voice.

[0082] With the development of the Internet era, people can communicate with each other without meeting each other, for example, by collecting the user's voice through MIC (microphone) or by typing. However, due to the unfortunate physical defects of some disabled people, the voice of people with speech impairments cannot be collected through MIC, and people with hearing impairments cannot know the voice collected by others with MIC. Therefore, MIC-related equipment cannot be used normally by disabled people.

[0083] In addition, collecting user voices through MIC will inevitably collect environmental noise. If the voice needs to be processed later, such as voice-to-text conversion, the recognition result will be wrong and the user's expression information will be inaccurate. Therefore, the method of obtaining user expression information is too simple.

[0084] Therefore, it can be seen that the above defects exist in the related expression information acquisition method. To solve the above defects, the embodiment of the present invention proposes a lip reading recognition method, which aims to determine the expression information by recognizing the target lip reading information, and increase the diversity of the way the device obtains the user's expression information.

[0085] The following is an explanation of the content claimed in the claims of the present invention through specific exemplary schemes, so that those skilled in the art can better understand the scope of protection of the claims of the present invention. It is understood that the following exemplary schemes do not limit the scope of protection of the present invention, but are only used to explain the present invention.

[0086] For example, refer to Figure 2 In one embodiment of the lip reading recognition method of the present invention, the lip reading recognition method comprises the following steps:

[0087] Step S10, acquiring detection data of the ultrasonic detection array;

[0088] In this embodiment, the main body executing the lip reading recognition method can be a terminal device, including a head-mounted device, a mobile phone, a tablet, a computer, etc. The terminal device includes an ultrasonic detection array, the ultrasonic detection array includes a plurality of ultrasonic detection units, each ultrasonic detection unit includes at least one ultrasonic transmitter, the ultrasonic detection array transmits an ultrasonic signal to a target detection area, and determines the detection data by receiving the arriving echo signal. The detection data can be the relative position relationship between the detection point in the target detection area and the ultrasonic detection array or the ultrasonic detection unit. The target detection area at least includes the lip area of ​​the user to be identified.

[0089] Optionally, the ultrasonic detection units of the ultrasonic detection array are controlled to transmit ultrasonic signals to the detection points, and echo signals of the ultrasonic signals are received; and the position information of each detection point is determined according to the echo signals.

[0090] The ultrasonic detection unit is controlled to transmit an ultrasonic signal to a detection point, receive an echo signal of the ultrasonic signal, and determine the position information of each detection point according to the echo information. Each ultrasonic detection unit of the ultrasonic detection array transmits an ultrasonic signal to each detection point, and receives an echo signal of the ultrasonic signal. Each ultrasonic detection unit may include an ultrasonic transmitter and an ultrasonic receiver. The ultrasonic receiver correspondingly receives the echo signal of the ultrasonic signal sent by the ultrasonic transmitter of the ultrasonic detection unit. The echo signal may also be uniformly received by a receiver. The position information of each detection point may be determined according to the reception time of the echo signal. Due to the difference in structure and material, the location of the detection point may also be determined according to the signal strength of the super echo signal.

[0091] Optionally, when a lip reading recognition instruction is received, the lip reading recognition mode is started and the step of acquiring the detection data of the ultrasonic detection array is executed; when a voice acquisition instruction is received, the lip reading recognition mode is switched to the voice mode and the microphone is called to acquire the voice.

[0092] When a preset condition is met, the lip reading recognition mode is started and the lip reading recognition method is executed. The preset condition can be that the device is turned on or a lip reading recognition instruction is received. When the device is turned on or a lip reading recognition instruction is received, the lip reading recognition mode is started and the lip reading recognition method is executed. In the process of executing the lip reading recognition method, the lip reading recognition mode can also be converted to a voice mode when a preset condition is met, and the surrounding voice is acquired through a microphone. The preset condition is that a voice acquisition instruction is received.

[0093] Step S20, determining the relative distance between each detection point according to the detection data;

[0094] In this embodiment, the target detection area of ​​the ultrasonic detection array includes at least the lip area. There are multiple detection points on the lip area. The detection points can be points on the lip contour or array points set when the lips are in the original posture. When the mouth shape changes, the position of the array point will change, and the position of the array point after the change is the detection point.

[0095] Optionally, the relative distance between each detection point is determined according to the position information of each detection point.

[0096] The detection data can be the position information between each detection point, or it can be an ultrasonic detection image. The relative distance between each detection point can be determined based on the detection data of the ultrasonic detection array. If the detection data is the position information of each detection point, the detection point is pre-set, and the relative distance between each detection point is determined based on the position information of the detection point. If the detection data is an ultrasonic detection image, the detection point is calibrated on the ultrasonic detection image to determine the relative distance between each detection point.

[0097] Step S30, generating an ultrasonic detection distance matrix according to the relative distance;

[0098] In this embodiment, an ultrasonic detection distance matrix is ​​generated based on the same distance between each detection point. When the human body makes different lip movements, the mouth shape is different, the relative distance between each detection point on the lip area is different, and the generated ultrasonic detection distance matrix is ​​also different.

[0099] Step S40: Acquire lip reading data associated with a pre-stored matrix that matches the ultrasonic detection distance matrix as target lip reading data.

[0100] In this embodiment, the pre-stored matrix is ​​a pre-stored ultrasonic detection distance matrix, and the corresponding lip reading movements and lip reading data are known. The ultrasonic detection distance corresponding to the same lip reading movement is the same, so the lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix can be used as the target lip reading data of the user to be identified. The lip reading data can be voice, lip reading image and / or text, and the target lip reading data can be the lip reading data of the user to be identified, including voice, lip reading image and / or text.

[0101] Optionally, a pre-stored matrix matching the ultrasonic detection distance matrix is ​​determined; pre-stored data corresponding to the pre-stored matrix is ​​acquired; and the lip reading data is determined according to the pre-stored data.

[0102] First, a pre-stored matrix matching the ultrasonic detection distance matrix is ​​determined from a pre-stored matrix library. The pre-stored matrix library stores the pre-stored matrix and the pre-stored data in association. The pre-stored data may include identity information, lip reading data, etc. After the pre-stored matrix is ​​determined, its associated pre-stored data may be obtained, and the lip reading data is determined from the pre-stored data as the target lip reading data. The pre-stored matrix library may be a database corresponding to the user to be identified, wherein the pre-stored matrix is ​​the ultrasonic detection distance matrix collected by the user to be identified before performing a preset action. The lip reading data, pre-stored matrix and identity information management corresponding to the preset action are stored in the pre-stored matrix library. The preset matrix library may also be the ultrasonic detection distance matrix collected by other users performing preset actions. The lip reading data and pre-stored matrix corresponding to the preset action are stored in the pre-stored matrix library.

[0103] In the technical solution disclosed in the present embodiment, detection data of an ultrasonic detection array is acquired; the relative distance between each detection point is determined based on the detection data; an ultrasonic detection distance matrix is ​​generated based on the relative distance; and lip reading data associated with a pre-stored matrix matching the ultrasonic detection distance matrix is ​​taken as target lip reading data. In this way, the relative distance between each detection point is determined through the detection data of the ultrasonic detection array and an ultrasonic detection distance matrix is ​​constructed. The lip reading data of the preset matrix matching the ultrasonic detection distance matrix is ​​taken as target lip reading data to express the user's meaning. Compared with the method of collecting voice, the method provided in the present embodiment has lower requirements on the use environment and the user, and achieves the effect of increasing the diversity of the way the device obtains user expression information.

[0104] Optionally, refer to Figure 3 Based on any of the above embodiments, in another embodiment of the lip reading recognition method of the present invention, the lip reading recognition method further comprises:

[0105] Step S50: determining the identity information of the user to be identified;

[0106] In this embodiment, the user to be identified is a user who can be detected in the target detection area of ​​the ultrasonic detection array. The lip area of ​​the user to be identified is in the target detection area. According to the detection data of the ultrasonic detection array including the detection data of the lip area, the target lip reading data is the lip reading data of the user to be identified. The expression information of the user to be identified can be determined by the target lip reading data.

[0107] Optionally, biometric features of a user to be identified are obtained; and identity information of the user to be identified is determined based on the biometric features.

[0108] Biometric features include voiceprints, fingerprints, facial information, etc. Biometric features can be obtained through independent identification devices, such as fingerprint identification devices, camera devices, etc. to obtain the biometric features of the user to be identified. The biometric features of the user to be identified can also be determined through an ultrasonic detection array. The ultrasonic detection array can determine the detection points of the human face or body, and then determine the relative distance between each detection point. The facial features or body features of the human body can be determined based on the relative distance. The identity information of the user to be identified can be determined based on the above biometric features.

[0109] Step S60, obtaining a pre-stored matrix library corresponding to the user to be identified according to the identity information;

[0110] The pre-stored matrix library corresponding to the user to be identified is obtained according to the identity information. The pre-stored matrix library associates and stores pre-stored data and pre-stored matrices. The pre-stored matrix is ​​an ultrasonic detection distance matrix detected when the user to be identified made lip reading movements in the past. The pre-stored data includes the lip reading data corresponding to the lip reading movements.

[0111] Optionally, if the pre-stored matrix library corresponding to the user to be identified is not obtained, the user to be identified is instructed to perform a preset action; when the user to be identified performs the preset action, the pre-stored matrix corresponding to the preset action is confirmed through the ultrasonic detection array; the lip reading data corresponding to the preset action and the pre-stored matrix are associated and saved as a pre-stored matrix library, and the pre-stored matrix library is associated and saved with the identity information of the user to be identified.

[0112] If the preset matrix library corresponding to the user to be identified is not obtained, it is considered that the user to be identified has not established a preset matrix library or the preset matrix library is damaged. In this case, it is necessary to re-establish the preset matrix library of the user to be identified, and instruct the user to be identified to perform a preset action, which includes a lip reading action. The lip reading data of the preset action is determined. When the user performs the preset action, the detection data of the user when performing the preset action is detected by an ultrasonic detection array, and the relative distance between each detection point is determined according to the detection data. An ultrasonic detection distance matrix is ​​established as a preset matrix according to the relative distance, and the lip reading data is saved as a pre-stored matrix library as pre-stored data and pre-stored matrix management, and the pre-stored matrix library is associated with the identity information of the user to be identified and saved.

[0113] Step S70: Determine a pre-stored matrix in the pre-stored matrix library that matches the ultrasonic detection distance matrix.

[0114] In this embodiment, a pre-stored matrix that matches the ultrasonic detection distance matrix in a pre-stored matrix library corresponding to the user to be identified and pre-stored data associated with the pre-stored matrix are used as the target lip reading data of the user to be identified.

[0115] In the technical solution disclosed in this embodiment, the identity information of the user to be identified is determined; the pre-stored matrix library corresponding to the user to be identified is obtained according to the identity information; and the pre-stored matrix matching the ultrasonic detection distance matrix is ​​determined in the pre-stored matrix library. The pre-stored matrix library is established according to the user to be identified, reducing the influence of the habits and facial features of the user to be identified on the matching results, so that the target lip reading data to be identified is more accurate, and the accuracy of lip reading recognition is improved.

[0116] In addition, an embodiment of the present invention further proposes a lip reading recognition device, which includes a memory, a processor, and a lip reading recognition program stored in the memory and executable on the processor. When the lip reading recognition program is executed by the processor, the steps of the lip reading recognition method described in the above embodiments are implemented.

[0117] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a lip reading recognition program is stored. When the lip reading recognition program is executed by a processor, the steps of the lip reading recognition method described in the above embodiments are implemented.

[0118] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0119] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling the lip reading recognition device to execute the methods described in each embodiment of the present invention.

[0121] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A lip reading recognition method, characterized in that: The lip reading recognition method comprises: Acquiring detection data of an ultrasonic detection array; Controlling the ultrasonic detection units of the ultrasonic detection array to transmit ultrasonic signals to each detection point in the target area; receiving an echo signal of the ultrasonic signal; Determining the position information of each of the detection points according to the signal strength or reception time of the echo signal; According to the position information of each detection point, the relative distance between each detection point is determined, the target detection area of ​​the ultrasonic detection array is the lip area, there are multiple detection points on the lip area, the detection points include lip contour points, or array points set by the original lip posture, when the lip shape changes, the position of the array point will change, and the position of the array point after the change is the detection point; generating an ultrasonic detection distance matrix according to the relative distance; Obtain lip reading data associated with a pre-stored matrix that matches the ultrasonic detection distance matrix as target lip reading data, wherein a pre-stored matrix that matches the ultrasonic detection distance matrix is ​​determined; pre-stored data corresponding to the pre-stored matrix is ​​obtained; and the lip reading data is determined based on the pre-stored data, wherein the target lip reading data includes voice, lip reading image and / or text.

2. The lip reading recognition method according to claim 1, characterized in that: Before the step of acquiring the lip reading data associated with the pre-stored matrix matching the ultrasonic detection distance matrix, the method further includes: Determine the identity information of the user to be identified; Acquire a pre-stored matrix library corresponding to the user to be identified according to the identity information; A pre-stored matrix matching the ultrasonic detection distance matrix is ​​determined in the pre-stored matrix library.

3. The lip reading recognition method according to claim 2, characterized in that: The step of acquiring the pre-stored matrix library corresponding to the user to be identified according to the identity information comprises: If the pre-stored matrix library corresponding to the user to be identified is not obtained, instructing the user to be identified to perform a preset action; When the user to be identified performs the preset action, confirming the pre-stored matrix corresponding to the preset action through the ultrasonic detection array; The lip reading data corresponding to the preset action is associated with the pre-stored matrix and saved as a pre-stored matrix library, and the pre-stored matrix library is associated with the identity information of the user to be identified and saved.

4. The lip reading recognition method according to claim 2, wherein: The step of determining the identity information of the user to be identified comprises: Obtaining biometric features of the user to be identified; The identity information of the user to be identified is determined according to the biometric feature.

5. The lip reading recognition method according to claim 1, wherein: Before the step of acquiring the detection data of the ultrasonic detection array, the method further includes: When receiving the lip reading recognition instruction, starting the lip reading recognition mode and executing the step of acquiring the detection data of the ultrasonic detection array; When receiving the voice collection instruction, switch the lip reading recognition mode to the voice mode and call the microphone to obtain the voice.

6. A lip reading recognition device, characterized in that: The lip reading recognition device comprises: a memory, a processor, and a lip reading recognition program stored in the memory and executable on the processor. When the lip reading recognition program is executed by the processor, the steps of the lip reading recognition method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a lip reading recognition program, and when the lip reading recognition program is executed by the processor, the steps of the lip reading recognition method according to any one of claims 1 to 5 are implemented.

Images