Lip language and sign language recognition device and method in real environment

Abstract

The invention discloses a lip language and sign language recognition device and method in a real environment, and belongs to the technical field of voice communication. As a system platform scheme, various target tasks can be switched at any time: lip language recognition is executed when an illumination target object is a lip, and sign language recognition, air handwriting input and other tasks are executed when the illumination target object is a double-hand object; distance measurement is not affected by the size, color and reflectivity of the target, and identification can be accurately completed even in a dark environment; under different language scenes, the machine can automatically match built-in AI language models, such as Chinese, dialect, English and the like; the embedded language inter-translation module can convert and output corresponding languages, such as English mouth shapes to Chinese speech; through an external interface or a wireless communication OTA (over the air technology) interface, software upgrading of different tasks can be realized, and requirements of different applications are met.

Description

Technical Field

[0001] The present invention relates to the technical field of language communication, and in particular to a lip language and sign language recognition device and method in a real environment. Background Art

[0002] With the development of artificial intelligence (AI) technology, intelligent recognition of lip reading and sign language has emerged. There are two main types of recognition objects: video images, and lip reading and sign language gestures in real environments. The current technical points are as follows:

[0003] 1. Use software to analyze the mouth movements (or hand movements) of the video image files in the corpus, extract the geometric features of the lips (or fingers) (shape, degree of opening and closing, movement trajectory, etc.), and the dynamic changes of the lips (or fingers) during the communication process (speed, direction and shape changes), generate a lip language library (or gesture library), and complete the visual pre-training language model;

[0004] 2. Vocabulary extraction: In actual use, the software analyzes the mouth shape or gestures in the 2D video image, and the pre-trained model maps the corresponding words or phrases;

[0005] 3. Syntactic and semantic understanding: The extracted words or phrases are combined into complete sentences, and then the context is understood; ensuring the correctness of grammar and semantics to achieve the purpose of outputting speech or text;

[0006] 4. During use, the pre-trained language model can perform self-learning at any time.

[0007] However, these image recognition technologies have the following disadvantages:

[0008] 1. It takes time to process. When faced with video content that is spoken too fast, only part of the content can be recognized, which may lead to the risk of missing content.

[0009] 2. There is a limit on the length of the video. Currently, only videos within a few minutes can be processed;

[0010] 3. Depends on the texture of the target (lip shape, gesture) and ambient lighting. The color, surface and reflectivity of the target have a great impact on the recognition performance, and it does not have the ability to accurately recognize in a backlit environment, so it cannot be used in a dark environment;

[0011] 4. The lip reading and sign language gestures in the real environment need to be converted into video images and then processed according to steps 1-3 above.

[0012] Therefore, it is necessary to provide a lip reading and sign language recognition device and method in a real environment. Summary of the invention

[0013] In view of the problems existing in the above-mentioned prior art, the present invention provides a lip reading and sign language recognition device and method in a real environment. The present invention adopts a time sensor (or radar) to obtain the distance information of the target, reconstructs the complete geometric information of the lip shape and gesture in the real three-dimensional scene, and uses relative coordinates with depth information to realize accurate digitization of the scene, thereby realizing key functions such as high-precision recognition, positioning, and language understanding.

[0014] In order to achieve the above-mentioned purpose, the present invention is implemented through the following technical scheme: a lip reading and sign language recognition device in a real environment, which adopts a time sensor to recognize and extract the lip reading and sign language in the real environment. The time sensor includes an infrared transmitter, an infrared receiver, a signal processor, and a power supply. The power supply supplies power to the time sensor. The output end of the signal processor is connected to the infrared transmitter to drive the infrared transmitter with an output signal, and the projected light beam is irradiated to the surface of the target object for reflection. The receiving end of the signal processor is connected to the infrared receiver to input information to the signal processor for calculating the depth information by time difference.

[0015] Preferably, a transmitting diffraction optical lens is docked at the transmitting end of the infrared transmitter.

[0016] Preferably, the axial distance between the emitting diffractive optical lens and the window of the infrared emitter can be adjusted forward and backward to adjust the field of view angle.

[0017] Preferably, the infrared receiver adopts a single photon avalanche diode (SPAD) array for infrared receiver optical imaging reception.

[0018] Preferably, the infrared receiver and the receiving diffraction optical lens are connected and docked after being cascaded and fixed to form an infrared receiver array to detect each area one by one.

[0019] Preferably, the axial spacing between the receiving diffraction optical lens and the window of the infrared receiver can be adjusted front to back so as to adjust the width of the receiving field of view.

[0020] Preferably, the multi-region pulse irradiation surfaces projected by the infrared transmitter through the emitting diffraction optical lens are all equipped with motion indicators, the motion indicators are connected to the input end of the signal processor, the signal processor is connected to the servo mechanism, and the signal processor drives the ranging sensor to automatically track the target.

[0021] Preferably, the infrared transmitter and infrared receiver are both embedded in a vehicle mounted with a telescope, or on the ground or in the air.

[0022] Preferably, the time sensor is a radar.

[0023] A method for recognizing lip reading and sign language in a real environment, using a time sensor to recognize and extract lip reading and sign language in a real environment, the time sensor can be set as a radar, and the time sensor includes an infrared transmitter, an infrared receiver, a signal processor, and a power supply;

[0024] Step 1: The power supply powers the time sensor, and the output signal of the signal processor drives the infrared transmitter. The light beam projected by the infrared transmitter is reflected by the surface of the target object, and the photons enter the signal processor through the infrared receiver; the signal processor calculates the depth information by measuring the time difference between the emission and reception of the light pulse;

[0025] Step 2: Based on step 1, the infrared transmitter is docked with a transmitting diffraction optical lens to project a multi-region pulse irradiation surface. Each irradiation surface area of ​​the multi-region pulse irradiation surface is equipped with a motion indicator. When the target moves beyond the target field of view, the signal processor controls the servo mechanism to drive the ranging sensor to automatically track the target and put the target back into the diagonal field of view;

[0026] Step 3: Based on step 2, the axial spacing between the emitting diffractive optical lens and the infrared emitter window can be adjusted forward and backward, and the field of view angle can be changed to achieve two-dimensional planar diagonal fields of view of different sizes, so that the irradiation surface can cover the complete target field of view area under different distances;

[0027] Step 4: Based on step 1, the infrared receiver uses a single photon avalanche diode (SPAD) array, and the reflected light pulse is received by the optical imaging of the infrared receiver.

[0028] Step 5: Based on step 2, the infrared receiver and the receiving diffraction optical lens are connected and fixed in cascade, and then docked to form an infrared receiving array to detect each area one by one; and the axial spacing between the receiving diffraction optical lens and the window of the infrared receiver can be adjusted forward and backward to achieve width adjustment of the field of view.

[0029] Step 6: Based on step 1, the infrared transmitter and infrared receiver are embedded in the optical system of the telescope, ground or air vehicle to achieve missions at the kilometer level or even farther.

[0030] In summary, the present invention provides a lip reading and sign language recognition device and method in a real environment. Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. As a system platform solution, the present invention can switch between multiple target tasks at any time: when the illuminated target is the lips, lip reading recognition is performed; when the illuminated target is the hands, sign language recognition, air handwriting input and other tasks are performed; distance measurement is not affected by the target size, color and reflectivity, and recognition can be accurately completed even in dark environments; in different language scenarios, the machine can automatically match the built-in AI language model, such as Chinese, dialects, English, etc.; the embedded language translation module can convert and output the corresponding language, such as English lip shape to Chinese speech; through the external interface or wireless communication OTA (over-the-air download technology) interface, software upgrades for different tasks can be realized to meet the needs of different applications.

[0032] 2. The present invention is different from the prior art that estimates the action only by relying on the amount of light reflected back from the target object. Instead, it uses a time sensor or radar, rather than a camera, to accurately measure the time required for light to reach the target object and reflect back to the sensor, to extract more subtle action information. Lossless or low-damage extraction of information and language fluency, providing a lip reading and sign language recognition method and device in a realistic environment. By placing it in a handheld device, real-time, highly accurate, and precise language pickup of up to hundreds of meters can be achieved; by embedding it in the optical system of a telescope, ground or aerial vehicle (such as a drone), it can achieve tasks at the kilometer level or even farther. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic structural diagram of a time sensor device of the present invention;

[0034] Figure 2 is a schematic diagram of a time sensor system method of the present invention;

[0035] Figure 3 is a schematic diagram of a time sensor system of the present invention;

[0036] Figure 4 is a schematic diagram of a multi-zone ranging sensor of the present invention;

[0037] Figure 5 is a schematic diagram of adjusting the width of the field of view of the present invention; DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings.

[0039] like Figures 1 to 5 As shown:

[0040] The present invention discloses a lip reading and sign language recognition device and method in a real environment. A time sensor is used to recognize and extract the lip reading and sign language in the real environment. The time sensor comprises an infrared transmitter, an infrared receiver, a signal processor and a power supply. The power supply supplies power to the time sensor. The output end of the signal processor is connected to the infrared transmitter to drive the infrared transmitter with an output signal. The projected light beam is irradiated on the surface of a target object for reflection. The receiving end of the signal processor is connected to the infrared receiver to input information to the signal processor for calculating depth information by time difference.

[0041] In at least one embodiment, a transmitting diffraction optical lens is butt-connected to the transmitting end of the infrared transmitter.

[0042] In at least one embodiment, the axial spacing between the emitting diffractive optical lens and the window of the infrared emitter can be adjusted forward and backward to adjust the field of view angle.

[0043] In at least one embodiment, the infrared receiver uses a single photon avalanche diode (SPAD) array for infrared receiver optical imaging reception.

[0044] In at least one embodiment, the infrared receiver and the receiving diffraction optical lens are connected and fixed in cascade connection and then docked to form an infrared receiver array to detect each area one by one.

[0045] In at least one embodiment, the axial spacing between the receiving diffractive optical lens and the window of the infrared receiver can be adjusted front to back so as to adjust the width of the receiving field of view.

[0046] In at least one embodiment, the multi-region pulse irradiation surface projected by the infrared transmitter through the emitting diffraction optical lens is equipped with a motion indicator, the motion indicator is connected to the input end of the signal processor, the signal processor is connected to the servo mechanism, and the signal processor drives the ranging sensor to automatically track the target.

[0047] In at least one embodiment, the infrared transmitter and infrared receiver are embedded in a vehicle mounted with a telescope, or on the ground or in the air.

[0048] In at least one embodiment, the time sensor is implemented as a radar.

[0049] A method for recognizing lip reading and sign language in a real environment, using a time sensor to recognize and extract lip reading and sign language in a real environment, the time sensor can be set as a radar, and the time sensor includes an infrared transmitter, an infrared receiver, a signal processor, and a power supply;

[0050] Step 1: The power supply powers the time sensor, and the output signal of the signal processor drives the infrared transmitter. The light beam projected by the infrared transmitter is reflected by the surface of the target object, and the photons enter the signal processor through the infrared receiver; the signal processor calculates the depth information by measuring the time difference between the emission and reception of the light pulse;

[0051] A more specific method for calculating the time difference depth information for step 1 is as follows: distance measurement is based on the speed of light, the photons emitted by the transmitter are reflected by the target and detected by the receiver, and the actual distance of the target is calculated with high accuracy through the time difference between emission and reception.

[0052] The distance formula is: distance = speed of light * time difference / 2

[0053] Step 2: Based on step 1, the infrared transmitter is docked with a transmitting diffraction optical lens to project a multi-region pulse irradiation surface. Each irradiation surface area of ​​the multi-region pulse irradiation surface is equipped with a motion indicator. When the target moves beyond the target field of view, the signal processor controls the servo mechanism to drive the ranging sensor to automatically track the target and put the target back into the diagonal field of view;

[0054] Specifically: Multi-area, multi-target measurement in the field of view: Each area in the field of view can measure the distance to the infrared emitter, obtain the accurate real-time position in the Cartesian coordinate system (X, Y and Z), and create a depth map covering multiple areas.

[0055] The matrix multi-zone ranging sensor integrates an invisible infrared light source. The reflected light is suppressed by an infrared filter for ambient light suppression and the receiving lens is focused onto a single-photon detector array to measure the distance of multiple targets covered within the field of view (FoV).

[0056] like Figure 4 Shown: Schematic diagram of a multi-zone ranging sensor that can be designed with a diagonal stereo field of view up to 60°.

[0057] Step 3: Based on step 2, the axial spacing between the emitting diffractive optical lens and the infrared emitter window can be adjusted forward and backward, and the field of view angle can be changed to achieve two-dimensional planar diagonal fields of view of different sizes, so that the irradiation surface can cover the complete target field of view area under different distances;

[0058] Step 4: Based on step 1, the infrared receiver uses a single photon avalanche diode (SPAD) array, and the reflected light pulse is received by the optical imaging of the infrared receiver.

[0059] Step 5: Based on step 2, the infrared receiver and the receiving diffraction optical lens are connected and fixed in cascade, and then docked to form an infrared receiving array to detect each area one by one; and the axial spacing between the receiving diffraction optical lens and the window of the infrared receiver can be adjusted forward and backward to achieve width adjustment of the field of view.

[0060] Specifically: based on the movement range of a person's head and hands, it is often a rectangular area with a short vertical axis and a long horizontal axis, so the length and width adjustment function of the rectangular area is necessary.

[0061] like Figure 5 Shown is a schematic diagram of adjusting the width of the field of view of the present invention; the adjustment range varies according to the distance of the target, and the preferred adjustment range is: 60° to 90°.

[0062] Step 6: Based on step 1, the infrared transmitter and infrared receiver are embedded in the optical system of the telescope, ground or air vehicle to achieve missions at the kilometer level or even farther.

[0063] The embodiments of the present invention are only used to illustrate the present invention and do not constitute a limitation on the scope of the claims. Other substantially equivalent substitutes that can be thought of by those skilled in the art are all within the protection scope of the present invention.

Claims

1. A lip reading and sign language recognition device in a real environment, characterized in that: A time sensor is used to identify and extract lip reading and sign language in a real environment. The time sensor includes an infrared transmitter, an infrared receiver, a signal processor, and a power supply. The power supply supplies power to the time sensor. The output end of the signal processor is connected to the infrared transmitter to drive the infrared transmitter with an output signal, and the projected light beam is irradiated on the surface of the target object for reflection. The receiving end of the signal processor is connected to the infrared receiver to input information to the signal processor for calculating depth information by time difference.

2. The lip reading and sign language recognition device in a real environment according to claim 1, characterized in that: A transmitting diffraction optical lens is butt-connected to the transmitting end of the infrared transmitter.

3. The lip reading and sign language recognition device in a real environment according to claim 2, characterized in that: The axial distance between the emitting diffraction optical lens and the window of the infrared emitter can be adjusted forward and backward to adjust the field angle.

4. The lip reading and sign language recognition device in a real environment according to claim 1, characterized in that: The infrared receiver adopts a single photon avalanche diode (SPAD) array for infrared receiver optical imaging reception.

5. The lip reading and sign language recognition device in a real environment according to claim 1, characterized in that: The infrared receiver and the receiving diffraction optical lens are connected and fixed in cascade connection and then butted against each other to form an infrared receiver array for receiving and detecting each area one by one.

6. The lip reading and sign language recognition device in a real environment according to claim 1, characterized in that: The axial distance between the receiving diffraction optical lens and the window of the infrared receiver can be adjusted forward and backward to adjust the width of the receiving field of view.

7. The lip reading and sign language recognition device in a real environment according to claim 3, characterized in that: The multi-region pulse irradiation surfaces projected by the infrared transmitter through the diffraction optical lens are all equipped with motion indicators, which are connected to the input end of the signal processor, which is connected to the servo mechanism, and the signal processor drives the distance sensor to automatically track the target.

8. The lip reading and sign language recognition device in a real environment according to claim 1, characterized in that: The infrared transmitter and the infrared receiver are embedded in a vehicle installed with a telescope, on the ground or in the air.

9. The lip reading and sign language recognition device in a real environment according to claim 1, characterized in that: The time sensor used is radar. A lip reading and sign language recognition method in a real environment, characterized in that: According to the lip reading and sign language recognition device in a real environment as described in any one of claims 1 to 9, a time sensor is used to recognize and extract the lip reading and sign language in the real environment, and the time sensor can be set as a radar, and the time sensor includes an infrared transmitter, an infrared receiver, a signal processor, and a power supply; Step 1: The power supply powers the time sensor, and the output signal of the signal processor drives the infrared transmitter. The light beam projected by the infrared transmitter is reflected by the surface of the target object, and the photons enter the signal processor through the infrared receiver; the signal processor calculates the depth information by measuring the time difference between the emission and reception of the light pulse; Step 2: Based on step 1, the infrared transmitter is docked with a transmitting diffraction optical lens to project a multi-region pulse irradiation surface. Each irradiation surface area of ​​the multi-region pulse irradiation surface is equipped with a motion indicator. When the target moves beyond the target field of view, the signal processor controls the servo mechanism to drive the ranging sensor to automatically track the target and put the target back into the diagonal field of view; Step 3: Based on step 2, the axial spacing between the emitting diffractive optical lens and the infrared emitter window can be adjusted forward and backward, and the field of view angle can be changed to achieve two-dimensional planar diagonal fields of view of different sizes, so that the irradiation surface can cover the complete target field of view area under different distances; Step 4: Based on step 1, the infrared receiver uses a single photon avalanche diode (SPAD) array, and the reflected light pulse is received by the optical imaging of the infrared receiver. Step 5: Based on step 2, the infrared receiver and the receiving diffraction optical lens are connected and fixed in cascade, and then docked to form an infrared receiving array to detect each area one by one; and the axial spacing between the receiving diffraction optical lens and the window of the infrared receiver can be adjusted forward and backward to achieve width adjustment of the field of view. Step 6: Based on step 1, the infrared transmitter and infrared receiver are embedded in the optical system of the telescope, ground or air vehicle to achieve missions at the kilometer level or even farther.

Images