Gesture recognition method, interaction method based on gesture recognition and mixed reality glasses

Abstract

The present invention discloses a gesture recognition method, which comprises the following steps: obtaining a depth map sequence of a gesture to be recognized; obtaining a key frame sequence of the depth map sequence based on the depth map sequence; inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized; and obtaining a semantic result of the gesture to be recognized based on the first semantic sequence. The present invention also discloses an interaction method based on gesture recognition and mixed reality glasses. The method provided by the present invention is intended to solve the technical problem that the existing gesture recognition algorithm based on deep learning has low accuracy due to the error limit of the preset algorithm of the depth map.

Description

Technical Field

[0001] The present invention relates to the field of geographic information technology, and in particular to a gesture recognition method, an interaction method based on gesture recognition, and mixed reality glasses. Background Art

[0002] Currently, there are gesture recognition algorithms that help the deaf and mute communicate normally with ordinary people. However, most of the 3D gesture recognition algorithms based on deep learning first predict the depth of the RGB image acquired by the ordinary lens to obtain the depth information, and then perform gesture recognition on the predicted depth information. The accuracy of gesture recognition of this method is limited by the error of the depth prediction algorithm, and the accuracy is relatively low.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a gesture recognition method, an interaction method based on gesture recognition and mixed reality glasses, aiming to solve the technical problem that the existing gesture recognition algorithm based on deep learning is limited by the error of the preset algorithm of the depth map and has low accuracy.

[0005] To achieve the above object, the present invention provides a gesture recognition method, which comprises the following steps:

[0006] Obtain a depth map sequence of the gesture to be recognized;

[0007] Based on the depth map sequence, obtaining a key frame sequence of the depth map sequence;

[0008] Inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized;

[0009] Based on the first semantic sequence, a semantic result of the gesture to be recognized is obtained.

[0010] Optionally, the step of inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized includes:

[0011] Extracting image spatial features of each frame image in the key frame sequence through a convolutional layer 3D CNN;

[0012] Extracting the temporal relationship features of the key frame sequence through a time recursive layer LSTM RNN;

[0013] Combining the image spatial features with the temporal relationship features to obtain the spatiotemporal features of the key frame sequence;

[0014] The spatiotemporal features are input into the classifier output layer to obtain the semantic sequence of the gesture to be recognized.

[0015] Optionally, the step of obtaining the semantic result of the gesture to be recognized based on the semantic sequence includes:

[0016] The semantic sequence is input into a pre-trained semantic translation model to obtain the semantic result.

[0017] Optionally, the step of obtaining a depth map sequence of a gesture to be recognized includes:

[0018] Get the depth map sequence of the gesture to be recognized collected by the depth camera.

[0019] In addition, in order to solve the above problems, the present invention also provides an interaction method based on gesture recognition, which is applied to a mixed reality device and includes the following steps:

[0020] Obtain a depth map of the gesture to be recognized;

[0021] Based on the depth map, obtaining a key frame sequence of the depth map;

[0022] Inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized;

[0023] Based on the first semantic sequence, obtaining a semantic result of the gesture to be recognized;

[0024] Outputting the semantic result;

[0025] Obtaining voice response information for the semantic result;

[0026] Extracting a voice segment based on the voice response information;

[0027] Inputting the speech segment into a pre-trained speech translation model to obtain a second semantic sequence of the speech response information;

[0028] Based on the second semantic sequence, acquiring a gesture graphic sequence;

[0029] The gesture graphic sequence is displayed.

[0030] Optionally, the step of inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized includes:

[0031] Extracting image spatial features of each frame image in the key frame sequence through a convolutional layer 3D CNN;

[0032] Extracting the temporal relationship features of the key frame sequence through a time recursive layer LSTM RNN;

[0033] Combining the image spatial features with the temporal relationship features to obtain the spatiotemporal features of the key frame sequence;

[0034] The spatiotemporal features are input into the classifier output layer to obtain the semantic sequence of the gesture to be recognized.

[0035] Optionally, the step of obtaining the semantic result of the gesture to be recognized based on the semantic sequence includes:

[0036] The semantic sequence is input into a pre-trained semantic translation model to obtain the semantic result.

[0037] Optionally, the step of obtaining a depth map sequence of a gesture to be recognized includes:

[0038] Get the depth map sequence of the gesture to be recognized collected by the depth camera.

[0039] In addition, in order to solve the above problems, the present invention also provides a gesture recognition device, the device comprising:

[0040] A first acquisition module, used to acquire a depth map sequence of a gesture to be recognized;

[0041] A second acquisition module, configured to acquire a key frame sequence of the depth map sequence based on the depth map sequence;

[0042] A recognition module, used for inputting the key frame sequence into a pre-trained gesture recognition model to obtain a semantic sequence of the gesture to be recognized;

[0043] An acquisition module is used to acquire a semantic result of the gesture to be recognized based on the semantic sequence.

[0044] In addition, in order to solve the above problems, the present invention also provides an interactive device based on gesture recognition, which is applied to a mixed reality device. The interactive device based on gesture recognition includes:

[0045] A first acquisition module, used to acquire a depth map of the gesture to be recognized;

[0046] A second acquisition module, configured to acquire a key frame sequence of the depth map based on the depth map;

[0047] A recognition module, used for inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized;

[0048] An acquisition module, which acquires a semantic result of the gesture to be recognized based on the first semantic sequence;

[0049] An interaction module, used for outputting the semantic result and obtaining voice response information for the semantic result;

[0050] An extraction module, used for extracting a voice segment based on the voice response information;

[0051] A translation module, used for inputting the speech segment into a pre-trained speech translation model to obtain a second semantic sequence of the speech response information;

[0052] A fourth acquisition module, used for acquiring a gesture graphic sequence based on the second semantic sequence;

[0053] The display module is used to display the gesture graphic sequence.

[0054] In addition, in order to solve the above problem, the present invention also provides an electronic device, the device comprising: a memory, a processor, and a gesture recognition program stored in the memory and executable on the processor, the gesture recognition program being configured to implement the steps of the gesture recognition method as described above; or

[0055] The electronic device comprises: a memory, a processor, and an interactive program based on gesture recognition stored in the memory and executable on the processor, wherein the interactive program based on gesture recognition is configured to implement the steps of the interactive method based on gesture recognition as described above.

[0056] In addition, in order to solve the above-mentioned problem, the present invention also provides a mixed reality glasses, and the mixed reality glasses include the above-mentioned electronic device.

[0057] In addition, in order to solve the above problem, the present invention further provides a storage medium, on which a gesture recognition program is stored, and when the gesture recognition program is executed by a processor, the steps of the gesture recognition method as described above are implemented; or

[0058] The storage medium stores an interaction program based on gesture recognition, and when the interaction program based on gesture recognition is executed by the processor, the steps of the interaction method based on gesture recognition as described above are implemented.

[0059] The embodiments of the present invention propose a gesture recognition method, an interaction method based on gesture recognition, and mixed reality glasses. The gesture recognition method directly obtains the depth map of the gesture made by the user, avoiding the error and corresponding time cost caused by the traditional deep learning method estimating the depth information from the ordinary RGB map, thereby improving the accuracy and efficiency of gesture recognition. It is also beneficial to the accuracy and immediacy of the expression of the interactive meaning using the gesture recognition interaction method in actual application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a mixed reality glasses according to an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a flow chart of an embodiment of a gesture recognition method of the present invention;

[0062] Figure 3 for Figure 2 A detailed flow chart of step S300;

[0063] Figure 4 It is a flowchart of an embodiment of an interaction method based on gesture recognition of the present invention;

[0064] Figure 5 is a structural block diagram of an embodiment of a gesture recognition device of the present invention;

[0065] Figure 6 It is a structural block diagram of an embodiment of an interactive device based on gesture recognition of the present invention.

[0066] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0068] Most deep learning-based 3D gesture recognition algorithms first perform depth prediction on ordinary RGB images to obtain depth information, and then perform gesture recognition based on the predicted depth information. For example, gestures are captured through the camera mounted on a mobile phone or tablet terminal to obtain an ordinary RGB image, and then the depth prediction algorithm mounted on the mobile phone or tablet terminal performs depth prediction on the RGB image to obtain depth information, and then gesture recognition is performed based on the predicted depth information. However, the recognition accuracy of this type of gesture recognition algorithm is limited by the error of the depth prediction algorithm, that is, the accuracy of gesture recognition is limited.

[0069] In order to solve the above problems, the embodiment of the present invention provides a gesture recognition method to directly obtain the depth map of the gesture made by the user, avoiding the error and corresponding time cost caused by the traditional deep learning method for estimating the RGB image, thereby improving the accuracy and efficiency of gesture recognition. It is also beneficial to the accuracy and immediacy of the expression of the meaning of the interaction using the gesture recognition interaction method in actual application.

[0070] Reference Figure 1 , Figure 1 The diagram is a recommended device structure diagram of a gesture recognition method in a hardware operating environment and an interaction method based on gesture recognition according to an embodiment of the present invention.

[0071] The device may be a user equipment (UE) such as mixed reality glasses, a mobile phone, a smart phone, a laptop, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem, a mobile station (MS), etc. The device may be called a user terminal, a portable terminal, a desktop terminal, etc.

[0072] Typically, the device includes: at least one processor 301, a memory 302, and a gesture recognition program stored in the memory and executable on the processor, wherein the gesture recognition program is configured to implement the steps of the gesture recognition method or the interaction method based on gesture recognition as described above.

[0073] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. The processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process gesture recognition operations so that the gesture recognition model can be trained and learned autonomously to improve efficiency and accuracy.

[0074] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 801 to implement the gesture recognition method provided in the method embodiment of the present application.

[0075] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0076] The communication interface 303 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. The communication interface 303 is used to receive the movement trajectories and other data of multiple mobile terminals uploaded by users through the peripheral device. In some embodiments, the processor 301, the memory 302 and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302 and the communication interface 303 can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0077] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with the communication network and other communication devices through electromagnetic signals, so as to obtain the movement trajectory and other data of multiple mobile terminals. The radio frequency circuit 304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 304 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0078] The display screen 305 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 305 is a touch display screen, the display screen 305 also has the ability to collect touch signals on the surface or above the surface of the display screen 305. The touch signal can be input to the processor 301 as a control signal for processing. At this time, the display screen 305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 305 can be one, the front panel of the electronic device; in other embodiments, the display screen 305 can be at least two, respectively arranged on different surfaces of the electronic device or in a folding design; in some further embodiments, the display screen 305 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device. Even, the display screen 305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 305 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0079] The power supply 306 is used to power various components in the electronic device. The power supply 306 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 306 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0080] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0081] The embodiment of the present invention provides a gesture recognition method, referring to Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a gesture recognition method according to the present invention.

[0082] In this embodiment, a gesture recognition method is provided, the method comprising the following steps:

[0083] Step S100, obtaining a depth map of the gesture to be recognized.

[0084] Specifically, in the above steps, the gesture to be recognized can be a static gesture or a dynamic gesture. Compared with a static gesture, a dynamic gesture can be composed of many hand movements, such as the sign language used by the deaf-mute, that is, the gesture to be recognized can be a single gesture, or a dynamic gesture sequence composed of multiple gestures that are coherent and connected.

[0085] The above steps are used to directly obtain a depth map. Compared with an RGB map obtained by a common lens, a depth map includes not only a planar image of the gesture but also position and size information of the gesture.

[0086] For example, the depth map can be obtained through a depth camera. In this case, step S100 obtains the depth map of the gesture to be recognized, and is adaptively modified as follows:

[0087] Step S100 ′, obtaining a depth map sequence of the gesture to be recognized collected by the depth camera.

[0088] Specifically, in the above steps, the depth map sequence is composed of several frames of depth maps, and the coordinates of each pixel in the image signal of the depth map sequence are three-dimensional coordinates, which can be obtained based on a spatial coordinate system established with the depth camera as the origin. The three-dimensional coordinates of the pixel points of the depth map can be used to accurately capture the user's gestures.

[0089] For example, in one embodiment, the mixed reality glasses are equipped with a depth camera using a CMOS light-sensing element to obtain the depth information of the current scene, and the depth camera using the CMOS light-sensing element can be used to collect the depth map of the gesture to be recognized by the user or the interactive object. For example, the three-dimensional coordinate system can be based on the depth camera as the origin, the horizontal direction at the position of the depth camera as the x-axis direction, the vertical direction at the position of the depth camera as the y-axis direction, and the direction perpendicular to the aforementioned x-axis and y-axis at the position of the depth camera as the z-axis direction. Therefore, in the image signal obtained by the depth camera, the z value in the coordinates of each pixel point can be used to indicate the distance of the position point corresponding to the pixel point from the depth camera, that is, the z value can be used to indicate the depth of field.

[0090] Step S200: acquiring a key frame sequence of the depth map sequence based on the depth map sequence.

[0091] Specifically, the key frame sequence is used to represent the meaning of the gesture to be recognized. Since the expression of the meaning of the gesture requires a coherent expression between the gestures, the extracted key frames need to be arranged in time sequence, that is, the key frame sequence of the depth map sequence is obtained. For example, the object captured by the depth camera is a sign language gesture made by a deaf-mute person. At this time, the sign language gesture includes multiple gestures and corresponding connecting gestures. That is, the corresponding depth map sequence includes a key frame sequence for expression of meaning, and a non-key frame sequence for non-expression of meaning. Therefore, the above steps are used to obtain a key frame sequence for expression of meaning.

[0092] Step S300: inputting a key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of a gesture to be recognized.

[0093] Specifically, the above steps are used to perform recognition operations on gesture actions to obtain the meaning expressed by the gesture to be recognized. In the above steps, two-dimensional gesture recognition based on feature point extraction can be used, and a gesture recognition method based on deep learning can also be used, which is not limited in this application.

[0094] Optional, since most traditional gesture recognition networks use 2D CNN models for training and recognition. However, when the gesture to be recognized is the sign language used by the deaf and mute, due to the connectivity between different gestures, the coherence of contextual semantics is particularly emphasized. At this time, the traditional gesture recognition network that mainly recognizes static gestures for machine interaction is difficult to meet the needs of sign language recognition.

[0095] To this end, in one embodiment, see Figure 3 Step S300, inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized, including:

[0096] Step S301, extracting the image spatial features of each frame image in the key frame sequence through the convolutional layer 3D CNN.

[0097] Step S302: extracting the temporal relationship features of the key frame sequence through the time recursive layer LSTM RNN.

[0098] Step S303: combining the image spatial features and the temporal relationship features to obtain the spatiotemporal features of the key frame sequence.

[0099] Step S304: input the spatiotemporal features into the output layer of the classifier to obtain a first semantic sequence of the gesture to be recognized.

[0100] Specifically, the gesture recognition network is a combined neural network, which includes a convolutional layer 3D CNN and a time recurrent neural layer LSTM RNN. The convolutional layer 3D CNN has the property of being transformation invariant, and can be used to extract spatial features with scale invariant characteristics, such as the palm shape and direction features of each gesture image in a dynamic gesture sequence. The number of layers of the convolutional layer 3D CNN is preset, for example, the convolutional layer 3D CNN can specifically include 3 convolutional layers and 3 downsampling layers, and the convolution kernels of all convolutional layers can be consistent, for example, the convolution kernel size is 5*5.

[0101] The function of the one-layer time recursive neural network LSTM RNN is to extract the temporal relationship features of the dynamic gesture sequence. The temporal relationship features are used to express the contextual connection of the gesture sequence. The function of the time recursive neural network LSTM RNN is to analyze the feature context of adjacent frames, so as to integrate and transfer them. That is, the spatial features obtained by the previous convolutional layer 3D CNN can be combined to obtain the spatiotemporal features required for the final classification. Specifically, the 1-layer LSTM RNN network is connected in series to the last layer of the convolutional layer 3D CNN network.

[0102] The classifier output layer may include several softmax units. The function of the classifier output layer is to combine the temporal relationship features obtained by context feature analysis with the spatial features to obtain spatiotemporal features, and input the spatiotemporal features into a classifier with softmax units for classification output to obtain the first semantic sequence of the gesture to be recognized.

[0103] Step S400: obtaining a semantic result of the gesture to be recognized based on the first semantic sequence.

[0104] Specifically, in the above steps, the semantic result can be data such as Mandarin text that can be directly output through a display or a player or other interactive devices. The above steps are used to convert the obtained first semantic sequence into data that can be processed by other interactive devices, so as to facilitate the output of the meaning expression of the gesture to be recognized.

[0105] For example, in one embodiment, step S400, based on the semantic sequence, obtains the semantic result of the gesture to be recognized, including:

[0106] Step S400 ′, inputting the semantic sequence into a pre-trained semantic translation model to obtain a semantic result.

[0107] In the above steps, the first semantic sequence can be input into the semantic translation model, and the corresponding Mandarin text can be obtained through the semantic translation model, and the Mandarin text is the semantic result of the gesture to be recognized. It is easy to understand that the semantic translation model is an existing technology that those skilled in the art know how to implement, and will not be repeated here.

[0108] Compared with the existing gesture recognition method, which first predicts the depth of an ordinary RGB image to obtain depth information, and then performs gesture recognition based on the predicted depth information, the recognition accuracy is limited by the error of the depth prediction algorithm. The gesture recognition algorithm provided in this embodiment can directly obtain the depth map sequence of the gesture made by the user through a device such as a depth camera, avoiding the error and corresponding time cost caused by the traditional deep learning method for estimating the depth map, thereby improving the accuracy and efficiency of gesture recognition. It is also conducive to the accuracy and immediacy of the interactive expression of the meaning of the interaction using the gesture recognition method in actual application. Therefore, it is conducive to use for special groups such as the deaf and dumb, so as to help such special groups communicate with ordinary people.

[0109] In addition, in order to solve the above problems, the present invention also provides an embodiment of an interaction method based on gesture recognition, which is applied to a mixed reality device. The following is a specific description of the interaction method based on gesture recognition used in mixed reality glasses as an example. It is easy to understand that the mixed reality glasses are only used as an example and do not constitute a limitation on the embodiments of the present application.

[0110] See also Figure 4 , Figure 4 The figure is a flow chart of an embodiment of an interaction method based on gesture recognition according to the present invention.

[0111] In this embodiment, the interaction method based on gesture recognition includes the following steps:

[0112] Step S100, obtaining a depth map of the gesture to be recognized.

[0113] Specifically, in the above steps, the gesture to be recognized can be a static gesture or a dynamic gesture. Compared with a static gesture, a dynamic gesture can be composed of many hand movements, such as the sign language used by the deaf-mute, that is, the gesture to be recognized can be a single gesture, or a dynamic gesture sequence composed of multiple gestures that are coherent and connected.

[0114] The above steps are used to directly obtain a depth map. Compared with an RGB map obtained by a common lens, a depth map includes not only a planar image of the gesture but also position and size information of the gesture.

[0115] For example, the depth map can be obtained through a depth camera. In this case, step S100 obtains the depth map of the gesture to be recognized, and is adaptively modified as follows:

[0116] Step S100 ′, obtaining a depth map sequence of the gesture to be recognized collected by the depth camera.

[0117] Specifically, in the above steps, the depth map sequence is composed of several frames of depth maps, and the coordinates of each pixel in the image signal of the depth map sequence are three-dimensional coordinates, which can be obtained based on a spatial coordinate system established with the depth camera as the origin. The three-dimensional coordinates of the pixel points of the depth map can be used to accurately capture the user's gestures.

[0118] For example, in one embodiment, the mixed reality glasses are equipped with a depth camera using a CMOS light-sensing element to obtain the depth information of the current scene, and the depth camera using the CMOS light-sensing element can be used to collect the depth map of the gesture to be recognized by the user or the interactive object. For example, the three-dimensional coordinate system can be based on the depth camera as the origin, the horizontal direction at the position of the depth camera as the x-axis direction, the vertical direction at the position of the depth camera as the y-axis direction, and the direction perpendicular to the aforementioned x-axis and y-axis at the position of the depth camera as the z-axis direction. Therefore, in the image signal obtained by the depth camera, the z value in the coordinates of each pixel point can be used to indicate the distance of the position point corresponding to the pixel point from the depth camera, that is, the z value can be used to indicate the depth of field.

[0119] Step S200: acquiring a key frame sequence of the depth map sequence based on the depth map sequence.

[0120] Specifically, the key frame sequence is used to represent the meaning of the gesture to be recognized. Since the expression of the meaning of the gesture requires a coherent expression between the gestures, the extracted key frames need to be arranged in time sequence, that is, the key frame sequence of the depth map sequence is obtained. For example, the object captured by the depth camera is a sign language gesture made by a deaf-mute person. At this time, the sign language gesture includes multiple gestures and corresponding connecting gestures. That is, the corresponding depth map sequence includes a key frame sequence for expression of meaning, and a non-key frame sequence for non-expression of meaning. Therefore, the above steps are used to obtain a key frame sequence for expression of meaning.

[0121] Step S300: inputting a key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of a gesture to be recognized.

[0122] Specifically, the above steps are used to perform recognition operations on gesture actions to obtain the meaning expressed by the gesture to be recognized. In the above steps, two-dimensional gesture recognition based on feature point extraction can be used, and a gesture recognition method based on deep learning can also be used, which is not limited in this application.

[0123] Optional, since most traditional gesture recognition networks use 2D CNN models for training and recognition. However, when the gesture to be recognized is the sign language used by the deaf and mute, due to the connectivity between different gestures, the coherence of contextual semantics is particularly emphasized. At this time, the traditional gesture recognition network that mainly recognizes static gestures for machine interaction is difficult to meet the needs of sign language recognition.

[0124] To this end, in one embodiment, see Figure 3 Step S300, inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized, including:

[0125] Step S301, extracting the image spatial features of each frame image in the key frame sequence through the convolutional layer 3D CNN.

[0126] Step S302: extracting the temporal relationship features of the key frame sequence through the time recursive layer LSTM RNN.

[0127] Step S303: combining the image spatial features and the temporal relationship features to obtain the spatiotemporal features of the key frame sequence.

[0128] Step S304: input the spatiotemporal features into the output layer of the classifier to obtain a first semantic sequence of the gesture to be recognized.

[0129] Specifically, the gesture recognition network is a combined neural network, which includes a convolutional layer 3D CNN and a time recurrent neural layer LSTM RNN. The convolutional layer 3D CNN has the property of being transformation invariant, and can be used to extract spatial features with scale invariant characteristics, such as the palm shape and direction features of each gesture image in a dynamic gesture sequence. The number of layers of the convolutional layer 3D CNN is preset, for example, the convolutional layer 3D CNN can specifically include 3 convolutional layers and 3 downsampling layers, and the convolution kernels of all convolutional layers can be consistent, for example, the convolution kernel size is 5×5.

[0130] The function of the one-layer time recursive neural network LSTM RNN is to extract the temporal relationship features of the dynamic gesture sequence. The temporal relationship features are used to express the contextual connection of the gesture sequence. The function of the time recursive neural network LSTM RNN is to analyze the feature context of adjacent frames, so as to integrate and transfer them. That is, the spatial features obtained by the previous convolutional layer 3D CNN can be combined to obtain the spatiotemporal features required for the final classification. Specifically, the 1-layer LSTM RNN network is connected in series to the last layer of the convolutional layer 3D CNN network.

[0131] The classifier output layer may include several softmax units. The function of the classifier output layer is to combine the temporal relationship features obtained by context feature analysis with the spatial features to obtain spatiotemporal features, and input the spatiotemporal features into a classifier with softmax units for classification output to obtain the first semantic sequence of the gesture to be recognized.

[0132] Step S400: obtaining a semantic result of the gesture to be recognized based on the first semantic sequence.

[0133] Specifically, in the above steps, the semantic result can be data such as Mandarin text that can be directly output through a display or a player or other interactive devices. The above steps are used to convert the obtained first semantic sequence into data that can be processed by other interactive devices, so as to facilitate the output of the meaning expression of the gesture to be recognized.

[0134] For example, in one embodiment, step S400, based on the semantic sequence, obtains the semantic result of the gesture to be recognized, including:

[0135] Step S400 ′, inputting the semantic sequence into a pre-trained semantic translation model to obtain a semantic result.

[0136] In the above steps, the first semantic sequence can be input into the semantic translation model, and the corresponding Mandarin text can be obtained through the semantic translation model. The Mandarin text is the semantic result of the gesture to be recognized. It is easy to understand that the semantic translation model is a prior art that those skilled in the art know how to implement, and it will not be repeated here. Specifically, the specific implementation process of the above steps can refer to the above embodiments. Since the above steps adopt all the technical solutions of all the above embodiments, they at least have all the beneficial effects brought by the technical solutions of the above embodiments, which will not be repeated here one by one.

[0137] Step S500: output the semantic result and obtain voice response information for the semantic result.

[0138] Specifically, the above steps are used to output the semantic result of the gesture to be recognized through a display or a player or other interactive device. For example, it is displayed on a display screen, or played through a speaker or other similar interactive device. It is easy to understand that the output can be received by the expression object of the gesture to be recognized, and a corresponding response is made after receiving the expression of the semantic result.

[0139] In one embodiment, the user of the mixed reality glasses is a deaf-mute person, and the deaf-mute person communicates face-to-face with an ordinary person. The mixed reality glasses recognize the sign language made by the deaf-mute user and recognize the sign language as a semantic result, and then output the semantic result through the equipped speaker, for example, the Mandarin expression of the sign language meaning. After receiving the Mandarin expression played by the speaker, the normal person makes a corresponding response, which can be a voice response information.

[0140] For example, a deaf-mute wearing mixed reality glasses goes shopping in a mall and makes the corresponding sign language "How much is the discount on this dress?" The depth camera on the mixed reality glasses directly obtains the depth map sequence of the sign language and recognizes it as the corresponding semantic result: the Mandarin text "How much is the discount on this dress?" Then the speaker on the mixed reality glasses plays in Mandarin: "How much is the discount on this dress?" After hearing the voice played by the speaker, the shopping guide in the mall makes a corresponding voice response "Hello, this dress is 30% off." At this time, the earpiece on the mixed reality glasses obtains and obtains the voice response information "Hello, this dress is 30% off" for the semantic result "How much is the discount on this dress?"

[0141] Step S600: extracting voice segments based on voice response information.

[0142] Specifically, the voice segment in the above step is a key voice segment that records the response information. Specifically, the acquired voice response information can be subjected to noise reduction processing, and then the corresponding voice segment is extracted.

[0143] Step S700: input the speech segment into a pre-trained speech translation model to obtain a second semantic sequence of speech response information.

[0144] Specifically, the above steps are used to convert the speech segment into a corresponding second semantic sequence. The second semantic sequence records the meaning of the speech response information.

[0145] Step S800: acquiring a gesture graphic sequence based on a second semantic sequence.

[0146] Specifically, in the above steps, the second semantic sequence can be converted into a corresponding sign language representation through semantic analysis, and the corresponding sign language model animation can be loaded from the corresponding sign language database, and the corresponding sign language model animation can be converted into a gesture graphic sequence through the timing of the second semantic sequence.

[0147] Step S900: displaying a gesture graphic sequence.

[0148] Specifically, this step is used to realize the interaction between the mixed reality glasses and the deaf-mute user. The interaction can be realized by displaying the gesture graphic sequence on the mixed reality glasses, that is, deploying and playing in the real scene. The deaf-mute wearing the mixed reality glasses can directly receive the gesture graphic sequence through vision, thereby obtaining the meaning of the corresponding voice response information.

[0149] Compared with the existing single-end gesture recognition or voice recognition method, the gesture-based interaction method provided by this embodiment integrates gesture recognition or voice recognition into one, forming an end-to-end closure, thereby realizing real-time communication and communication between special groups such as the deaf and dumb and the general public. In addition, the method is used in mixed reality devices, so that the interaction can be demonstrated in a real and vivid model animation through mixed reality glasses, thereby improving the quality of communication.

[0150] In addition, the present invention also provides an embodiment of a gesture recognition device, see Figure 5 , Figure 5 This is a structural block diagram of this embodiment, the device includes:

[0151] A first acquisition module 10, used to acquire a depth map of the gesture to be recognized;

[0152] A second acquisition module 20 is used to acquire a key frame sequence of the depth map based on the depth map;

[0153] A recognition module 30, configured to input a key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of a gesture to be recognized;

[0154] The obtaining module 40 obtains the semantic result of the gesture to be recognized based on the first semantic sequence.

[0155] Compared with the existing gesture recognition device, which first performs depth prediction on ordinary RGB images to obtain depth information, and then performs gesture recognition based on the predicted depth information, the recognition accuracy is limited by the error of the depth prediction algorithm. The gesture recognition device provided in this embodiment can directly obtain the depth map sequence of the gesture made by the user through a depth camera and other devices, avoiding the error and corresponding time cost caused by the traditional deep learning method for estimating the depth map, thereby improving the accuracy and efficiency of gesture recognition. It is also conducive to the accuracy and immediacy of the interactive expression of the meaning of the interaction using the gesture recognition interaction method in actual application. Therefore, it is conducive to use for special groups such as the deaf and dumb, so as to help such special groups communicate with ordinary people.

[0156] In addition, the present invention also provides an embodiment of an interactive device based on gesture recognition, which is applied to a mixed reality device. Figure 6 , Figure 6 This is a structural block diagram of this embodiment.

[0157] In this embodiment, the interactive device based on gesture recognition includes:

[0158] A first acquisition module 10, used to acquire a depth map of the gesture to be recognized;

[0159] A second acquisition module 20 is used to acquire a key frame sequence of the depth map based on the depth map;

[0160] A recognition module 30, configured to input a key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of a gesture to be recognized;

[0161] An acquisition module 40, which acquires a semantic result of the gesture to be recognized based on the first semantic sequence;

[0162] Interaction module 50, used to output semantic results and obtain voice response information for the semantic results;

[0163] An extraction module 60, for extracting a voice segment based on the voice response information;

[0164] A translation module 70, for inputting the speech segment into a pre-trained speech translation model to obtain a second semantic sequence of speech response information;

[0165] A fourth acquisition module 80, configured to acquire a gesture graphic sequence based on the second semantic sequence;

[0166] The display module 90 is used to display the gesture graphic sequence.

[0167] Compared with the existing single-end gesture recognition or voice recognition method, the interaction method provided by the gesture-based interaction device provided in this embodiment integrates gesture recognition or voice recognition into one, forming an end-to-end closure, thereby realizing real-time communication and communication between special groups such as the deaf and dumb and the general public. In addition, the method is used in mixed reality devices, so that the interaction can be demonstrated in a real and vivid model animation through mixed reality glasses, thereby improving the quality of communication.

[0168] Other embodiments or specific implementations of the gesture recognition device and the interaction device based on gesture recognition of the present invention may refer to the above-mentioned method embodiments, which will not be described in detail here.

[0169] In addition, in order to solve the above-mentioned problem, the present invention also provides a mixed reality glasses, which includes the above-mentioned electronic device.

[0170] Specifically, the mixed reality glasses are equipped with a depth camera, a speaker and a receiver. The depth camera is used to collect depth maps. It is easy to understand that the depth camera can not only collect gestures made by the user, but also collect gestures made by the object of the user's communication. That is, the mixed reality glasses can not only be used by the deaf and mute, but also by ordinary people to communicate with normal people. The speaker is used to output semantic results, which will help improve the immediacy of communication between the deaf and mute and ordinary people.

[0171] In addition, in order to solve the above problem, the present invention further provides a storage medium, on which a gesture recognition program is stored, and when the gesture recognition program is executed by a processor, the steps of the gesture recognition method as described above are implemented; or

[0172] The storage medium stores an interactive program based on gesture recognition, and when the interactive program based on gesture recognition is executed by the processor, the steps of the interactive method based on gesture recognition as described above are implemented. Therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0174] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without creative work.

[0175] Through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present invention, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

Claims

1. A gesture recognition method, characterized in that: The method comprises the following steps: Obtain a depth map sequence of the gesture to be recognized; Based on the depth map sequence, obtaining a key frame sequence of the depth map sequence; Inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized; Based on the first semantic sequence, obtaining a semantic result of the gesture to be recognized; The step of inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized comprises: The image spatial features of each frame image in the key frame sequence are extracted by a convolutional layer 3D CNN; the convolutional layer 3D CNN includes multiple convolutional layers and multiple downsampling layers, and the convolution kernels of all convolutional layers are consistent; The temporal relationship features of the key frame sequence are extracted through a time recursive layer LSTM RNN; the LSTM RNN is serially connected to the last layer of the convolutional layer 3D CNN network; Combining the image spatial features with the temporal relationship features to obtain the spatiotemporal features of the key frame sequence; The spatiotemporal features are input into the output layer of the classifier to obtain the semantic sequence of the gesture to be recognized.

2. The gesture recognition method according to claim 1, characterized in that: The step of obtaining the semantic result of the gesture to be recognized based on the semantic sequence includes: The semantic sequence is input into a pre-trained semantic translation model to obtain the semantic result.

3. The gesture recognition method according to claim 1, characterized in that: The step of obtaining a depth map sequence of the gesture to be recognized includes: Get the depth map sequence of the gesture to be recognized collected by the depth camera.

4. An interactive method based on gesture recognition, the method being applied to a mixed reality device, characterized in that: The method comprises the following steps: Obtain a depth map sequence of the gesture to be recognized; Based on the depth map sequence, obtaining a key frame sequence of the depth map sequence; Inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized; Based on the semantic sequence, obtaining a semantic result of the gesture to be recognized; Outputting the semantic result and obtaining voice response information for the semantic result; Extracting a voice segment based on the voice response information; Inputting the speech segment into a pre-trained speech translation model to obtain a second semantic sequence of the speech response information; Based on the second semantic sequence, acquiring a gesture graphic sequence; displaying the gesture graphic sequence; The step of inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized includes: The image spatial features of each frame in the key frame sequence are extracted through the convolutional layer 3D CNN; the convolutional layer 3D CNN includes multiple convolutional layers and multiple downsampling layers, and the convolution kernels of all convolutional layers are consistent; The temporal relationship features of the key frame sequence are extracted through the time recursive layer LSTM RNN; the LSTM RNN is connected in series to the last layer of the convolutional layer 3D CNN network Combining the image spatial features with the temporal relationship features, the spatiotemporal features of the key frame sequence are obtained; The spatiotemporal features are input into the classifier output layer to obtain the first semantic sequence of the gesture to be recognized.

5. A gesture recognition device, characterized in that: The device comprises: A first acquisition module, used to acquire a depth map of the gesture to be recognized; A second acquisition module, configured to acquire a key frame sequence of the depth map based on the depth map; A recognition module, used for inputting the key frame sequence into a pre-trained gesture recognition model to obtain a semantic sequence of the gesture to be recognized; An acquisition module, used for acquiring a semantic result of the gesture to be recognized based on the semantic sequence; The recognition module is specifically used to extract the image spatial features of each frame image in the key frame sequence through the convolution layer 3D CNN; the convolution layer 3D CNN includes multiple convolution layers and multiple downsampling layers, and the convolution kernels of all convolution layers are consistent; the time relationship features of the key frame sequence are extracted through the time recursive layer LSTM RNN; the LSTM RNN is connected in series to the last layer of the convolution layer 3D CNN network to combine the image spatial features and the time relationship features to obtain the spatiotemporal features of the key frame sequence; the spatiotemporal features are input into the classifier output layer to obtain the semantic sequence of the gesture to be recognized.

6. An interactive device based on gesture recognition, applied to a mixed reality device, wherein the mixed reality device includes a depth camera, a user interaction terminal and a mixed reality display terminal, characterized in that: The gesture recognition-based interaction device comprises: A first acquisition module, used to acquire a depth map of the gesture to be recognized; A second acquisition module, configured to acquire a key frame sequence of the depth map based on the depth map; A recognition module, used for inputting the key frame sequence into a pre-trained gesture recognition model to obtain a first semantic sequence of the gesture to be recognized; An acquisition module, which acquires a semantic result of the gesture to be recognized based on the first semantic sequence; An interaction module, used for outputting the semantic result and obtaining voice response information for the semantic result; An extraction module, used for extracting a voice segment based on the voice response information; A translation module, used for inputting the speech segment into a pre-trained speech translation model to obtain a second semantic sequence of the speech response information; A fourth acquisition module, used for acquiring a gesture graphic sequence based on the second semantic sequence; A display module, used for displaying the gesture graphic sequence; The recognition module is specifically used to extract the image spatial features of each frame image in the key frame sequence through the convolution layer 3D CNN; the convolution layer 3D CNN includes multiple convolution layers and multiple downsampling layers, and the convolution kernels of all convolution layers are consistent; the time relationship features of the key frame sequence are extracted through the time recursive layer LSTM RNN; the LSTM RNN is serially connected to the last layer of the convolution layer 3D CNN network; the image spatial features and the time relationship features are combined to obtain the spatiotemporal features of the key frame sequence; the spatiotemporal features are input into the classifier output layer to obtain the semantic sequence of the gesture to be recognized.

7. An electronic device, characterized in that: The device comprises: a memory, a processor, and a gesture recognition program stored in the memory and executable on the processor, wherein the gesture recognition program is configured to implement the steps of the gesture recognition method according to any one of claims 1 to 3; or The electronic device comprises: a memory, a processor, and an interactive program based on gesture recognition stored in the memory and executable on the processor, wherein the interactive program based on gesture recognition is configured to implement the steps of the interactive method based on gesture recognition as claimed in claim 4.

8. A mixed reality glasses, characterized in that: The mixed reality glasses include the electronic device as described in claim 7.

9. A storage medium, characterized in that: The storage medium stores a gesture recognition program, and when the gesture recognition program is executed by the processor, the steps of the gesture recognition method according to any one of claims 1 to 3 are implemented; or The storage medium stores an interaction program based on gesture recognition, and when the interaction program based on gesture recognition is executed by the processor, the steps of the interaction method based on gesture recognition as claimed in claim 4 are implemented.

Images