Gesture recognition method and related device

Abstract

The present application discloses a gesture recognition method and related equipment. The related embodiments can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The gesture image to be recognized and the millimeter wave gesture detection data for the target object can be collected. The gesture image to be recognized is subjected to gesture recognition to obtain an image gesture recognition result. The millimeter wave gesture detection data is divided and processed to obtain at least one gesture detection sub-data. The gesture recognition is performed according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain a millimeter wave gesture recognition result. Based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data, the target gesture recognition result corresponding to the target object is determined. The present application can combine the gesture image to be recognized and the millimeter wave gesture detection data to determine the gesture recognition result of the target object, which can improve the accuracy of gesture recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a gesture recognition method and related equipment. Background Art

[0002] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as human-computer interaction. Gesture recognition is currently a key research task in the field of human-computer interaction, and gesture recognition has a wide range of application value. By recognizing the human gestures in the gesture image to be recognized, the human operation can be judged, which can be widely used in various smart home devices; in addition, gesture recognition can also be used for human-computer interaction and various human-computer interaction applications can be developed.

[0003] However, in the current related technologies, gesture recognition algorithms generally only predict gestures based on the gesture images to be recognized collected by the camera. In this way, the generalization ability of the algorithm is general and the gesture recognition accuracy is low. Summary of the invention

[0004] The embodiments of the present application provide a gesture recognition method and related equipment. The related equipment may include a gesture recognition device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of gesture recognition.

[0005] The present application provides a gesture recognition method, including:

[0006] Collect gesture images to be recognized and millimeter wave gesture detection data of the target object;

[0007] Performing gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object;

[0008] dividing the millimeter wave gesture detection data to obtain at least one gesture detection sub-data;

[0009] Perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data;

[0010] Based on the image gesture recognition result and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data, a target gesture recognition result corresponding to the target object is determined.

[0011] Accordingly, an embodiment of the present application provides a gesture recognition device, including:

[0012] A collection unit, used for collecting gesture images to be recognized and millimeter wave gesture detection data of a target object;

[0013] A first recognition unit, configured to perform gesture recognition on the gesture image to be recognized, and obtain an image gesture recognition result corresponding to the target object;

[0014] A division unit, configured to divide the millimeter wave gesture detection data to obtain at least one gesture detection sub-data;

[0015] A second recognition unit is used to perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data;

[0016] The determination unit is used to determine a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data.

[0017] Optionally, in some embodiments of the present application, the first recognition unit may include an image division subunit, a first extraction subunit, a determination subunit and a first recognition subunit, as follows:

[0018] The image division subunit is used to perform image division processing on the gesture image to be recognized to obtain at least one gesture image block to be recognized;

[0019] The first extraction subunit is used to extract features from each gesture image block to be identified, and obtain sub-feature information corresponding to each gesture image block to be identified;

[0020] A determination subunit, configured to determine image feature information corresponding to the gesture image to be identified based on sub-feature information corresponding to each gesture image block to be identified;

[0021] The first recognition subunit is used to perform gesture recognition according to the image feature information to obtain an image gesture recognition result corresponding to the target object.

[0022] Optionally, in some embodiments of the present application, the first extraction subunit can be specifically used to perform convolution processing on each gesture image block to be identified based on a preset gradient convolution factor to obtain a first direction gradient value and a second direction gradient value corresponding to the gesture image block to be identified; and calculate sub-feature information corresponding to the gesture image block to be identified based on the first direction gradient value and the second direction gradient value.

[0023] Optionally, in some embodiments of the present application, the determination sub-unit can be specifically used to perform attention processing on the sub-feature information corresponding to each gesture image block to be identified, and obtain the attention weight of the sub-feature information corresponding to each gesture image block to be identified; according to the attention weight, the sub-feature information corresponding to each gesture image block to be identified is fused to obtain the image feature information corresponding to the gesture image to be identified.

[0024] Optionally, in some embodiments of the present application, the second recognition unit may include a second extraction subunit and a second recognition subunit, as follows:

[0025] The second extraction subunit is used to extract spectrum features of each gesture detection sub-data in at least one dimension to obtain millimeter wave feature information of the gesture detection sub-data in at least one dimension;

[0026] The second recognition subunit is used to perform gesture recognition according to the millimeter wave feature information of the gesture detection sub-data in at least one dimension, and obtain the millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0027] Optionally, in some embodiments of the present application, the gesture detection sub-data are gesture detection sub-data sorted according to detection time;

[0028] The second recognition unit may include a third extraction subunit and a third recognition subunit, as follows:

[0029] The third extraction sub-unit is used to extract the time series features of each gesture detection sub-data based on the previous and next gesture detection sub-data of the gesture detection sub-data, so as to obtain the millimeter wave feature information corresponding to the gesture detection sub-data;

[0030] The third recognition subunit is used to perform gesture recognition according to the millimeter wave feature information corresponding to the gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0031] Optionally, in some embodiments of the present application, the first recognition unit may be specifically configured to perform gesture recognition on the gesture image to be recognized through a gesture recognition model to obtain an image gesture recognition result corresponding to the target object.

[0032] Optionally, in some embodiments of the present application, the gesture recognition device may further include a training unit, which is used to train a gesture recognition model; the training unit may be specifically used to obtain training data, wherein the training data includes a sample image of a gesture to be recognized of a sample object, millimeter-wave gesture sample detection data, and an expected gesture recognition result corresponding to the sample object; through a preset gesture recognition model, gesture recognition is performed on the sample image of the gesture to be recognized to obtain an actual image gesture recognition result corresponding to the sample object; the millimeter-wave gesture sample detection data is divided and processed to obtain at least one gesture sample detection sub-data; gesture recognition is performed according to millimeter-wave feature information corresponding to each gesture sample detection sub-data to obtain an actual millimeter-wave gesture recognition result corresponding to each gesture sample detection sub-data; based on the actual image gesture recognition result, the actual millimeter-wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain a gesture recognition model.

[0033] Optionally, in some embodiments of the present application, the step of “adjusting parameters of a preset gesture recognition model based on the actual image gesture recognition result, the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result to obtain a gesture recognition model” may include:

[0034] Based on the actual image gesture recognition result and the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, voting is performed on the gesture recognition result of the sample object to obtain a target actual gesture recognition result of the sample object;

[0035] Based on the target actual gesture recognition result and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain a gesture recognition model.

[0036] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the gesture recognition method provided in the embodiment of the present application.

[0037] The embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in the gesture recognition method provided in the embodiment of the present application.

[0038] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the gesture recognition method provided in the embodiment of the present application.

[0039] The embodiment of the present application provides a gesture recognition method and related equipment, which can collect gesture images to be recognized and millimeter wave gesture detection data for a target object; perform gesture recognition on the gesture images to be recognized to obtain image gesture recognition results corresponding to the target object; divide and process the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; perform gesture recognition based on millimeter wave feature information corresponding to each gesture detection sub-data to obtain millimeter wave gesture recognition results corresponding to each gesture detection sub-data; determine the target gesture recognition result corresponding to the target object based on the image gesture recognition results and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data. The present application can combine the gesture images to be recognized and the millimeter wave gesture detection data to determine the gesture recognition results of the target object, which can improve the accuracy of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1a is a schematic diagram of a scene of a gesture recognition method provided in an embodiment of the present application;

[0042] Figure 1b is a flowchart of a gesture recognition method provided by an embodiment of the present application;

[0043] Figure 1c is an illustration of a gesture recognition method provided in an embodiment of the present application;

[0044] Figure 1d is a model architecture diagram of the gesture recognition method provided in the embodiment of the present application;

[0045] Figure 1e is another flow chart of the gesture recognition method provided by an embodiment of the present application;

[0046] Figure 1f is another illustration of the gesture recognition method provided by an embodiment of the present application;

[0047] Figure 1g is another illustration of the gesture recognition method provided by an embodiment of the present application;

[0048] Figure 1h is a page schematic diagram of a gesture recognition method provided in an embodiment of the present application;

[0049] Figure 1i is another flow chart of the gesture recognition method provided by an embodiment of the present application;

[0050] Figure 1j is another illustration of the gesture recognition method provided by an embodiment of the present application;

[0051] Figure 2 is another flow chart of the gesture recognition method provided by an embodiment of the present application;

[0052] Figure 3 is a structural diagram of a gesture recognition device provided in an embodiment of the present application;

[0053] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0055] The embodiment of the present application provides a gesture recognition method and related equipment, and the related equipment may include a gesture recognition device, an electronic device, a computer-readable storage medium and a computer program product. The gesture recognition device may be integrated in an electronic device, and the electronic device may be a terminal or a server.

[0056] It is understandable that the gesture recognition method of this embodiment can be executed on a terminal, can be executed on a server, or can be executed by both the terminal and the server. The above examples should not be construed as limiting the present application.

[0057] like Figure 1a As shown, a gesture recognition method is performed by a terminal and a server together as an example. The gesture recognition system provided in the embodiment of the present application includes a terminal 10 and a server 11, etc. The terminal 10 and the server 11 are connected via a network, such as a wired or wireless network connection, etc., wherein the gesture recognition device can be integrated in the terminal.

[0058] The terminal 10 can be used to: collect gesture images to be identified and millimeter wave gesture detection data for the target object; send gesture images to be identified and millimeter wave gesture detection data to the server 11 to trigger the server 11 to perform gesture recognition based on the gesture images to be identified and the millimeter wave gesture detection data, and obtain the target gesture recognition result corresponding to the target object; the terminal 10 can also receive the target gesture recognition result sent by the server 11. The terminal 10 can include a mobile phone, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, a tablet computer, a laptop computer, or a personal computer (PC). A client can also be set on the terminal 10, and the client can be an application client or a browser client, etc.

[0059] Among them, the server 11 can be used to: receive the gesture image to be identified and the millimeter wave gesture detection data of the target object sent by the terminal 10, perform gesture recognition on the gesture image to be identified, and obtain the image gesture recognition result corresponding to the target object; divide and process the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data; determine the target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data, and send the target gesture recognition result to the terminal 10. Among them, the server 11 can be a single server, or a server cluster or cloud server composed of multiple servers. In the gesture recognition method or device disclosed in the present application, multiple servers can be composed of a blockchain, and the server is a node on the blockchain.

[0060] The steps of gesture recognition and the like in the server 11 may also be executed by the terminal 10 .

[0061] The gesture recognition method provided in the embodiment of the present application relates to computer vision technology and machine learning in the field of artificial intelligence.

[0062] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Among them, artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0063] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0064] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0065] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0066] This embodiment will be described from the perspective of a gesture recognition device. The gesture recognition device may be integrated into an electronic device, which may be a server or a terminal.

[0067] It is understandable that in the specific implementation of the present application, user information, such as user gestures and other related data, is involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0068] The gesture recognition method of the embodiment of the present application can be applied to scenes such as gesture payment, etc. The present embodiment can be applied to various scenes such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0069] like Figure 1b As shown, the specific process of the gesture recognition method can be as follows:

[0070] 101. Collect gesture images to be recognized and millimeter wave gesture detection data of the target object.

[0071] Among them, the target object may be an object to be gesture recognized; the gesture image to be recognized may be collected by an image sensor, and may include the gesture of the target object; the millimeter wave gesture detection data may specifically be millimeter wave data of the gesture of the target object, which may be collected by a millimeter wave radar sensor.

[0072] Specifically, millimeter waves usually refer to electromagnetic waves in the frequency range of 30 to 300 GHz (wavelength of 1 to 10 mm), and their wavelength is between centimeter waves and light waves. Therefore, millimeter waves have the advantages of both microwave guidance and photoelectric guidance. Compared with centimeter wave radar sensors, millimeter wave radar sensors are small in size, easy to integrate, and have high spatial resolution.

[0073] In one embodiment, an image acquisition device on an electronic device may capture an image of a gesture to be recognized. For example, when a shooting instruction is received, the image acquisition device is turned on to capture an image, and the captured image is used as the image of the gesture to be recognized. The image acquisition device may be a camera, such as a time-of-flight depth camera.

[0074] Among them, the time-of-flight depth camera, also known as TOF (Time of Flight) depth camera, has the basic principle of continuously emitting light pulses (generally invisible light) to the observed object, and then using a sensor to receive the light returned from the object, and obtaining the distance of the target object by detecting the flight (round-trip) time of the light pulse.

[0075] In another embodiment, millimeter-wave gesture detection data may be collected by a millimeter-wave radar sensor on an electronic device. For example, upon receiving a collection instruction, the millimeter-wave radar sensor uses a high-frequency circuit to generate electromagnetic waves of a specific debugging frequency, and transmits and receives electromagnetic waves reflected by the target object through an antenna, and then calculates the parameters of sending and receiving the electromagnetic waves to obtain the gesture data of the target object (such as relative distance, relative speed, angle, direction of movement, etc.), and uses the acquired gesture data as the millimeter-wave gesture detection data of the target object.

[0076] The gesture recognition method provided in the present application can read the hand motion data of the target object based on the millimeter wave sensor and the image sensor (specifically, it can include the gesture image to be recognized and the millimeter wave gesture detection data), and perform gesture recognition on the acquired hand motion data, and interpret it as the corresponding gesture. Since gesture recognition is performed in combination with the gesture image to be recognized and the millimeter wave gesture detection data, the accuracy of gesture recognition can be improved. The power consumption of the millimeter wave sensor and the image sensor is relatively low, and the accuracy of gesture recognition can also be improved on low-power devices.

[0077] 102. Perform gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object.

[0078] Optionally, in this embodiment, the step of “performing gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object” may include:

[0079] Performing image segmentation processing on the gesture image to be recognized to obtain at least one gesture image block to be recognized;

[0080] Extract features from each gesture image block to be identified, and obtain sub-feature information corresponding to each gesture image block to be identified;

[0081] Determining image feature information corresponding to the gesture image to be identified based on sub-feature information corresponding to each gesture image block to be identified;

[0082] Gesture recognition is performed according to the image feature information to obtain an image gesture recognition result corresponding to the target object.

[0083] In some embodiments, before performing image segmentation processing on the gesture image to be recognized, the gesture image to be recognized may be preprocessed, and the preprocessing may include image size adjustment, image data enhancement, and / or image rotation, etc. The image data enhancement may include histogram equalization, sharpening, and smoothing, etc.

[0084] There are many ways to process the image segmentation. For example, the gesture image to be identified can be evenly divided into multiple gesture image blocks of the same size. Specifically, the gesture image to be identified can be divided according to a preset resolution, which can be set according to actual conditions. For example, the preset resolution can be 60*60, and the resolution of each gesture image block to be identified obtained by division is 60*60.

[0085] Optionally, in some embodiments, feature extraction is performed on each gesture image block to be recognized. Specifically, convolution processing and pooling processing can be performed on each gesture image block to be recognized, wherein the pooling processing can include maximum pooling (Max-pooling, Maximum Pooling), average pooling (Avg-pooling, Average Pooling) and generalized mean pooling (GEM-pooling, Generalized-mean Pooling), etc.

[0086] Optionally, in this embodiment, the step of “extracting features from each gesture image block to be identified to obtain sub-feature information corresponding to each gesture image block to be identified” may include:

[0087] For each gesture image block to be identified, convolution processing is performed on the gesture image block to be identified based on a preset gradient convolution factor to obtain a first direction gradient value and a second direction gradient value corresponding to the gesture image block to be identified;

[0088] The sub-feature information corresponding to the gesture image block to be recognized is calculated according to the first directional gradient value and the second directional gradient value.

[0089] Among them, the preset gradient convolution factor can be set according to actual conditions, and this embodiment does not limit this. Specifically, the preset gradient convolution factor may include a gradient convolution factor in the horizontal direction (recorded as a first gradient convolution factor) and a gradient convolution factor in the vertical direction (recorded as a second gradient convolution factor). The first gradient convolution factor can detect the edge in the horizontal direction of the image, and the second gradient convolution factor can detect the edge in the vertical direction of the image. Specifically, based on the preset gradient convolution factor, the gesture image block to be identified is convolved, and the first gradient convolution factor and the second gradient convolution factor can be convolved with the gesture image block to be identified, respectively, to obtain the first direction gradient value and the second direction gradient value.

[0090] The first direction gradient value may be a horizontal gradient value, and the second direction gradient value may be a vertical gradient value. Based on the first direction gradient value and the second direction gradient value, the calculated sub-feature information may include gradient feature information of the gesture image to be recognized.

[0091] The step of “calculating the sub-feature information corresponding to the gesture image block to be identified according to the first directional gradient value and the second directional gradient value” may include:

[0092] The first direction gradient value and the second direction gradient value are merged to obtain the gradient amplitude information corresponding to the gesture image block to be identified;

[0093] Performing direction calculation processing on the first direction gradient value and the second direction gradient value to obtain a gradient direction corresponding to the gesture image block to be recognized;

[0094] Based on the gradient amplitude information and the gradient direction, sub-feature information corresponding to the gesture image block to be recognized is determined.

[0095] There are many ways to fuse the first direction gradient value and the second direction gradient value, which are not limited in this embodiment. For example, the fusion method may be a square sum operation.

[0096] In some embodiments, the gradient amplitude information and the gradient direction may be directly determined as sub-feature information, and in other embodiments, the gradient amplitude information and the gradient direction may be feature combined to obtain sub-feature information of the gesture image block to be identified.

[0097] In a specific embodiment, the preset gradient convolution factor can be a Sobel operator, which is an edge detection filter. Specifically, the Sobel operator performs a weighted difference operation on the grayscale values ​​of the four neighborhoods of each pixel in the image, and combines them to reach an extreme value at the edge to perform edge detection. There are two Sobel operators, one for detecting horizontal edges, denoted by G x ; The other is to detect vertical edges, denoted as G y ; In the process of algorithm implementation, G x and G y It can be a 3×3 template, and the 3×3 template is used as the convolution kernel to perform convolution and operation with each pixel in the image block to extract the edge. The specific process of extracting the gradient amplitude information and gradient direction by the Sobel operator is shown in equations (1) and (2) respectively:

[0098]

[0099]

[0100] Among them, I x and I y Represents the gradient values ​​in the horizontal and vertical directions respectively, I x Specifically, it is the first direction gradient value described in the above embodiment, Iy Specifically, it is the second direction gradient value described in the above embodiment, M(x, y) represents the amplitude value of the gradient, that is, the gradient amplitude information, and θ(x, y) represents the gradient direction.

[0101] Through the above processing, the directional gradient histogram (Histogram of Oriented Gradient, Hog) features (specifically including gradient amplitude information and gradient direction) of each gesture image block to be identified can be extracted, and then gesture recognition can be performed based on the directional gradient histogram features of each gesture image block to be identified. Specifically, the directional gradient histogram features of each gesture image block to be identified can be fused to obtain image feature information corresponding to the gesture image to be identified, and then gesture recognition can be performed based on the image feature information. There are many ways to fuse features, such as weighted fusion or splicing processing, which is not limited in this embodiment.

[0102] Among them, the Hog feature is specifically a feature descriptor used for object detection in computer vision and image processing. The Hog feature constructs the feature by calculating and counting the gradient direction histogram of the local area of ​​the image.

[0103] Optionally, in this embodiment, the step of “determining image feature information corresponding to the gesture image to be identified according to sub-feature information corresponding to each gesture image block to be identified” may include:

[0104] Performing attention processing on the sub-feature information corresponding to each gesture image block to be identified, and obtaining the attention weight of the sub-feature information corresponding to each gesture image block to be identified;

[0105] According to the attention weight, the sub-feature information corresponding to each to-be-recognized gesture image block is fused to obtain the image feature information corresponding to the to-be-recognized gesture image.

[0106] There are many ways to fuse the sub-feature information corresponding to each gesture image block to be recognized, which is not limited in this embodiment. For example, the fusion method may be weighted fusion.

[0107] Among them, the step of "performing attention processing on the sub-feature information corresponding to each gesture image block to be identified to obtain the attention weight of the sub-feature information corresponding to each gesture image block to be identified" may include:

[0108] Performing hyperbolic tangent processing on the sub-feature information corresponding to each gesture image block to be identified to obtain a processing result;

[0109] The processing result is subjected to nonlinear mapping processing to obtain the attention weight of the sub-feature information corresponding to each gesture image block to be identified.

[0110] In some embodiments, the sub-feature information (i.e., Hog features) corresponding to each gesture image block to be identified can be converted into a one-dimensional array, and then the one-dimensional arrays can be connected end to end and combined into a one-dimensional vector, which is used as the final image feature information.

[0111] Optionally, in this embodiment, the step of “performing gesture recognition according to the image feature information to obtain an image gesture recognition result corresponding to the target object” may include:

[0112] Predicting the probability that the gesture image to be identified belongs to each preset gesture category according to the image feature information;

[0113] Based on the probability, an image gesture recognition result corresponding to the target object is determined.

[0114] The probability that the gesture image to be identified belongs to each preset gesture category can be predicted by a classifier. The classifier can be a support vector machine (SVM), a recurrent neural network, a fully connected deep neural network (DNN), etc., which is not limited in this embodiment.

[0115] The preset gesture categories may include multiple gestures such as sliding left, sliding right, etc.

[0116] Among them, in some embodiments, the preset gesture category with the highest probability can be determined as the image gesture recognition result corresponding to the target object, and the preset gesture category with a probability greater than a preset value can be determined as the image gesture recognition result corresponding to the target object. The preset value can be set according to actual conditions.

[0117] 103. Divide and process the millimeter wave gesture detection data to obtain at least one gesture detection sub-data.

[0118] There are many ways to divide the millimeter wave gesture detection data, which is not limited in this embodiment. Specifically, the millimeter wave gesture detection data is gesture detection data within a certain time period. In some embodiments, the millimeter wave gesture detection data can be divided into gesture detection sub-data of equal duration. For example, the millimeter wave gesture detection data is divided into blocks based on a preset division duration (such as 5ms, which can be adjusted) to obtain each gesture detection sub-data.

[0119] Specifically, in some embodiments, before the millimeter wave gesture detection data is divided and processed, the millimeter wave gesture detection data may be filtered and cleaned. The filtering and cleaning process may be a median filter. The code corresponding to the median filter cleaning may be as follows:

[0120]

[0121]

[0122] Among them, rounded down means rounding the number downward (in the direction of decreasing absolute value) close to zero.

[0123] 104. Perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain a millimeter wave gesture recognition result corresponding to each gesture detection sub-data.

[0124] Optionally, in this embodiment, the step of “performing gesture recognition according to millimeter wave feature information corresponding to each gesture detection sub-data to obtain a millimeter wave gesture recognition result corresponding to each gesture detection sub-data” may include:

[0125] For each gesture detection sub-data, extracting spectrum features of the gesture detection sub-data in at least one dimension to obtain millimeter wave feature information of the gesture detection sub-data in at least one dimension;

[0126] Gesture recognition is performed according to the millimeter wave feature information of the gesture detection sub-data in at least one dimension to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0127] The spectral feature extraction in at least one dimension may include zero-crossing rate feature extraction, spectral flatness feature extraction, and spectral centroid feature extraction, etc., which are not limited in this embodiment. The millimeter wave feature information in at least one dimension obtained by extracting the spectral feature in at least one dimension may include the zero-crossing rate feature, spectral flatness feature, and spectral centroid feature of the gesture detection sub-data.

[0128] Specifically, binary encoding can be used to extract zero-crossing rate features, where the zero-crossing rate (ZCR) refers to the ratio of a signal's sign change, such as a signal changing from a positive number to a negative number or vice versa. For spectrum flatness feature extraction and spectrum centroid feature extraction, the fft size (signal window width of Fourier transform) can be used for extraction, and then the extracted spectrum flatness feature and spectrum centroid feature can be standardized.

[0129] In this embodiment, spectrum feature extraction is adopted to make the extracted millimeter wave feature information have a better anti-noise effect.

[0130] The step of “performing gesture recognition according to the millimeter wave feature information of the gesture detection sub-data in at least one dimension to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data” may include:

[0131] Fusing the millimeter wave feature information of the gesture detection sub-data in each dimension to obtain target millimeter wave feature information corresponding to the gesture detection sub-data;

[0132] Gesture recognition is performed based on the target millimeter wave characteristic information to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0133] There are many ways to fuse the millimeter wave feature information in various dimensions, which are not limited in this embodiment. For example, the fusion method can be weighted processing or splicing processing.

[0134] Among them, gesture recognition is performed based on the target millimeter wave feature information, specifically, the probability that the corresponding gesture detection sub-data belongs to each preset gesture category is predicted based on the target millimeter wave feature information; based on the probability, the millimeter wave gesture recognition result corresponding to the gesture detection sub-data is determined. Among them, the probability that the gesture detection sub-data belongs to each preset gesture category can be predicted by a classifier. In some embodiments, the preset gesture category with the highest probability can be determined as the millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0135] Optionally, in this embodiment, the gesture detection sub-data are gesture detection sub-data sorted according to detection time;

[0136] The step of “performing gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data” may include:

[0137] For each gesture detection sub-data, based on the previous and next gesture detection sub-data of the gesture detection sub-data, extract the time series features of the gesture detection sub-data to obtain the millimeter wave feature information corresponding to the gesture detection sub-data;

[0138] Gesture recognition is performed according to the millimeter wave feature information corresponding to the gesture detection sub-data to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0139] Among them, the memory network model can be used to extract the time series features of the gesture detection sub-data based on the previous and next gesture detection sub-data of the gesture detection sub-data, so as to obtain the millimeter wave feature information corresponding to the gesture detection sub-data. Among them, the memory network model can be a long short-term memory network (LSTM, Long Short-Term Memory), etc. LSTM is a time recurrent neural network, that is, a recurrent neural network (RNN, Recurrent Neural Network). LSTM is more suitable for extracting semantic features from time series data, and is often used to extract semantic features from contextual information in natural language processing tasks. LSTM can selectively forget part of the historical data and add part of the current input data through three gate structures (input gate, forget gate, output gate), and finally integrate it into the current state and generate an output state.

[0140] Among them, gesture recognition is performed according to the millimeter wave feature information corresponding to the gesture detection sub-data, and specifically, the probability that the corresponding gesture detection sub-data belongs to each preset gesture category is predicted based on the millimeter wave feature information; based on the probability, the millimeter wave gesture recognition result corresponding to the gesture detection sub-data is determined.

[0141] 105. Determine a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data.

[0142] In this embodiment, based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data, voting can be performed on each preset gesture category to obtain a target gesture recognition result corresponding to the target object. Specifically, the preset gesture category with the most votes can be determined as the target gesture recognition result of the target object.

[0143] For example, the image gesture recognition result is that the gesture of the target object is sliding to the left, and the millimeter-wave gesture recognition results corresponding to the three gesture detection sub-data obtained are sliding to the right, sliding to the left, and shaking up and down, respectively. Then, voting is performed on each preset gesture category, and the preset gesture category of sliding to the right can get 1 vote, the preset gesture category of sliding to the left can get 2 votes, and the preset gesture category of shaking up and down can get 1 vote. Therefore, the target gesture recognition result of the target object is sliding to the left.

[0144] Optionally, in this embodiment, the step of “performing gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object” may include:

[0145] The gesture recognition model is used to perform gesture recognition on the gesture image to be recognized, and an image gesture recognition result corresponding to the target object is obtained.

[0146] The step of “performing gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data” may include:

[0147] By using the gesture recognition model, gesture recognition is performed according to the millimeter wave feature information corresponding to each gesture detection sub-data, and a millimeter wave gesture recognition result corresponding to each gesture detection sub-data is obtained.

[0148] The gesture recognition model may be a residual network (ResNet) and a densely connected convolutional network (DenseNet), etc. However, it should be understood that the gesture recognition model of this embodiment is not limited to the types listed above.

[0149] Specifically, the gesture recognition model can be a fusion neural network, which is a neural network model with more than one prediction task, wherein the gesture recognition model can predict the image gesture recognition result corresponding to the target object based on the extracted image feature information of the gesture image to be recognized, and can also predict the millimeter wave gesture recognition results corresponding to the gesture detection sub-data based on the extracted millimeter wave feature information of each gesture detection sub-data.

[0150] In a specific embodiment, if there are 8 preset gesture categories, such as Figure 1c As shown, they are: sliding to the left, sliding to the right, moving downward, moving upward, shaking upward, shaking downward, turning the back of the hand to the palm, and turning the palm to the back of the hand; this embodiment can be classified based on the 8-classification method, and there can be multiple corresponding gesture recognition models. For example, the gesture recognition model used can be as follows Figure 1d A shown TD (denoted as Model 1), A 3D (denoted as Model 2), or A VI (denoted as Model 3), wherein the radar input tensor is specifically the millimeter wave gesture detection data in the above embodiment, and the image input tensor is specifically the gesture image to be recognized in the above embodiment. The architectures of these three models are described in detail as follows:

[0151] Model 1 may include a time distributed encoding cell, a connection layer, and a fully connected layer, etc. The time distributed encoding cell may include a convolution layer, a maximum pooling layer, and a linear rectifier unit, etc. Specifically, layers 1-2 of model 1 are convolutional neural two-dimensional networks. If the number of layers is greater than 8, an LSTM (long short-term memory network) layer may be added. The model architecture can be represented by the following code:

[0152] layers.Conv2D / / Two-dimensional convolution layer

[0153] layers.GlobalAveragePooling2D(), / / 2D global average pooling layer

[0154] layers.Conv2D / / Two-dimensional convolution layer

[0155] layers.ReLU(), / / Linear rectification layer

[0156] layers.Dropout(.5), / / used to prevent overfitting

[0157] layers.LSTM(), / / LSTM layer

[0158] layers.MaxPooling1D(), / / Maximum pooling layer

[0159] layers.Dense(8), / / density layer

[0160] layers.Dense(self.num_of_classes,activation='softmax') / / Density layer (classification, logical activation)

[0161] Model 2 may include an encoding cell, a connection layer, and a fully connected layer, etc. The encoding cell may include a three-dimensional convolutional layer, a three-dimensional maximum pooling layer, and a linear rectifier unit, etc. Specifically, layers 1-2 of model 2 may be a convolutional neural three-dimensional network, and if the number of convolutional layers is greater than 8, an LSTM layer is added.

[0162] Among them, model 3 may include an encoding cell, a connection layer, a fully connected layer, etc. The encoding unit may include a convolutional layer, a maximum pooling layer, and a linear rectifier unit, etc. Specifically, layers 1-2 of model 3 may be a convolutional neural two-dimensional network, and the third layer may be a maximum two-dimensional convolutional neural network.

[0163] For the training of the gesture recognition model, the parameters and data required by the network model can be trained according to the specific model selected.

[0164] Specifically, Figure 1eAs shown, the gesture recognition method of this embodiment may include five modules, namely, predefined gestures, collection of training data, feature engineering, model training, and model deployment. Among them, the predefined gestures are preset gesture categories corresponding to the defined gestures (such as sliding left or sliding right, etc.); the collection of training data can specifically obtain the gesture data of the sample object through a sensor for capturing gestures (such as a camera or a millimeter wave wireless radar sensor, etc.), and then perform feature extraction and classification on these gesture data through a preset gesture recognition model, and train the preset gesture recognition model according to the actual classification results and the expected classification results, obtain the trained gesture recognition model, and put it into practical application.

[0165] It should be noted that the gesture recognition model can be trained by other equipment and then provided to the gesture recognition device, or the gesture recognition device can perform training by itself.

[0166] If the gesture recognition device performs training by itself, before the step of "performing gesture recognition on the gesture image to be recognized through a gesture recognition model to obtain an image gesture recognition result corresponding to the target object", the gesture recognition method may further include:

[0167] Acquire training data, wherein the training data includes a sample image of a gesture to be recognized of a sample object, millimeter wave gesture sample detection data, and an expected gesture recognition result corresponding to the sample object;

[0168] By using a preset gesture recognition model, gesture recognition is performed on the gesture sample image to be recognized, so as to obtain an actual image gesture recognition result corresponding to the sample object;

[0169] The millimeter wave gesture sample detection data is divided and processed to obtain at least one gesture sample detection sub-data; gesture recognition is performed according to the millimeter wave feature information corresponding to each gesture sample detection sub-data to obtain an actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data;

[0170] Based on the actual image gesture recognition result, the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain the gesture recognition model.

[0171] In this embodiment, the collection of training data may include the collection of sample images of gestures to be recognized and the collection of millimeter-wave gesture sample detection data.

[0172] The collection of sample images of gestures to be recognized can be done in the following way: Figure 1fIn the payment scenario shown, the gesture image of palm swiping is obtained through the image acquisition device. Specifically, the palm can be facing the camera and the palm can be kept more than two fists away from the payment page. The resolution of the collected gesture sample image is related to the camera of the machine, and generally the maximum resolution supported by the camera is adopted.

[0173] Among them, for the collection of millimeter wave gesture sample detection data, you can first determine the target gesture category (such as sliding up), collect a certain amount of gesture data of the target gesture category as a positive sample, then determine the negative sample gesture (such as shaking left and right), collect the same amount of negative sample gesture data as the positive sample, and then perform data cleaning on the collected positive and negative samples, such as finding duplicate values, missing values, and outliers, etc., clear invalid data, and obtain the final millimeter wave gesture sample detection data. Specifically, the millimeter wave gesture sample detection data can also be cleaned by median filtering. Figure 1g As shown, the millimeter wave gesture sample detection data can be collected by a microphone inertial measurement sensor. Specifically, the collected data can be amplified by a preamplifier, and then converted by an analog-to-digital converter and a multiplexer to obtain computer-processable millimeter wave gesture sample detection data. Then, the millimeter wave gesture sample detection data can be divided and processed to obtain at least one gesture sample detection sub-data, and each gesture sample detection sub-data can correspond to different gesture recognition results.

[0174] Optionally, in this embodiment, the step of “adjusting parameters of a preset gesture recognition model based on the actual image gesture recognition result, the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result to obtain a gesture recognition model” may include:

[0175] Based on the actual image gesture recognition result and the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, voting is performed on the gesture recognition result of the sample object to obtain a target actual gesture recognition result of the sample object;

[0176] Based on the target actual gesture recognition result and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain a gesture recognition model.

[0177] The target actual gesture recognition result may specifically be the actual probability that the gesture of the sample object belongs to each preset gesture category, and the expected gesture recognition result may specifically be the expected probability that the gesture of the sample object belongs to each preset gesture category.

[0178] Among them, the gesture recognition results of the sample objects are voted on, specifically, the votes of the sample object's gesture in each preset gesture category are counted, and the actual probability that the sample object's gesture belongs to each preset gesture category is determined based on the statistical results, thereby obtaining the target actual gesture recognition result of the sample object.

[0179] In this embodiment, the training process may specifically be to first predict the target actual gesture recognition result of the sample object, then use the back propagation algorithm to adjust the parameters of the preset gesture recognition model, and optimize the preset gesture recognition model based on the target actual gesture recognition result and the expected gesture recognition result, so that the predicted target actual gesture recognition result approaches the expected gesture recognition result, and obtain a trained gesture recognition model. Specifically, the loss value between the predicted target actual gesture recognition result and the expected gesture recognition result can be made less than a preset value, and the preset value can be set according to actual conditions.

[0180] After obtaining the trained gesture recognition model, it can be put into practical application. The algorithm can be run on the palm-swiping terminal equipment to collect the gesture image to be recognized of the target object through the camera of the palm-swiping terminal, and the millimeter-wave gesture detection data of the target object can be collected through the millimeter-wave radar sensor. The collected gesture image can also be displayed on the display screen of the palm-swiping terminal, such as Figure 1h Then, the gesture recognition model is used to perform gesture recognition on the gesture image to be recognized and the millimeter wave gesture detection data.

[0181] Among them, the deployment of gesture recognition model can be as follows Figure 1i As shown, the millimeter wave gesture detection data (S R ), and multiple gesture images captured by the camera (S T ) is preprocessed, and then the preprocessed millimeter wave gesture detection data τ is respectively processed by the gesture recognition model. R (S R ), and the preprocessed gesture image to be recognized τ T (S T ) to extract features, and obtain millimeter-wave feature information corresponding to each gesture detection sub-data, and image feature information corresponding to the gesture image to be recognized; then, gesture recognition is performed according to the millimeter-wave feature information corresponding to each gesture detection sub-data, and the millimeter-wave gesture recognition results corresponding to each gesture detection sub-data are obtained; and gesture recognition is performed based on the image feature information to obtain the image gesture recognition result corresponding to the target object; finally, the image gesture recognition result and the millimeter-wave gesture recognition results corresponding to each gesture detection sub-data are combined to obtain and output the target gesture recognition result corresponding to the target object.

[0182] In a specific embodiment, if Figure 1j As shown, multiple gesture images for the target object can be collected by a time-of-flight camera, and millimeter-wave gesture detection data can be collected by a millimeter-wave radar sensor (which can be 60GHz). Then, a suitable fusion neural network is selected for gesture recognition. Different fusion neural networks correspond to different evaluation methods. Different fusion neural networks have different memory requirements (Memory requirements) and inference time (Inference time) for the device. Specifically, different fusion neural networks have different requirements for artificial intelligence systems, acceleration chips, and neural computing sticks. For example, a fusion neural network for gesture recognition can be selected from Model 1, Model 2, and Model 3 in the above embodiment. The fusion neural network includes a radar neural network and an image neural network. Based on the multimedia control command, the millimeter-wave gesture detection data and the gesture image to be recognized can be input into the fusion neural network, and the millimeter-wave gesture detection data can be feature extracted by the radar neural network to obtain the millimeter-wave gesture recognition result corresponding to each gesture detection sub-data; the gesture image to be recognized is feature extracted by the image neural network to obtain the image gesture recognition result; finally, the image gesture recognition result and the millimeter-wave gesture recognition result corresponding to the target object are combined to obtain and output.

[0183] In this embodiment, gesture recognition specifically refers to a solution in which a millimeter-wave radar sensor or camera transmits gesture signals to an algorithm for prediction after the target object makes a corresponding gesture. In a specific scenario, relevant payment can also be made after the corresponding gesture is recognized.

[0184] As can be seen from the above, this embodiment can collect gesture images to be identified and millimeter wave gesture detection data for the target object; perform gesture recognition on the gesture images to be identified to obtain the image gesture recognition result corresponding to the target object; divide and process the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; perform gesture recognition based on the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data; determine the target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data. This application can combine the gesture images to be identified and the millimeter wave gesture detection data to determine the gesture recognition result of the target object, which can improve the accuracy of gesture recognition.

[0185] According to the method described in the above embodiment, the gesture recognition device will be further described in detail below by taking the example of being specifically integrated in a terminal.

[0186] The present application embodiment provides a gesture recognition method, such as Figure 2As shown, the specific process of the gesture recognition method can be as follows:

[0187] 201. The terminal collects a gesture image to be recognized and millimeter wave gesture detection data of a target object.

[0188] Among them, the target object may be an object to be gesture recognized; the gesture image to be recognized may be collected by an image sensor, and may include the gesture of the target object; the millimeter wave gesture detection data may specifically be millimeter wave data of the gesture of the target object, which may be collected by a millimeter wave radar sensor.

[0189] 202. The terminal performs gesture recognition on the gesture image to be recognized, and obtains an image gesture recognition result corresponding to the target object.

[0190] Optionally, in this embodiment, the step of “performing gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object” may include:

[0191] Performing image segmentation processing on the gesture image to be recognized to obtain at least one gesture image block to be recognized;

[0192] Extract features from each gesture image block to be identified, and obtain sub-feature information corresponding to each gesture image block to be identified;

[0193] Determining image feature information corresponding to the gesture image to be identified based on sub-feature information corresponding to each gesture image block to be identified;

[0194] Perform gesture recognition according to the image feature information to obtain an image gesture recognition result corresponding to the target object.

[0195] In some embodiments, before performing image segmentation processing on the gesture image to be recognized, the gesture image to be recognized may be preprocessed, and the preprocessing may include image size adjustment, image data enhancement, and / or image rotation, etc. The image data enhancement may include histogram equalization, sharpening, and smoothing, etc.

[0196] Optionally, in this embodiment, the step of “extracting features from each gesture image block to be identified to obtain sub-feature information corresponding to each gesture image block to be identified” may include:

[0197] For each gesture image block to be identified, convolution processing is performed on the gesture image block to be identified based on a preset gradient convolution factor to obtain a first direction gradient value and a second direction gradient value corresponding to the gesture image block to be identified;

[0198] The sub-feature information corresponding to the gesture image block to be recognized is calculated according to the first directional gradient value and the second directional gradient value.

[0199] Among them, the preset gradient convolution factor can be set according to actual conditions, and this embodiment does not limit this. Specifically, the preset gradient convolution factor may include a gradient convolution factor in the horizontal direction (recorded as a first gradient convolution factor) and a gradient convolution factor in the vertical direction (recorded as a second gradient convolution factor). The first gradient convolution factor can detect the edge in the horizontal direction of the image, and the second gradient convolution factor can detect the edge in the vertical direction of the image. Specifically, based on the preset gradient convolution factor, the gesture image block to be identified is convolved, and the first gradient convolution factor and the second gradient convolution factor can be convolved with the gesture image block to be identified, respectively, to obtain the first direction gradient value and the second direction gradient value.

[0200] The first direction gradient value may be a horizontal gradient value, and the second direction gradient value may be a vertical gradient value. Based on the first direction gradient value and the second direction gradient value, the calculated sub-feature information may include gradient feature information of the gesture image to be recognized.

[0201] The step of “calculating the sub-feature information corresponding to the gesture image block to be identified according to the first directional gradient value and the second directional gradient value” may include:

[0202] The first direction gradient value and the second direction gradient value are merged to obtain the gradient amplitude information corresponding to the gesture image block to be identified;

[0203] Performing direction calculation processing on the first direction gradient value and the second direction gradient value to obtain a gradient direction corresponding to the gesture image block to be recognized;

[0204] Based on the gradient amplitude information and the gradient direction, sub-feature information corresponding to the gesture image block to be recognized is determined.

[0205] There are many ways to fuse the first direction gradient value and the second direction gradient value, which are not limited in this embodiment. For example, the fusion method may be a square sum operation.

[0206] In some embodiments, the gradient amplitude information and the gradient direction may be directly determined as sub-feature information, and in other embodiments, the gradient amplitude information and the gradient direction may be feature combined to obtain sub-feature information of the gesture image block to be identified.

[0207] Optionally, in this embodiment, the step of “determining image feature information corresponding to the gesture image to be identified according to sub-feature information corresponding to each gesture image block to be identified” may include:

[0208] Performing attention processing on the sub-feature information corresponding to each gesture image block to be identified, and obtaining the attention weight of the sub-feature information corresponding to each gesture image block to be identified;

[0209] According to the attention weight, the sub-feature information corresponding to each gesture image block to be recognized is fused to obtain the image feature information corresponding to the gesture image to be recognized.

[0210] There are many ways to fuse the sub-feature information corresponding to each gesture image block to be recognized, which is not limited in this embodiment. For example, the fusion method may be weighted fusion.

[0211] Among them, the step of "performing attention processing on the sub-feature information corresponding to each gesture image block to be identified to obtain the attention weight of the sub-feature information corresponding to each gesture image block to be identified" may include:

[0212] Performing hyperbolic tangent processing on the sub-feature information corresponding to each gesture image block to be identified to obtain a processing result;

[0213] The processing result is subjected to nonlinear mapping processing to obtain the attention weight of the sub-feature information corresponding to each gesture image block to be identified.

[0214] In some embodiments, the sub-feature information (i.e., Hog features) corresponding to each gesture image block to be identified can be converted into a one-dimensional array, and then the one-dimensional arrays can be connected end to end and combined into a one-dimensional vector, which is used as the final image feature information.

[0215] Optionally, in this embodiment, the step of “performing gesture recognition according to the image feature information to obtain an image gesture recognition result corresponding to the target object” may include:

[0216] Predicting the probability that the gesture image to be identified belongs to each preset gesture category according to the image feature information;

[0217] Based on the probability, an image gesture recognition result corresponding to the target object is determined.

[0218] The probability that the gesture image to be identified belongs to each preset gesture category can be predicted by a classifier. The classifier can be a support vector machine (SVM), a recurrent neural network, a fully connected deep neural network (DNN), etc., which is not limited in this embodiment.

[0219] Among them, in some embodiments, the preset gesture category with the highest probability can be determined as the image gesture recognition result corresponding to the target object, and the preset gesture category with a probability greater than a preset value can be determined as the image gesture recognition result corresponding to the target object. The preset value can be set according to actual conditions.

[0220] 203. The terminal divides and processes the millimeter wave gesture detection data to obtain at least one gesture detection sub-data.

[0221] There are many ways to divide the millimeter wave gesture detection data, which is not limited in this embodiment. Specifically, the millimeter wave gesture detection data is gesture detection data within a certain time period. In some embodiments, the millimeter wave gesture detection data can be divided into gesture detection sub-data of equal duration. For example, the millimeter wave gesture detection data is divided into blocks based on a preset division duration (such as 5ms, which can be adjusted) to obtain each gesture detection sub-data.

[0222] 204. The terminal performs gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtains the millimeter wave gesture recognition result corresponding to each gesture detection sub-data.

[0223] Optionally, in this embodiment, the step of “performing gesture recognition according to millimeter wave feature information corresponding to each gesture detection sub-data to obtain a millimeter wave gesture recognition result corresponding to each gesture detection sub-data” may include:

[0224] For each gesture detection sub-data, extracting spectrum features of the gesture detection sub-data in at least one dimension to obtain millimeter wave feature information of the gesture detection sub-data in at least one dimension;

[0225] Gesture recognition is performed according to the millimeter wave feature information of the gesture detection sub-data in at least one dimension to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0226] The spectral feature extraction in at least one dimension may include zero-crossing rate feature extraction, spectral flatness feature extraction, and spectral centroid feature extraction, etc., which are not limited in this embodiment. The millimeter wave feature information in at least one dimension obtained by extracting the spectral feature in at least one dimension may include the zero-crossing rate feature, spectral flatness feature, and spectral centroid feature of the gesture detection sub-data.

[0227] Specifically, binary encoding can be used to extract zero-crossing rate features, where the zero-crossing rate (ZCR) refers to the ratio of a signal's sign change, such as a signal changing from a positive number to a negative number or vice versa. For spectrum flatness feature extraction and spectrum centroid feature extraction, the fft size (signal window width of Fourier transform) can be used for extraction, and then the extracted spectrum flatness feature and spectrum centroid feature can be standardized.

[0228] In this embodiment, spectrum feature extraction is adopted to make the extracted millimeter wave feature information have a better anti-noise effect.

[0229] The step of “performing gesture recognition according to the millimeter wave feature information of the gesture detection sub-data in at least one dimension to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data” may include:

[0230] Fusing the millimeter wave feature information of the gesture detection sub-data in each dimension to obtain target millimeter wave feature information corresponding to the gesture detection sub-data;

[0231] Gesture recognition is performed based on the target millimeter wave characteristic information to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0232] There are many ways to fuse the millimeter wave feature information in various dimensions, which are not limited in this embodiment. For example, the fusion method can be weighted processing or splicing processing.

[0233] Among them, gesture recognition is performed based on the target millimeter wave feature information, specifically, the probability that the corresponding gesture detection sub-data belongs to each preset gesture category is predicted based on the target millimeter wave feature information; based on the probability, the millimeter wave gesture recognition result corresponding to the gesture detection sub-data is determined. Among them, the probability that the gesture detection sub-data belongs to each preset gesture category can be predicted by a classifier. In some embodiments, the preset gesture category with the highest probability can be determined as the millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0234] Optionally, in this embodiment, the gesture detection sub-data are gesture detection sub-data sorted according to detection time;

[0235] The step of “performing gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data” may include:

[0236] For each gesture detection sub-data, based on the previous and next gesture detection sub-data of the gesture detection sub-data, extract the time series features of the gesture detection sub-data to obtain the millimeter wave feature information corresponding to the gesture detection sub-data;

[0237] Gesture recognition is performed according to the millimeter wave feature information corresponding to the gesture detection sub-data to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0238] Among them, the memory network model can be used to extract the time series features of the gesture detection sub-data based on the previous and next gesture detection sub-data of the gesture detection sub-data, so as to obtain the millimeter wave feature information corresponding to the gesture detection sub-data.

[0239] 205. The terminal determines a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data.

[0240] In this embodiment, based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data, voting can be performed on each preset gesture category to obtain a target gesture recognition result corresponding to the target object. Specifically, the preset gesture category with the most votes can be determined as the target gesture recognition result of the target object.

[0241] For example, the image gesture recognition result is that the gesture of the target object is sliding to the left, and the millimeter-wave gesture recognition results corresponding to the three gesture detection sub-data obtained are sliding to the right, sliding to the left, and shaking up and down, respectively. Then, voting is performed on each preset gesture category, and the preset gesture category of sliding to the right can get 1 vote, the preset gesture category of sliding to the left can get 2 votes, and the preset gesture category of shaking up and down can get 1 vote. Therefore, the target gesture recognition result of the target object is sliding to the left.

[0242] The gesture recognition method provided in the present application can read the hand motion data of the target object based on the millimeter wave sensor and the image sensor (specifically, it can include the gesture image to be recognized and the millimeter wave gesture detection data), and perform gesture recognition on the acquired hand motion data, and interpret it as the corresponding gesture. Since gesture recognition is performed in combination with the gesture image to be recognized and the millimeter wave gesture detection data, the accuracy of gesture recognition can be improved. The power consumption of the millimeter wave sensor and the image sensor is relatively low, and the accuracy of gesture recognition can also be improved on low-power devices.

[0243] As can be seen from the above, this embodiment can collect gesture images to be identified and millimeter wave gesture detection data for the target object through the terminal; perform gesture recognition on the gesture images to be identified to obtain the image gesture recognition result corresponding to the target object; divide and process the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data; determine the target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data. This application can combine the gesture images to be identified and the millimeter wave gesture detection data to determine the gesture recognition result of the target object, which can improve the accuracy of gesture recognition.

[0244] In order to better implement the above method, the present application embodiment also provides a gesture recognition device, such as Figure 3 As shown, the gesture recognition device may include a collection unit 301, a first recognition unit 302, a division unit 303, a second recognition unit 304 and a determination unit 305, as follows:

[0245] (1) Collection unit 301;

[0246] The acquisition unit is used to acquire gesture images to be recognized and millimeter wave gesture detection data of the target object.

[0247] (2) a first identification unit 302;

[0248] The first recognition unit is used to perform gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object.

[0249] Optionally, in some embodiments of the present application, the first recognition unit may include an image division subunit, a first extraction subunit, a determination subunit and a first recognition subunit, as follows:

[0250] The image division subunit is used to perform image division processing on the gesture image to be recognized to obtain at least one gesture image block to be recognized;

[0251] The first extraction subunit is used to extract features from each gesture image block to be identified, and obtain sub-feature information corresponding to each gesture image block to be identified;

[0252] A determination subunit, configured to determine image feature information corresponding to the gesture image to be identified based on sub-feature information corresponding to each gesture image block to be identified;

[0253] The first recognition subunit is used to perform gesture recognition according to the image feature information to obtain an image gesture recognition result corresponding to the target object.

[0254] Optionally, in some embodiments of the present application, the first extraction subunit can be specifically used to perform convolution processing on each gesture image block to be identified based on a preset gradient convolution factor to obtain a first direction gradient value and a second direction gradient value corresponding to the gesture image block to be identified; and calculate sub-feature information corresponding to the gesture image block to be identified based on the first direction gradient value and the second direction gradient value.

[0255] Optionally, in some embodiments of the present application, the determination sub-unit can be specifically used to perform attention processing on the sub-feature information corresponding to each gesture image block to be identified, and obtain the attention weight of the sub-feature information corresponding to each gesture image block to be identified; according to the attention weight, the sub-feature information corresponding to each gesture image block to be identified is fused to obtain the image feature information corresponding to the gesture image to be identified.

[0256] (3) a division unit 303;

[0257] The division unit is used to divide the millimeter wave gesture detection data to obtain at least one gesture detection sub-data.

[0258] (4) a second identification unit 304;

[0259] The second recognition unit is used to perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data.

[0260] Optionally, in some embodiments of the present application, the second recognition unit may include a second extraction subunit and a second recognition subunit, as follows:

[0261] The second extraction subunit is used to extract spectrum features of each gesture detection sub-data in at least one dimension to obtain millimeter wave feature information of the gesture detection sub-data in at least one dimension;

[0262] The second recognition subunit is used to perform gesture recognition according to the millimeter wave feature information of the gesture detection sub-data in at least one dimension, and obtain the millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0263] Optionally, in some embodiments of the present application, the gesture detection sub-data are gesture detection sub-data sorted according to detection time;

[0264] The second recognition unit may include a third extraction subunit and a third recognition subunit, as follows:

[0265] The third extraction sub-unit is used to extract the time series features of each gesture detection sub-data based on the previous and next gesture detection sub-data of the gesture detection sub-data, so as to obtain the millimeter wave feature information corresponding to the gesture detection sub-data;

[0266] The third recognition subunit is used to perform gesture recognition according to the millimeter wave feature information corresponding to the gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

[0267] (5) determining unit 305;

[0268] The determination unit is used to determine a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data.

[0269] Optionally, in some embodiments of the present application, the first recognition unit may be specifically configured to perform gesture recognition on the gesture image to be recognized through a gesture recognition model to obtain an image gesture recognition result corresponding to the target object.

[0270] Optionally, in some embodiments of the present application, the gesture recognition device may further include a training unit, which is used to train a gesture recognition model; the training unit may be specifically used to obtain training data, wherein the training data includes a sample image of a gesture to be recognized of a sample object, millimeter-wave gesture sample detection data, and an expected gesture recognition result corresponding to the sample object; through a preset gesture recognition model, gesture recognition is performed on the sample image of the gesture to be recognized to obtain an actual image gesture recognition result corresponding to the sample object; the millimeter-wave gesture sample detection data is divided and processed to obtain at least one gesture sample detection sub-data; gesture recognition is performed according to millimeter-wave feature information corresponding to each gesture sample detection sub-data to obtain an actual millimeter-wave gesture recognition result corresponding to each gesture sample detection sub-data; based on the actual image gesture recognition result, the actual millimeter-wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain a gesture recognition model.

[0271] Optionally, in some embodiments of the present application, the step of “adjusting parameters of a preset gesture recognition model based on the actual image gesture recognition result, the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result to obtain a gesture recognition model” may include:

[0272] Based on the actual image gesture recognition result and the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, voting is performed on the gesture recognition result of the sample object to obtain a target actual gesture recognition result of the sample object;

[0273] Based on the target actual gesture recognition result and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain a gesture recognition model.

[0274] As can be seen from the above, in this embodiment, the collection unit 301 can collect gesture images to be identified and millimeter wave gesture detection data for the target object; the first recognition unit 302 performs gesture recognition on the gesture images to be identified to obtain the image gesture recognition result corresponding to the target object; the division unit 303 divides the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; the second recognition unit 304 performs gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data; the determination unit 305 determines the target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data. The present application can determine the gesture recognition result of the target object in combination with the gesture images to be identified and the millimeter wave gesture detection data, which can improve the accuracy of gesture recognition.

[0275] The present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, and the electronic device may be a terminal or a server, etc. Specifically:

[0276] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0277] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0278] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0279] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0280] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0281] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0282] Collect gesture images to be recognized and millimeter-wave gesture detection data for a target object; perform gesture recognition on the gesture images to be recognized to obtain an image gesture recognition result corresponding to the target object; divide and process the millimeter-wave gesture detection data to obtain at least one gesture detection sub-data; perform gesture recognition based on millimeter-wave feature information corresponding to each gesture detection sub-data to obtain a millimeter-wave gesture recognition result corresponding to each gesture detection sub-data; and determine a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter-wave gesture recognition results corresponding to each gesture detection sub-data.

[0283] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0284] As can be seen from the above, this embodiment can collect gesture images to be identified and millimeter wave gesture detection data for the target object; perform gesture recognition on the gesture images to be identified to obtain the image gesture recognition result corresponding to the target object; divide and process the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; perform gesture recognition based on the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data; determine the target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition result corresponding to each gesture detection sub-data. This application can combine the gesture images to be identified and the millimeter wave gesture detection data to determine the gesture recognition result of the target object, which can improve the accuracy of gesture recognition.

[0285] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0286] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the gesture recognition methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0287] Collect gesture images to be recognized and millimeter-wave gesture detection data for a target object; perform gesture recognition on the gesture images to be recognized to obtain an image gesture recognition result corresponding to the target object; divide and process the millimeter-wave gesture detection data to obtain at least one gesture detection sub-data; perform gesture recognition based on millimeter-wave feature information corresponding to each gesture detection sub-data to obtain a millimeter-wave gesture recognition result corresponding to each gesture detection sub-data; and determine a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter-wave gesture recognition results corresponding to each gesture detection sub-data.

[0288] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0289] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0290] Since the instructions stored in the computer-readable storage medium can execute the steps in any gesture recognition method provided in the embodiments of the present application, the beneficial effects that can be achieved by any gesture recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0291] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations of the above-mentioned gesture recognition aspects.

[0292] The above is a detailed introduction to a gesture recognition method and related devices provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A gesture recognition method, characterized in that: include: Collect gesture images to be recognized and millimeter wave gesture detection data of the target object; Performing gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object; dividing the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; Perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data; Determining a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data; The performing gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data includes: For each gesture detection sub-data, extracting spectrum features of the gesture detection sub-data in multiple dimensions to obtain millimeter wave feature information of the gesture detection sub-data in multiple dimensions; Fusing the millimeter wave feature information of the gesture detection sub-data in each dimension to obtain target millimeter wave feature information corresponding to the gesture detection sub-data; Gesture recognition is performed based on the target millimeter wave characteristic information to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

2. The method according to claim 1, characterized in that The performing gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object includes: Performing image segmentation processing on the gesture image to be recognized to obtain at least one gesture image block to be recognized; Extract features from each gesture image block to be identified, and obtain sub-feature information corresponding to each gesture image block to be identified; Determining image feature information corresponding to the gesture image to be identified based on sub-feature information corresponding to each gesture image block to be identified; Gesture recognition is performed according to the image feature information to obtain an image gesture recognition result corresponding to the target object.

3. The method according to claim 2, characterized in that The feature extraction of each gesture image block to be identified to obtain sub-feature information corresponding to each gesture image block to be identified includes: For each gesture image block to be identified, based on a preset gradient convolution factor, convolution processing is performed on the gesture image block to be identified to obtain a first direction gradient value and a second direction gradient value corresponding to the gesture image block to be identified; The sub-feature information corresponding to the gesture image block to be recognized is calculated according to the first directional gradient value and the second directional gradient value.

4. The method according to claim 2, characterized in that: The determining, according to the sub-feature information corresponding to each to-be-recognized gesture image block, the image feature information corresponding to the to-be-recognized gesture image includes: Performing attention processing on the sub-feature information corresponding to each gesture image block to be identified, and obtaining the attention weight of the sub-feature information corresponding to each gesture image block to be identified; According to the attention weight, the sub-feature information corresponding to each to-be-recognized gesture image block is fused to obtain the image feature information corresponding to the to-be-recognized gesture image.

5. The method according to claim 1, characterized in that The gesture detection sub-data are gesture detection sub-data sorted according to detection time; The performing gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data to obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data includes: For each gesture detection sub-data, based on the previous and next gesture detection sub-data of the gesture detection sub-data, extract the time series features of the gesture detection sub-data to obtain the millimeter wave feature information corresponding to the gesture detection sub-data; Gesture recognition is performed according to the millimeter wave feature information corresponding to the gesture detection sub-data to obtain a millimeter wave gesture recognition result corresponding to the gesture detection sub-data.

6. The method according to claim 1, characterized in that The performing gesture recognition on the gesture image to be recognized to obtain an image gesture recognition result corresponding to the target object includes: The gesture recognition model is used to perform gesture recognition on the gesture image to be recognized, and an image gesture recognition result corresponding to the target object is obtained.

7. The method according to claim 6, characterized in that Before performing gesture recognition on the gesture image to be recognized by using the gesture recognition model to obtain the image gesture recognition result corresponding to the target object, the method further includes: Acquire training data, wherein the training data includes a sample image of a gesture to be recognized of a sample object, millimeter wave gesture sample detection data, and an expected gesture recognition result corresponding to the sample object; By using a preset gesture recognition model, gesture recognition is performed on the gesture sample image to be recognized, so as to obtain an actual image gesture recognition result corresponding to the sample object; The millimeter wave gesture sample detection data is divided and processed to obtain at least one gesture sample detection sub-data; gesture recognition is performed according to the millimeter wave feature information corresponding to each gesture sample detection sub-data to obtain an actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data; Based on the actual image gesture recognition result, the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain the gesture recognition model.

8. The method according to claim 7, characterized in that The step of adjusting the parameters of a preset gesture recognition model based on the actual image gesture recognition result, the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, and the expected gesture recognition result to obtain the gesture recognition model includes: Based on the actual image gesture recognition result and the actual millimeter wave gesture recognition result corresponding to each gesture sample detection sub-data, voting is performed on the gesture recognition result of the sample object to obtain a target actual gesture recognition result of the sample object; Based on the target actual gesture recognition result and the expected gesture recognition result, the parameters of the preset gesture recognition model are adjusted to obtain a gesture recognition model.

9. A gesture recognition device, characterized in that: include: A collection unit, used for collecting gesture images to be recognized and millimeter wave gesture detection data of a target object; A first recognition unit, configured to perform gesture recognition on the gesture image to be recognized, and obtain an image gesture recognition result corresponding to the target object; A division unit, configured to divide the millimeter wave gesture detection data to obtain at least one gesture detection sub-data; The second recognition unit is used to perform gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data; wherein, the gesture recognition according to the millimeter wave feature information corresponding to each gesture detection sub-data, and obtain the millimeter wave gesture recognition result corresponding to each gesture detection sub-data, includes: for each gesture detection sub-data, extracting the spectrum features of the gesture detection sub-data in multiple dimensions, and obtaining the millimeter wave feature information of the gesture detection sub-data in multiple dimensions; fusing the millimeter wave feature information of the gesture detection sub-data in each dimension, and obtaining the target millimeter wave feature information corresponding to the gesture detection sub-data; performing gesture recognition based on the target millimeter wave feature information, and obtaining the millimeter wave gesture recognition result corresponding to the gesture detection sub-data; The determination unit is used to determine a target gesture recognition result corresponding to the target object based on the image gesture recognition result and the millimeter wave gesture recognition results corresponding to each gesture detection sub-data.

10. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to perform the operations in the gesture recognition method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the gesture recognition method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the gesture recognition method according to any one of claims 1 to 8 are implemented.

Images