Gesture recognition method and related device

Abstract

The invention discloses a gesture recognition method and a related device, and relates to the field of artificial intelligence, a gesture recognition model comprises a positioning branch, a matching branch and a classification branch, and positioning information of multiple components of at least one target object in an image is recognized through the positioning branch; the method comprises the following steps: obtaining a matching relationship among a plurality of components belonging to the same target object through a matching branch, and if two hands can accurately distinguish the matching relationship between a finger part and a palm part in each hand in an image, because the matching branch has accurately identified the matching relationship among the plurality of components belonging to the same target object, the matching relationship between the plurality of components belonging to the same target object can be accurately distinguished; on the basis, the matching relation among the multiple components of the at least one target object and the positioning information of the components are input into the classification branch, even complex gestures needing two hands can be accurately recognized, and the recognition accuracy of the complex gestures is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a gesture recognition method and related devices. Background Art

[0002] With the rapid development of artificial intelligence technology, gesture recognition based on artificial intelligence technology has gradually become one of the main ways of human-computer interaction. Gesture recognition analyzes the user's hand posture to identify the user's intention and respond to the user's intention. It is widely used in virtual reality, smart home and other fields.

[0003] Currently, gesture recognition technology mainly recognizes gestures based on one hand, but the recognition accuracy of complex gestures that require two hands is low. Summary of the invention

[0004] In view of the above problems, the present application provides a gesture recognition method and related devices, which effectively improve the accuracy of complex gesture recognition. The specific scheme is as follows:

[0005] The first aspect of the present application provides a gesture recognition method, comprising:

[0006] Obtain an image to be recognized;

[0007] Inputting the image into a positioning branch in a gesture recognition model to identify positioning information of multiple components of at least one target object in the image, wherein the multiple components of the target object include at least a finger portion and a palm portion;

[0008] Inputting the positioning information of the multiple components of at least one of the target objects into the matching branch in the gesture recognition model to obtain a matching relationship between the multiple components belonging to the same target object;

[0009] The matching relationship between the multiple components of at least one of the target objects and the positioning information of the multiple components are input into the classification branch in the gesture recognition model to determine the target gesture type corresponding to the image.

[0010] In a possible implementation, the positioning branch includes a finger positioning branch and a palm orientation positioning branch, and the gesture recognition method further includes:

[0011] Acquire a sample set of example gestures, each sample in the sample set of example gestures includes at least one of a directory type of finger location directory, a palm orientation location target, a directory of matching relationships between finger location and palm orientation location, and a directory of gesture identifications of gesture types corresponding to the finger location and the palm orientation location;

[0012] The corresponding branches in the model to be trained are trained separately using the directory types included in each sample in the example gesture sample set to obtain the gesture recognition model, wherein the finger location directory corresponds to the finger location branch in the model to be trained, the palm orientation location directory corresponds to the palm orientation location branch in the model to be trained, the matching relationship directory between the finger location and the palm orientation location corresponds to the matching branch in the model to be trained, and the gesture identification directory of the gesture types corresponding to the finger location and the palm orientation location corresponds to the classification branch in the model to be trained.

[0013] In a possible implementation, the positioning branch includes a finger positioning branch and a palm orientation positioning branch, and the step of inputting the image into the positioning branch of the gesture recognition model to identify positioning information of multiple components of at least one target object in the image includes:

[0014] Inputting the image into the finger positioning branch, identifying positioning information of at least one finger part in the image;

[0015] The image is input into the palm orientation location branch to identify the palm orientation location of at least one of the palm parts in the image.

[0016] In a possible implementation, inputting the positioning information of the multiple components of at least one of the target objects into the matching branch in the gesture recognition model to obtain the matching relationship between the multiple components belonging to the same target object includes:

[0017] Inputting the positioning information of at least one of the finger parts and the palm orientation positioning of at least one of the palm parts in the image into the matching branch;

[0018] In the matching branch, for each finger part, a reference palm orientation location corresponding to the positioning information of the finger part is determined, and a target palm orientation location corresponding to the reference palm orientation is determined based on the reference palm orientation location and the palm orientation location of at least one of the palm parts, and the target palm orientation location is determined as the palm orientation location that matches the positioning information of the finger part in the same target object.

[0019] In a possible implementation, determining a reference palm orientation location corresponding to the location information of the finger portion includes:

[0020] The second AI vector value is calculated based on the first AI vector value of the finger part and the preset vector, and the second AI vector value is used to represent the reference palm orientation positioning. The first AI vector value of the finger part is the AI ​​vector value corresponding to the center positioning of the finger identification frame, and the preset vector is a vector pointing to the palm orientation positioning of the finger in the same hand obtained by model training.

[0021] In a possible implementation, determining a target palm orientation location corresponding to the reference palm orientation according to the reference palm orientation location and a palm orientation location of at least one of the palm parts includes:

[0022] The palm orientation location of the palm portion having the smallest distance from the reference palm orientation location is determined as the target palm orientation location corresponding to the reference palm orientation location.

[0023] In a possible implementation, inputting the matching relationship between the multiple components of at least one of the target objects and the positioning information of the multiple components into the classification branch in the gesture recognition model to determine the target gesture type corresponding to the image includes:

[0024] Inputting the palm orientation location matched with the location information of the finger part in the same target object into the classification branch;

[0025] In the classification branch, a feature vector corresponding to each preset gesture type is generated according to the palm orientation positioning that matches the positioning information of the finger part in the same target object, and the feature vector corresponding to each preset gesture type is dimensionlessly simplified to obtain a dimensionless simplified value corresponding to each preset gesture type, and the preset gesture type corresponding to the maximum dimensionless simplified value is determined as the target gesture type.

[0026] A second aspect of the present application provides a gesture recognition device, comprising:

[0027] An acquisition unit, used for acquiring an image to be recognized;

[0028] a positioning unit, configured to input the image into a positioning branch in a gesture recognition model, and identify positioning information of multiple components of at least one target object in the image, wherein the multiple components of the target object at least include a finger portion and a palm portion;

[0029] a matching unit, configured to input the positioning information of the multiple components of at least one of the target objects into a matching branch in the gesture recognition model to obtain a matching relationship between the multiple components belonging to the same target object;

[0030] The classification unit is used to input the matching relationship between the multiple components of at least one of the target objects and the positioning information of the multiple components into the classification branch in the gesture recognition model to determine the target gesture type corresponding to the image.

[0031] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is used to execute the computer program so that the electronic device can implement the gesture recognition method of the first aspect or any implementation manner of the first aspect.

[0034] A fourth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the gesture recognition method of the first aspect or any implementation of the first aspect.

[0035] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the gesture recognition method of the first aspect or any implementation of the first aspect.

[0036] By means of the above technical scheme, the present application provides a gesture recognition method and related devices, wherein the gesture recognition model includes a positioning branch, a matching branch and a classification branch. The positioning branch is used to identify the positioning information of multiple components of at least one target object in the image, and the matching branch is used to obtain the matching relationship between the multiple components belonging to the same target object. If the image includes two hands, the matching relationship between the finger part and the palm part of each hand can be accurately distinguished. Since the matching branch has accurately identified the matching relationship between the multiple components belonging to the same target object, on this basis, the matching relationship between the multiple components of at least one target object and the positioning information of the components are input into the classification branch. Even complex gestures requiring two hands can be accurately recognized, thereby effectively improving the accuracy of complex gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0038] Figure 1 A system architecture diagram provided for an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a terminal structure provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of a server structure provided in an embodiment of the present application;

[0041] Figure 4 A flowchart of a gesture recognition method provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the structure of a gesture recognition model provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of a three-dimensional coordinate system provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of preset gesture types provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of a gesture recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0047] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0048] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0049] See also Figure 1 , Figure 1A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the gesture recognition method provided in the embodiment of the present application for one or more terminals.

[0050] Among them, an application can be installed on the terminal 100, and the above application can provide an interface. The terminal 100 can obtain the image to be identified and send the image to be identified to the server 200. The server 200 can perform gesture recognition based on the received image, obtain the target gesture type, and return the target gesture type to the terminal 100.

[0051] It should be understood that in some optional implementations, the terminal 100 can also complete the actions of acquiring the image to be recognized and performing gesture recognition on the image by itself without the cooperation of the server, which is not limited in the embodiments of the present application.

[0052] Next describe Figure 1 The product form of the mid-terminal 100;

[0053] The terminal 100 in the embodiment of the present application may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, etc., and the embodiment of the present application does not impose any limitation on this.

[0054] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0055] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150, an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0056] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal 100. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0057] Among them, the input device 132 can receive input data and the like.

[0058] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display interfaces, processing results, etc.

[0059] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0060] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0061] The memory 120 may be used to store software codes related to the gesture recognition method, the processor 170 may execute the steps of the gesture recognition method, and may also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0062] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0063] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0064] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0065] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0066] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0067] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0068] Next describe Figure 1 The product form of the server 200;

[0069] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0070] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0071] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0072] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0073] The memory 204 may be used to store software codes related to the gesture recognition method, and the processor 202 may execute the steps of the gesture recognition method of the chip, and may also schedule other units to implement corresponding functions.

[0074] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0075] The embodiment of the present application provides a gesture recognition method. The gesture recognition method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0076] Reference Figure 4 , Figure 4 A flowchart of a gesture recognition method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, a gesture recognition method provided in an embodiment of the present application may include steps 401 to 404, and these steps are described in detail below.

[0077] 401: Obtain the image to be identified;

[0078] Specifically, an image acquisition device (such as a depth camera, a depth camera) is used to obtain a depth image to be recognized, and the image to be recognized is used to recognize a user gesture.

[0079] In order to improve the image quality, the brightness parameter of the image acquisition device can be adjusted according to the ambient light before acquiring the image to be recognized.

[0080] 402: Inputting the image into a positioning branch in a gesture recognition model to identify positioning information of multiple components of at least one target object in the image, where the multiple components of the target object at least include a finger part and a palm part;

[0081] like Figure 5 The structural diagram of the gesture recognition model shown in FIG. 1 , the gesture recognition model includes a positioning branch, a matching branch and a classification branch.

[0082] The localization branch may be a target detection model, which is used to perform target detection on an image and identify multiple components of at least one target object in the image.

[0083] Based on different gesture recognition scenarios, the target object can be different.

[0084] In some gesture recognition scenarios, the target object is a hand, and the components of the target object include a finger part and a palm part.

[0085] In some gesture recognition scenarios, the target object is an arm, and the components of the target object include a finger part, a palm part, and an upper arm part.

[0086] The above gesture recognition scenarios are only examples, and the present application is not limited thereto.

[0087] Taking the target object including finger parts and palm parts as an example, in order to accurately identify the positioning information of the finger parts and the palm parts, the positioning branch includes a finger positioning branch and a palm orientation positioning branch. The image is input into the finger positioning branch to identify the positioning information of at least one finger part in the image, and the image is input into the palm orientation positioning branch to identify the palm orientation positioning of at least one palm part in the image.

[0088] 403: Inputting the positioning information of multiple components of at least one target object into a matching branch in a gesture recognition model to obtain a matching relationship between multiple components belonging to the same target object;

[0089] Taking the example that the target object includes finger parts and palm parts, the matching branch is used to identify the matching relationship between the finger parts and palm parts belonging to the same hand. When there is more than one hand in the image, the matching relationship between the finger parts and palm parts of each hand can be accurately identified.

[0090] The matching branch can be implemented based on a deep learning model.

[0091] 404: Input the matching relationship between the multiple components of the at least one target object and the positioning information of the multiple components into the classification branch in the gesture recognition model to determine the target gesture type corresponding to the image.

[0092] The present embodiment discloses a gesture recognition method, in which a gesture recognition model includes a positioning branch, a matching branch and a classification branch. The positioning branch is used to identify positioning information of multiple components of at least one target object in an image, and the matching branch is used to obtain a matching relationship between multiple components belonging to the same target object. If the image includes two hands, the matching relationship between the finger part and the palm part of each hand can be accurately distinguished. Since the matching branch has accurately identified the matching relationship between multiple components belonging to the same target object, on this basis, the matching relationship between multiple components of at least one target object and the positioning information of the components are input into the classification branch. Even complex gestures requiring two hands can be accurately recognized, thereby effectively improving the accuracy of complex gesture recognition.

[0093] It should be noted that the gesture recognition model in the above embodiment needs to be trained in advance. Considering the large workload of labeling samples, existing example gestures can be used as samples. The number of gesture identification directories that are labeled with finger location directories, palm orientation location targets, matching relationship directories between finger location and palm orientation location, and gesture types corresponding to finger location and palm orientation location in the example gesture samples is relatively small, and it takes a lot of time to fully label the directory types of the example gesture samples. In order to solve this technical problem, this embodiment trains each branch in the training model separately, that is, the finger location branch is trained using the example gesture samples labeled with the finger location directory, the palm orientation location branch is trained using the example gesture samples labeled with the palm orientation location directory, the matching branch is trained using the example gesture samples labeled with the matching relationship directory between finger location and palm orientation location, and the classification branch is trained using the gesture identification directory labeled with the gesture types corresponding to finger location and palm orientation location. Taking the target object including the finger part and the palm part as an example, the training method of the gesture recognition model includes the following steps A1-A2:

[0094] A1: Get a sample set of example gestures;

[0095] Each sample in the sample gesture sample set includes at least one of a finger location directory, a palm orientation location target, a matching relationship directory between finger location and palm orientation location, and a gesture identification directory of a gesture type corresponding to the finger location and the palm orientation location;

[0096] A2: Use the directory type included in each sample in the example gesture sample set to train the corresponding branches in the training model to obtain the gesture recognition model.

[0097] Among them, the finger positioning directory corresponds to the finger positioning branch in the model to be trained, the palm orientation positioning directory corresponds to the palm orientation positioning branch in the model to be trained, the matching relationship directory between finger positioning and palm orientation positioning corresponds to the matching branch in the model to be trained, and the gesture identification directory of the gesture types corresponding to the finger positioning and palm orientation positioning corresponds to the classification branch in the model to be trained.

[0098] Furthermore, as the gesture recognition model is put into use, for images recognized by the gesture recognition model, the positioning information of the finger parts, the palm orientation positioning, the palm orientation positioning matched with the positioning information of the finger parts, and the target gesture type obtained in the recognition process can also be collected, and the images can be annotated and used as training samples to update the example gesture sample set, thereby periodically training the gesture recognition model to improve the analysis performance and generalization performance of the gesture recognition model.

[0099] The following is a detailed description of each step of the gesture recognition method provided in the above embodiment.

[0100] In a possible implementation, the target object includes a finger part and a palm part, and the specific implementation of step 403 in the above embodiment includes the following steps 4031-4032:

[0101] 4031: Inputting the positioning information of at least one finger part and the palm orientation positioning information of at least one palm part in the image into a matching branch;

[0102] 4032: In the matching branch, for each finger part, determine a reference palm orientation location corresponding to the positioning information of the finger part, determine a target palm orientation location corresponding to the reference palm orientation based on the reference palm orientation location and the palm orientation location of at least one palm part, and determine the target palm orientation location as the palm orientation location that matches the positioning information of the finger part in the same target object.

[0103] like Figure 6 The image shown constructs a three-dimensional coordinate system with the thumb tip as the origin. The finger recognition frame obtained through the finger positioning branch includes three-dimensional coordinates, and the palm orientation recognition frame obtained through the palm orientation positioning branch also includes three-dimensional coordinates.

[0104] To match the positioning information of the finger part and the palm orientation positioning in the same target object, what needs to be matched is the center positioning of the finger recognition frame and the center positioning of the palm orientation recognition frame.

[0105] The AI ​​vector value corresponding to the center positioning of the finger recognition frame is used as the first AI vector value of the finger part, and the AI ​​vector value corresponding to the center positioning of the palm orientation recognition frame is used as the palm orientation positioning.

[0106] In a possible implementation, a second AI vector value is calculated based on the first AI vector value of the finger part and a preset vector, and the second AI vector value is used to represent the reference palm orientation location, wherein the preset vector is a vector pointing to the palm orientation location of the finger location in the same hand obtained in the model training phase. Then, the palm orientation location with the smallest distance from the reference palm orientation location in the palm orientation location of at least one palm part is determined as the target palm orientation location corresponding to the reference palm orientation location, and finally, the target palm orientation location is determined as the palm orientation location that matches the positioning information of the above-mentioned finger part in the same target object, so as to achieve accurate matching of fingers and palms.

[0107] In a possible implementation, the specific implementation of step 404 in the gesture recognition method provided in the above embodiment includes the following steps 4041-4042:

[0108] 4041: a palm orientation positioning input classification branch that matches the positioning information of the finger parts in the same target object;

[0109] 4042: In the classification branch, a feature vector corresponding to each preset gesture type is generated according to the palm orientation positioning matched with the positioning information of the finger part in the same target object, and the feature vector corresponding to each preset gesture type is dimensionlessly simplified to obtain the dimensionless simplified value corresponding to each preset gesture type, and the preset gesture type corresponding to the maximum dimensionless simplified value is determined as the target gesture type.

[0110] The preset gesture types include Figure 7 Taking the a1 gesture type and a2 gesture type shown as examples, in the classification branch, a feature vector corresponding to the a1 gesture type and a feature vector corresponding to the a2 gesture type are generated according to the palm orientation positioning matched with the positioning information of the finger part in the same target object. According to these two feature vectors, the target gesture type that the target object is most likely to perform can be determined. Specifically, the two feature vectors are dimensionally simplified, for example, the two feature vectors are dimensionally simplified using a softmax function, so that the scales of different features no longer affect the final classification results. The dimensionless simplified value corresponding to each gesture type represents the probability of the target object performing the gesture type, and therefore the preset gesture type corresponding to the maximum dimensionless simplified value is determined as the target gesture type.

[0111] A gesture recognition method provided by an embodiment of the present application is introduced above, and a device for executing the gesture recognition method is introduced below.

[0112] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a gesture recognition device provided in an embodiment of the present application. Figure 8 As shown, the gesture recognition device includes:

[0113] An acquisition unit 801 is used to acquire an image to be recognized;

[0114] A positioning unit 802, configured to input the image into a positioning branch in a gesture recognition model, and identify positioning information of multiple components of at least one target object in the image, wherein the multiple components of the target object at least include a finger part and a palm part;

[0115] A matching unit 803, configured to input the positioning information of the multiple components of at least one of the target objects into a matching branch in the gesture recognition model to obtain a matching relationship between the multiple components belonging to the same target object;

[0116] The classification unit 804 is used to input the matching relationship between the multiple components of at least one of the target objects and the positioning information of the multiple components into the classification branch in the gesture recognition model to determine the target gesture type corresponding to the image.

[0117] In a possible implementation, the gesture recognition device further includes:

[0118] A training unit is used to obtain a set of example gesture samples, each sample in the example gesture sample set includes at least one of a finger location directory, a palm orientation location target, a matching relationship directory between finger location and palm orientation location, and a gesture identification directory of gesture types corresponding to the finger location and palm orientation location; the corresponding branches in the to-be-trained model are trained respectively using the directory types included in each sample in the example gesture sample set to obtain the gesture recognition model, wherein the finger location directory corresponds to the finger location branch in the to-be-trained model, the palm orientation location directory corresponds to the palm orientation location branch in the to-be-trained model, the matching relationship directory between finger location and palm orientation location corresponds to the matching branch in the to-be-trained model, and the gesture identification directory of the gesture types corresponding to the finger location and palm orientation location corresponds to the classification branch in the to-be-trained model.

[0119] In a possible implementation, the positioning unit 802 includes:

[0120] A finger positioning subunit, configured to input the image into the finger positioning branch and identify positioning information of at least one finger part in the image;

[0121] The palm positioning subunit is used to input the image into the palm orientation positioning branch, and identify the palm orientation positioning of at least one of the palm parts in the image.

[0122] In a possible implementation, the matching unit 803 includes:

[0123] An input subunit, configured to input the positioning information of at least one of the finger parts and the palm orientation positioning of at least one of the palm parts in the image into the matching branch;

[0124] A first matching subunit, configured to determine, for each finger part in the matching branch, a reference palm orientation location corresponding to the location information of the finger part;

[0125] A second matching subunit is used to determine a target palm orientation location corresponding to the reference palm orientation based on the reference palm orientation location and the palm orientation location of at least one of the palm parts, and determine the target palm orientation location as a palm orientation location that matches the positioning information of the finger part in the same target object.

[0126] In a possible implementation, the first matching subunit is specifically used to calculate a second AI vector value based on the first AI vector value of the finger part and a preset vector, and use the second AI vector value to represent the reference palm orientation position, the first AI vector value of the finger part is the AI ​​vector value corresponding to the center positioning of the finger identification frame, and the preset vector is a vector pointing to the palm orientation positioning of the finger position in the same hand obtained by model training.

[0127] In a possible implementation, the second matching subunit is specifically configured to determine the palm orientation location of the palm portion having the smallest distance from the reference palm orientation location as the target palm orientation location corresponding to the reference palm orientation location.

[0128] In a possible implementation, the classification unit 804 is specifically used to input the palm orientation location that matches the positioning information of the finger part in the same target object into the classification branch; in the classification branch, a feature vector corresponding to each preset gesture type is generated based on the palm orientation location that matches the positioning information of the finger part in the same target object, and the feature vector corresponding to each preset gesture type is dimensionlessly simplified to obtain the dimensionless simplified value corresponding to each preset gesture type, and the preset gesture type corresponding to the maximum dimensionless simplified value is determined as the target gesture type.

[0129] The present embodiment discloses a gesture recognition device, wherein the gesture recognition model includes a positioning branch, a matching branch, and a classification branch. The positioning branch is used to identify positioning information of multiple components of at least one target object in an image, and the matching branch is used to obtain the matching relationship between multiple components belonging to the same target object. If the image includes two hands, the matching relationship between the finger part and the palm part of each hand can be accurately distinguished. Since the matching branch has accurately identified the matching relationship between multiple components belonging to the same target object, on this basis, the matching relationship between multiple components of the at least one target object and the positioning information of the components are input into the classification branch. Even complex gestures requiring two hands can be accurately recognized, thereby effectively improving the accuracy of complex gesture recognition.

[0130] An electronic device is also provided in an embodiment of the present application. The electronic device may be the terminal 100 or the server 200 in the above embodiment. The electronic device includes at least one processor and a memory connected to the processor, wherein:

[0131] The memory is used to store computer programs;

[0132] The processor is used to execute the computer program so that the electronic device can implement any gesture recognition method provided in the embodiments of the present application.

[0133] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any gesture recognition method provided in the embodiment of the present application.

[0134] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any gesture recognition method provided in the embodiment of the present application.

[0135] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0136] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0137] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0138] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A gesture recognition method, characterized in that: include: Obtain an image to be recognized; Inputting the image into a positioning branch in a gesture recognition model to identify positioning information of multiple components of at least one target object in the image, wherein the multiple components of the target object include at least a finger portion and a palm portion; Inputting the positioning information of the multiple components of at least one of the target objects into the matching branch in the gesture recognition model to obtain a matching relationship between the multiple components belonging to the same target object; The matching relationship between the multiple components of at least one of the target objects and the positioning information of the multiple components are input into the classification branch in the gesture recognition model to determine the target gesture type corresponding to the image.

2. The gesture recognition method according to claim 1, characterized in that: The positioning branch includes a finger positioning branch and a palm orientation positioning branch, and the gesture recognition method further includes: Acquire a sample set of example gestures, each sample in the sample set of example gestures includes at least one of a directory type of finger location directory, a palm orientation location target, a directory of matching relationships between finger location and palm orientation location, and a directory of gesture identifications of gesture types corresponding to the finger location and the palm orientation location; The corresponding branches in the model to be trained are trained separately using the directory types included in each sample in the example gesture sample set to obtain the gesture recognition model, wherein the finger location directory corresponds to the finger location branch in the model to be trained, the palm orientation location directory corresponds to the palm orientation location branch in the model to be trained, the matching relationship directory between the finger location and the palm orientation location corresponds to the matching branch in the model to be trained, and the gesture identification directory of the gesture types corresponding to the finger location and the palm orientation location corresponds to the classification branch in the model to be trained.

3. The gesture recognition method according to claim 1, characterized in that: The positioning branch includes a finger positioning branch and a palm orientation positioning branch, and the positioning branch inputting the image into the gesture recognition model, identifying the positioning information of multiple components of at least one target object in the image, includes: Inputting the image into the finger positioning branch, identifying positioning information of at least one finger part in the image; The image is input into the palm orientation location branch to identify the palm orientation location of at least one of the palm parts in the image.

4. The gesture recognition method according to claim 3, characterized in that: The step of inputting the positioning information of the multiple components of at least one of the target objects into the matching branch of the gesture recognition model to obtain the matching relationship between the multiple components belonging to the same target object includes: Inputting the positioning information of at least one of the finger parts and the palm orientation positioning of at least one of the palm parts in the image into the matching branch; In the matching branch, for each finger part, a reference palm orientation location corresponding to the positioning information of the finger part is determined, and a target palm orientation location corresponding to the reference palm orientation is determined based on the reference palm orientation location and the palm orientation location of at least one of the palm parts, and the target palm orientation location is determined as the palm orientation location that matches the positioning information of the finger part in the same target object.

5. The gesture recognition method according to claim 4, characterized in that: The determining of the reference palm orientation location corresponding to the location information of the finger portion includes: The second AI vector value is calculated based on the first AI vector value of the finger part and the preset vector, and the second AI vector value is used to represent the reference palm orientation positioning. The first AI vector value of the finger part is the AI ​​vector value corresponding to the center positioning of the finger identification frame, and the preset vector is a vector pointing to the palm orientation positioning of the finger in the same hand obtained by model training.

6. The gesture recognition method according to claim 4, characterized in that: The step of determining a target palm orientation location corresponding to the reference palm orientation according to the reference palm orientation location and the palm orientation location of at least one of the palm parts comprises: The palm orientation location of the palm portion having the smallest distance from the reference palm orientation location is determined as the target palm orientation location corresponding to the reference palm orientation location.

7. The gesture recognition method according to claim 4, characterized in that: The step of inputting the matching relationship between the multiple components of at least one of the target objects and the positioning information of the multiple components into the classification branch of the gesture recognition model to determine the target gesture type corresponding to the image includes: Inputting the palm orientation location matched with the location information of the finger part in the same target object into the classification branch; In the classification branch, a feature vector corresponding to each preset gesture type is generated according to the palm orientation positioning that matches the positioning information of the finger part in the same target object, and the feature vector corresponding to each preset gesture type is dimensionlessly simplified to obtain a dimensionless simplified value corresponding to each preset gesture type, and the preset gesture type corresponding to the maximum dimensionless simplified value is determined as the target gesture type.

8. A gesture recognition device, characterized in that: include: An acquisition unit, used for acquiring an image to be recognized; a positioning unit, configured to input the image into a positioning branch in a gesture recognition model, and identify positioning information of multiple components of at least one target object in the image, wherein the multiple components of the target object at least include a finger portion and a palm portion; a matching unit, configured to input the positioning information of the multiple components of at least one of the target objects into a matching branch in the gesture recognition model to obtain a matching relationship between the multiple components belonging to the same target object; The classification unit is used to input the matching relationship between the multiple components of at least one of the target objects and the positioning information of the multiple components into the classification branch in the gesture recognition model to determine the target gesture type corresponding to the image.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the gesture recognition method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the gesture recognition method as claimed in any one of claims 1 to 7.

11. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the gesture recognition method as described in any one of claims 1 to 7.

Images