Gesture interaction method, device and equipment based on desktop three-dimensional interaction all-in-one machine

Abstract

The invention discloses a gesture interaction method, device and equipment based on a desktop three-dimensional interaction all-in-one machine, and the method comprises the steps: carrying out the recognition of a current operation gesture data of a user in a rehabilitation training virtual scene for a training target model through a gesture recognition device, and obtaining a current operation recognition result; the 3D display all-in-one machine performs model operation on the training target model based on the current operation identification result to obtain a current target operation model; and the gesture recognition equipment continuously detects the hand of the user until the next operation gesture data is not detected or the operation ending instruction is detected. According to the embodiment of the invention, the hand image of the user can be quickly collected based on the gesture recognition equipment, the hand model of the user is correspondingly created in the corresponding three-dimensional space of the rehabilitation training virtual scene, and then the hand model of the user is driven by the actual hand action of the user to perform corresponding operation on the training target model; complex hand actions can be effectively simulated, and corresponding virtual rehabilitation training can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of desktop three-dimensional interaction technology, and in particular to a gesture interaction method, device and equipment based on a desktop three-dimensional interactive all-in-one machine. Background Art

[0002] Although the current desktop 3D interactive all-in-one machine can simulate the real operating environment to a certain extent, the use of laser pen as the main operating tool of the desktop 3D interactive all-in-one machine has become increasingly prominent, specifically: 1) The operation mode of the laser pen is not intuitive and flexible enough, and it is difficult to meet the precise operation requirements of users in the simulated real operation environment provided by the desktop 3D interactive all-in-one machine; 2) If the simulated real operating environment of the desktop 3D interactive all-in-one machine is specifically used for multi-scenario rehabilitation training, the operation method using the laser pen lacks a sense of immersion, making it difficult for users to obtain an immersive operating experience in the virtual environment, thereby reducing the rehabilitation training effect; 3) Existing desktop 3D interactive all-in-one machines cannot simulate complex hand movements. This makes it impossible to complete fine hand operations in many rehabilitation training fields that require fine operations (such as medicine, engineering, etc.), resulting in the inability to complete rehabilitation training. Summary of the invention

[0003] The embodiments of the present invention provide a gesture interaction method, device and equipment based on a desktop three-dimensional interactive all-in-one machine, aiming to solve the problem that the desktop three-dimensional interactive all-in-one machine in the prior art uses a laser pen as the main operating tool, cannot simulate complex hand movements, and cannot complete rehabilitation training in areas requiring fine operations.

[0004] In a first aspect, an embodiment of the present invention provides a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine, which is applied to the desktop three-dimensional interactive all-in-one machine, wherein the desktop three-dimensional interactive all-in-one machine includes a 3D display all-in-one machine, 3D glasses and a gesture recognition device, and the gesture recognition device is communicatively connected with the 3D display all-in-one machine; the gesture interaction method based on the desktop three-dimensional interactive all-in-one machine includes: The 3D display integrated machine responds to the rehabilitation training instruction, obtains the training target model corresponding to the rehabilitation training instruction and displays it in three dimensions in the rehabilitation training virtual scene, so that the user can view the training target model through 3D glasses; If the gesture recognition device detects and collects a set of hand images of the user, a user hand model is created based on the hand image set, and the user hand model is mapped to the rehabilitation training virtual scene; If the gesture recognition device detects the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene, the gesture recognition device recognizes the current operation gesture data to obtain a current operation recognition result; The 3D display integrated machine performs a model operation on the training target model based on the current operation recognition result to obtain a current target operation model; If the gesture recognition device detects the next operation gesture data, the next operation gesture data is used as the current operation gesture data, the current target operation model is used as the training target model, and the step of returning to execute if the gesture recognition device detects the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene, identifying the current operation gesture data to obtain the current operation recognition result is performed until the gesture recognition device stops detecting when it fails to detect the next operation gesture data or detects an operation end instruction.

[0005] In a second aspect, an embodiment of the present invention further provides a gesture interaction device based on a desktop three-dimensional interactive all-in-one machine, which is configured on the desktop three-dimensional interactive all-in-one machine, wherein the desktop three-dimensional interactive all-in-one machine includes a 3D display all-in-one machine, 3D glasses and a gesture recognition device, and the gesture recognition device is communicatively connected with the 3D display all-in-one machine; The 3D display integrated machine is used to respond to the rehabilitation training instruction, obtain the training target model corresponding to the rehabilitation training instruction and display it in three dimensions in the rehabilitation training virtual scene, so that the user can view the training target model through 3D glasses; The gesture recognition device is used to create a user hand model based on the hand image set if a user hand image set is detected and collected, and map the user hand model to the rehabilitation training virtual scene; The gesture recognition device is further configured to recognize the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene to obtain a current operation recognition result if the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene is detected; The 3D display integrated machine is further used to perform a model operation on the training target model based on the current operation recognition result to obtain a current target operation model; The gesture recognition device is also used to, if the next operation gesture data is detected, use the next operation gesture data as the current operation gesture data, use the current target operation model as the training target model, and return to execute the step of identifying the current operation gesture data if the gesture recognition device detects the user's current operation gesture data for the training target model in the rehabilitation training virtual scene to obtain the current operation recognition result, until the gesture recognition device stops detecting when it fails to detect the next operation gesture data or detects an operation end instruction.

[0006] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0008] The embodiment of the present invention provides a gesture interaction method, device and equipment based on a desktop three-dimensional interactive all-in-one machine, which is applied to the desktop three-dimensional interactive all-in-one machine, wherein the desktop three-dimensional interactive all-in-one machine includes a 3D display all-in-one machine, 3D glasses and a gesture recognition device, and the gesture recognition device is communicatively connected with the 3D display all-in-one machine; the method includes: the 3D display all-in-one machine responds to a rehabilitation training instruction, obtains a training target model corresponding to the rehabilitation training instruction and displays it in three dimensions in a rehabilitation training virtual scene, so that the user can view the training target model based on the 3D glasses; if the gesture recognition device detects and collects a user's hand image set, a user hand model is created based on the hand image set, and the user hand model is mapped to the rehabilitation training virtual scene; the gesture recognition device If the user's current operation gesture data for the training target model in the rehabilitation training virtual scene is detected, the current operation gesture data is identified to obtain the current operation recognition result; the 3D display integrated machine performs model operation on the training target model based on the current operation recognition result to obtain the current target operation model; if the gesture recognition device detects the next operation gesture data, the next operation gesture data is used as the current operation gesture data, and returns to execute the step of identifying the current operation gesture data to obtain the current operation recognition result, until the gesture recognition device does not detect the next operation gesture data or detects the operation end instruction. The embodiment of the present invention can quickly collect the user's hand image based on the gesture recognition device and create a corresponding user hand model in the three-dimensional space corresponding to the rehabilitation training virtual scene, and then use the actual user hand movements to drive the user hand model to perform corresponding operations on the training target model in the rehabilitation training virtual scene, which can effectively simulate complex hand movements and realize corresponding virtual rehabilitation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of an application scenario of a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided by an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided by an embodiment of the present invention; Figure 3 A schematic diagram of a sub-flow of a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided in an embodiment of the present invention; Figure 4A schematic diagram of a gesture recognition device and an interaction area in a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided in an embodiment of the present invention; Figure 5 A schematic diagram of an interactive area and a hand image in a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided in an embodiment of the present invention; Figure 6 A schematic diagram of a sub-flow of a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided in an embodiment of the present invention; Figure 7 A schematic diagram of an interactive area and a simplified hand node model in a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided in an embodiment of the present invention; Figure 8 A schematic diagram of a sub-flow of a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided in an embodiment of the present invention; Fig. 9 A schematic block diagram of a gesture interaction device based on a desktop three-dimensional interactive all-in-one machine provided by an embodiment of the present invention; Fig.10 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0013] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0014] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] Please also refer to Figure 1 and Figure 2 ,in Figure 1 Schematic diagram of a scene of a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine according to an embodiment of the present invention. Figure 2 FIG. 1 is a flow chart of a gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided by an embodiment of the present invention. Figure 1 As shown, the gesture interaction method based on a desktop three-dimensional interactive all-in-one machine provided in an embodiment of the present invention is applied to a desktop three-dimensional interactive all-in-one machine, wherein the desktop three-dimensional interactive all-in-one machine includes a 3D display all-in-one machine 10, 3D glasses 20 and a gesture recognition device 30, and the gesture recognition device 20 is communicatively connected with the 3D display all-in-one machine 10. Among them, the 3D display all-in-one machine 10 can be regarded as an all-in-one device integrating a system host (including core modules such as a processor, a memory, and a speaker) and a 3D display, wherein the 3D display can display various rehabilitation training virtual scenes and models of a multi-scene rehabilitation training system in the system host. A user input device (such as a mouse, a keyboard, etc.) can also be connected to the 3D display all-in-one machine 10, and the user can enter the rehabilitation training simulation system deployed in the desktop three-dimensional interactive all-in-one machine and perform subsequent operations by operating the user input device.

[0016] like Figure 2 As shown, the method includes the following steps S110-S150.

[0017] S110, the 3D display integrated machine responds to the rehabilitation training instruction, obtains the training target model corresponding to the rehabilitation training instruction and displays it in three dimensions in the rehabilitation training virtual scene, so that the user can view the training target model through 3D glasses.

[0018] In this embodiment, users who need rehabilitation training need to wear 3D glasses first. When the user subsequently operates the desktop three-dimensional interactive all-in-one machine and enters the rehabilitation training simulation system, on the user interaction interface of the desktop three-dimensional interactive all-in-one machine, a rehabilitation training virtual scene can be selected from the multiple rehabilitation training virtual scenes displayed (such as left-hand training virtual scenes, right-hand training virtual scenes, two-hand training virtual scenes, etc.) to trigger the generation of rehabilitation training instructions. Then, the selected rehabilitation training virtual scene and its corresponding training target model (such as a virtual model of reaching height, a kitchen simulation model, etc.) are displayed in three dimensions on the 3D display of the desktop three-dimensional interactive all-in-one machine. Because the user has worn 3D glasses, the training target model displayed in three dimensions can be viewed, and rehabilitation training can be performed on it in a three-dimensional virtual scene afterwards.

[0019] Among them, because the rehabilitation training virtual scene selected by the user is for a specific rehabilitation training virtual scene, when the training target model is a virtual model of reaching height, it has a corresponding three-dimensional display state. For example, the rehabilitation training virtual scene is a virtual scene of reaching height in the right hand training virtual scene, and the virtual scene of reaching height in the right hand training virtual scene includes at least a small ball model and a wall model of reaching height (a plurality of ball holes are provided at different heights of the wall model of reaching height to accommodate the small ball model). In the virtual scene of reaching height in the right hand training virtual scene, the user needs to pick up the small ball and place it in one of the ball holes in the wall model of reaching height to complete the rehabilitation training of reaching height for the right hand.

[0020] For another example, the rehabilitation training virtual scene is a kitchen virtual scene in the two-handed training virtual scene, and the kitchen virtual scene in the two-handed training virtual scene includes at least a bread model, a toaster model, a bread shovel model, a hamburger sauce model, a vegetable and hamburger meat model, a hamburger box model, etc. In the kitchen virtual scene in the two-handed training virtual scene, the user needs to first pick up the bread and put it into the toaster, cover the toaster, start the bread baking, and after baking, use a bread shovel to scoop the bread out of the toaster, squeeze hamburger sauce in the bread layer, place vegetables and hamburger meat in the bread layer to make a hamburger, and put the hamburger into the hamburger box. The kitchen operation rehabilitation training is completed by completing the above operations.

[0021] S120: If the gesture recognition device detects and collects a hand image set of the user, a user hand model is created based on the hand image set, and the user hand model is mapped to the rehabilitation training virtual scene.

[0022] In this embodiment, when the user performs rehabilitation training for the training target model in the rehabilitation training virtual scene, the gesture recognition device in the desktop three-dimensional interactive all-in-one machine (in specific implementation, a Leap Motion device including two infrared cameras and three LED infrared lights can be used, which is a gesture recognition device that combines sensors and infrared cameras and can be used to track and identify the movement of hands and fingertips) can first detect and collect at least one image of the user's hand to form a hand image set. Then, a user hand model is created based on the hand image set. Finally, the user hand model is mapped to the rehabilitation training virtual scene. It can be seen that through the above process, the user's hand image can be quickly detected and mapped to the rehabilitation training virtual scene where the training target model is located, and the user's hand is displayed as a three-dimensional model.

[0023] In one embodiment, if Figure 3 As shown, step S120 includes: S121, if it is detected that a user's hand enters an interaction area corresponding to the gesture recognition device, obtaining a preset hand image acquisition strategy; S122, performing image acquisition of the user's hands in the interaction area based on the hand image acquisition strategy to obtain the hand image set; S123, reconstructing a three-dimensional hand virtual model of the hand image set in the rehabilitation training virtual scene based on a preset three-dimensional reconstruction strategy to obtain the user hand model, and mapping the user hand model to the rehabilitation training virtual scene.

[0024] In this embodiment, before the user needs to operate the training target model in the selected rehabilitation training virtual scene, Figure 4 and Figure 5 As shown, first, after the user wears 3D glasses and sees the interactive area 31 corresponding to the gesture recognition device 30 (for example, a rectangular interactive area), the user's hands need to move to the interactive area 31. After that, the hand image acquisition strategy is obtained (for example, multiple hand images are acquired according to a preset acquisition cycle, more specifically, the preset acquisition cycle is set to 0.1s, 0.2s, 0.3s, etc.), and the binocular camera in the gesture recognition device 30 is used to acquire images of the user's hands in the interactive area 31 based on the hand image acquisition strategy to obtain the hand image set. Finally, based on the camera data and three-dimensional reconstruction strategy corresponding to the hand image set, a user hand model of the user's hands in the three-dimensional space of the rehabilitation training virtual scene (that is, the three-dimensional space corresponding to the real physical world viewed in the 3D display of the 3D display integrated machine after the user wears 3D glasses) is created, and the user hand model is mapped to the rehabilitation training virtual scene.

[0025] After the above-mentioned user hand model mapping operation is completed, various hand movements of the user's hands in the interactive area will drive the same movements of the user's hand model in the rehabilitation training virtual scene. For example, if the user performs actions such as pinching, releasing, pushing, turning, clicking, and confirming, the same movements of the user's hand model in the rehabilitation training virtual scene will be driven.

[0026] In one embodiment, after step S121 and before step S122, the following steps are further included: If it is detected that the user's hand is captured and gesture recognized in the interaction area corresponding to the gesture recognition device and corresponds to the preset gesture recognition result, a hand image capture instruction is generated to trigger the execution of the step of capturing the image of the user's hand in the interaction area based on the hand image capture strategy to obtain the hand image set.

[0027] In this embodiment, after the gesture recognition device is initialized, it detects for the first time that the user's hand has entered the interactive area. Therefore, in order to more accurately trigger the image acquisition operation of the user's hand, it is necessary to prompt the user's hand to make a specified gesture (for example, the system voice prompt can be played through the speaker of the 3D display all-in-one machine to prompt the user to make an OK gesture with the right hand). When the user's right hand makes a specified gesture in the interactive area (such as the OK gesture listed in the above example), it means that the gesture recognition device can perform subsequent hand image acquisition and user hand model creation operations. Afterwards, the user can quickly open the right hand and still stay in the interactive area like the left hand, and no longer make the right hand to make the specified gesture. It can be seen that by using the above-mentioned method of triggering the generation of hand image acquisition instructions with a specified gesture, it is avoided that the user needs to use complex operations to trigger the gesture recognition device to perform subsequent hand image acquisition and user hand model creation operations.

[0028] In one embodiment, if Figure 6 As shown, step S123 includes: S1231, performing image preprocessing on each hand image in the hand image set based on a preset image preprocessing strategy to obtain a preprocessed hand image corresponding to each hand image, and forming a preprocessed hand image set; S1232, identifying the key hand nodes of each preprocessed hand image in the preprocessed hand image set based on a preset hand joint point recognition model, and obtaining a simplified hand node model corresponding to each preprocessed hand image; S1233. Determine the user hand model based on the image information of each preprocessed hand image in the preprocessed hand image set and the corresponding simplified hand node model, and map the user hand model to the rehabilitation training virtual scene; wherein the user hand model is a three-dimensional hand virtual model corresponding to the hand image set and used to be displayed in the rehabilitation training virtual scene.

[0029] In this embodiment, when performing image preprocessing on each hand image in the hand image set based on the image preprocessing strategy, taking the processing of a hand image as an example, the image preprocessing strategy is used to process the hand image mainly to eliminate background noise (such as performing edge detection through the Canny operator, Roberts operator, etc. to eliminate background noise in the hand image and highlight the hand and fingers), thereby obtaining a corresponding preprocessed hand image. In the hand image set, each hand image and its corresponding preprocessed hand image retain depth information so that they can be mapped to a three-dimensional space later.

[0030] Afterwards, the hand joint point recognition model (specifically, a hand gesture estimation model based on a Deep Map depth image) is used to identify the key hand nodes of each pre-processed hand image in the pre-processed hand image set. For example, taking the processing of a pre-processed hand image as an example, the hand joint point recognition model is used to identify the key hand nodes of the pre-processed hand image (for example, the pre-processed hand image is Figure 5 21 key nodes of the hand model are identified in the left hand image and the right hand image respectively, as shown in the following figure: Figure 7 The simplified node model of the hand shown ( Figure 7 The rectangular box in the figure represents the interactive area 31).

[0031] Finally, based on the image information of each preprocessed hand image in the preprocessed hand image set (the image information mainly includes the depth of field information of each pixel point in the preprocessed hand image) and the corresponding simplified hand node model, the three-dimensional spatial coordinates of each key node of the hand model in the simplified hand node model in the three-dimensional space corresponding to the rehabilitation training virtual scene are determined, thereby realizing the mapping of the user hand model to the rehabilitation training virtual scene, and displaying the user's hand in the real physical world as a three-dimensional hand virtual model in the three-dimensional space corresponding to the rehabilitation training virtual scene. Because in the preprocessed hand image set, a user hand sub-model can be determined by combining the image information of a preprocessed hand image and the corresponding simplified hand node model, multiple preprocessed hand images can correspond to multiple user hand sub-models.

[0032] After the user hand sub-models are determined based on the image information of multiple pre-processed hand images and the corresponding simplified hand node models, at least one user hand sub-model can be finally used as the user hand model in the following way: if it is determined that the image information of each pre-processed hand image in the pre-processed hand image set and the user hand sub-models determined by the corresponding simplified hand node models are the same, then the user hand sub-model corresponding to the last pre-processed hand image in the pre-processed hand image set is used as the user hand model. That is, when it is determined that the image information of each pre-processed hand image in the pre-processed hand image set and the user hand sub-models determined by the corresponding simplified hand node models are the same, it means that the user keeps both hands at a fixed position in the interaction area for the gesture recognition device to collect hand images and create a user hand model. It can be seen that based on the above process, the user's hands are quickly mapped from the real physical world to the three-dimensional space of the virtual scene of rehabilitation training.

[0033] In one embodiment, step S1233 includes: Based on the reference coordinate information corresponding to the interactive area; Obtaining image information of the i-th pre-processed hand image in the pre-processed hand image set; wherein the initial value of i is 1; Based on the image information of the i-th preprocessed hand image, the simplified hand node model corresponding to the i-th preprocessed hand image and the reference coordinate information, determine the i-th user hand sub-model corresponding to the simplified hand node model corresponding to the i-th preprocessed hand image in the rehabilitation training virtual scene; Increment i by 1 to update the value of i; If it is determined that i does not exceed N, then return to the step of obtaining the image information of the i-th pre-processed hand image in the pre-processed hand image set; wherein N represents the total number of pre-processed hand images included in the pre-processed hand image set; If it is determined that i exceeds N, any one of the first to Nth user hand sub-models is used as the user hand model, and the user hand model is mapped to the rehabilitation training virtual scene.

[0034] In this embodiment, creating a user hand sub-model corresponding to a pre-processed hand image in the pre-processed hand image set is taken as an example, and more specifically, creating a user hand sub-model corresponding to the Nth pre-processed hand image is taken as an example. After user hand sub-models are created corresponding to the 1st pre-processed hand image to the N-1th pre-processed hand image, when creating the Nth user hand sub-model based on the image information of the Nth pre-processed hand image, the simplified hand node model corresponding to the Nth pre-processed hand image and the reference coordinate information, the reference coordinate information at least includes three-dimensional spatial coordinate information corresponding to the four vertices of the interaction area, and after combining the image information based on the Nth pre-processed hand image (wherein the depth of field information of each pixel point in the pre-processed hand image is known) and the spatial relative position information of each hand model key node in the simplified hand node model corresponding to the Nth pre-processed hand image compared to the four vertices in the interaction area, each hand model key node in the simplified hand node model corresponding to the Nth pre-processed hand image can be mapped to the three-dimensional space corresponding to the rehabilitation training virtual scene, thereby obtaining the Nth user hand sub-model (which can be regarded as a three-dimensional hand simplified node model mapped in the three-dimensional space). At this time, in order to determine the user hand model more quickly, any user hand sub-model from the first user hand sub-model to the Nth user hand sub-model can be directly used as the user hand model, and the user hand model can be mapped to the rehabilitation training virtual scene.

[0035] S130: If the gesture recognition device detects the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene, the gesture recognition device recognizes the current operation gesture data to obtain a current operation recognition result.

[0036] In this embodiment, since the rendering and display of the training target model have been completed in the rehabilitation training virtual scene (the training target model includes multiple target rehabilitation training sub-models, and the multiple target rehabilitation training sub-models are displayed in an initial combination or separation under the corresponding scene type of the rehabilitation training virtual scene), and the creation and display of the user hand model have been completed, the user's current operation gesture data for the training target model in the rehabilitation training virtual scene can be dynamically detected in real time based on the gesture recognition device. Afterwards, the current operation gesture data can still be recognized based on the gesture posture estimation model of the Deep Map depth image in the gesture recognition device to obtain the current operation recognition result, for example, the current operation recognition result is grabbing, pushing, turning, etc.

[0037] S140: The 3D display integrated machine performs a model operation on the training target model based on the current operation recognition result to obtain a current target operation model.

[0038] In this embodiment, after the current operation recognition result is determined in the gesture recognition device, the user hand model is driven to synchronously perform model operations on the training target model, thereby updating the training target model and obtaining the current target operation model (such as the specific operation of the virtual scene of reaching height in the right hand training virtual scene in the above example, or the specific operation of the virtual scene of kitchen in the virtual scene of two-hand training). It can be seen that through the above method, the corresponding model operation and model update of the training target model in the rehabilitation training virtual scene based on the user's real-time operation are realized quickly.

[0039] In one embodiment, if Figure 8 As shown, step S140 includes: S141, obtaining the current target operation sub-model targeted by the current operation recognition result in the training target model; S142. Update the model position and state of the current target operation sub-model based on the model operation corresponding to the current operation recognition result to update the training target model and obtain the current target operation model.

[0040] In this embodiment, when the current operation recognition result corresponding to the current operation gesture data is detected by the gesture recognition device, the user hand model can also be driven to move to the corresponding three-dimensional space position in the rehabilitation training virtual scene based on the current operation gesture data. If the user hand model at this time has an overlapping area with the target operation sub-model in the training target model, it means that the user needs to use the target operation sub-model as the current target operation sub-model and perform corresponding model operations, such as grabbing, pushing, rotating, etc. on the current target operation sub-model, and then specifically update the model position and state of the current target operation sub-model based on the model operation corresponding to the current operation recognition result, so as to update the training target model and obtain the current target operation model. It can be seen that the operation performed by the user's hand on the training target model in the interactive area in the real physical world can be synchronized to the target operation model in a timely manner, and the model state in the three-dimensional space can be updated in a timely manner.

[0041] S150: If the gesture recognition device detects the next operation gesture data, the next operation gesture data is used as the current operation gesture data, the current target operation model is used as the training target model, and the step of returning to execute: if the gesture recognition device detects the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene, the current operation gesture data is recognized to obtain the current operation recognition result, until the gesture recognition device stops detecting when it fails to detect the next operation gesture data or detects an operation end instruction.

[0042] In this embodiment, if the gesture recognition device detects the next operation gesture data, it means that the user still needs to perform corresponding model operations on the training target model in the rehabilitation training virtual scene. At this time, the next operation gesture data is first completed as the current operation gesture data, and the current target operation model is used as the update operation of the training target model, and then the execution returns to step S130.

[0043] Only when the gesture recognition device does not detect the next operation gesture data (for example, the gesture recognition device does not detect the user's hand image for a duration exceeding the first preset waiting time, such as the first preset waiting time of 30s, 45s, 60s, etc., it is deemed that the gesture recognition device does not detect the next operation gesture data), or detects an operation end instruction (for example, there is an end operation virtual button in a designated area of ​​the rehabilitation training virtual scene, and the user drives the user hand model to click the end operation virtual button, or the user issues a voice control instruction and is recognized as an operation end instruction, or the user drives the user hand model to make a designated gesture such as making a fist with both hands and determines the corresponding operation end instruction), the gesture recognition device can enter the standby state and end the detection and recording of the user's complete operation process for the training target model. For example, when the technical solution of the present application is applied to the intelligent training process of upper limb function assessment during specific implementation, the rehabilitation trainee can operate a more immersive and interactive three-dimensional virtual operation based on a desktop three-dimensional interactive all-in-one machine.

[0044] In one embodiment, after step S140, the method further includes: If the gesture recognition device does not detect the next operation gesture data or detects the operation end instruction, it obtains the recorded video of the complete operation process for the training target model and saves it to a local preset storage area.

[0045] In this embodiment, during the multiple rounds of execution of steps S130-S150, the 3D display integrated machine can record the complete operation process of the user on the training target model in the rehabilitation training virtual scene, and obtain the complete operation process video data corresponding to the training target model and save it locally. Afterwards, other users (for example, when the previous user is a student role and the other user is a teacher role) can play and review the complete operation process video data locally multiple times, and make text comments or ratings on the user's complete operation process on the training target model.

[0046] It can be seen that the embodiment of the method can quickly capture the user's hand image based on the gesture recognition device and create a corresponding user hand model in the three-dimensional space corresponding to the rehabilitation training virtual scene. Then, the user's actual hand movements are used to drive the user hand model to perform corresponding operations on the training target model in the rehabilitation training virtual scene, which can effectively simulate complex hand movements and realize corresponding virtual rehabilitation training.

[0047] Fig. 9 is a schematic block diagram of a gesture interaction device based on a desktop three-dimensional interactive all-in-one machine provided by an embodiment of the present invention. Fig. 9As shown, corresponding to the above-mentioned gesture interaction method based on the desktop 3D interactive all-in-one machine, the present invention also provides a gesture interaction device 100 based on the desktop 3D interactive all-in-one machine. Figure 1 and Fig. 9 As shown, the gesture interaction device 100 based on the desktop three-dimensional interactive all-in-one machine is configured on the desktop three-dimensional interactive all-in-one machine, and the desktop three-dimensional interactive all-in-one machine includes a 3D display all-in-one machine 10, 3D glasses 20 and a gesture recognition device 30, and the gesture recognition device 20 is communicatively connected with the 3D display all-in-one machine 10. Among them, the 3D display all-in-one machine 10 can be regarded as an all-in-one device integrating a system host (including core modules such as a processor, a memory, and a speaker) and a 3D display, wherein the 3D display can display various rehabilitation training virtual scenes and models of the multi-scene rehabilitation training system in the system host. User input devices (such as a mouse, a keyboard, etc.) can also be connected to the 3D display all-in-one machine 10. By operating the user input device, the user can enter the rehabilitation training simulation system deployed in the desktop three-dimensional interactive all-in-one machine and perform subsequent operations.

[0048] The 3D display integrated machine 10 is used to respond to the rehabilitation training instruction, obtain the training target model corresponding to the rehabilitation training instruction and display it in three dimensions in the rehabilitation training virtual scene, so that the user can view the training target model through 3D glasses.

[0049] In this embodiment, users who need rehabilitation training need to wear 3D glasses first. When the user subsequently operates the desktop three-dimensional interactive all-in-one machine and enters the rehabilitation training simulation system, on the user interaction interface of the desktop three-dimensional interactive all-in-one machine, a rehabilitation training virtual scene can be selected from the multiple rehabilitation training virtual scenes displayed (such as left-hand training virtual scenes, right-hand training virtual scenes, two-hand training virtual scenes, etc.) to trigger the generation of rehabilitation training instructions. Then, the selected rehabilitation training virtual scene and its corresponding training target model (such as a virtual model of reaching height, a kitchen simulation model, etc.) are displayed in three dimensions on the 3D display of the desktop three-dimensional interactive all-in-one machine. Because the user has worn 3D glasses, the training target model displayed in three dimensions can be viewed, and rehabilitation training can be performed on it in a three-dimensional virtual scene afterwards.

[0050] Among them, because the rehabilitation training virtual scene selected by the user is for a specific rehabilitation training virtual scene, when the training target model is a virtual model of reaching height, it has a corresponding three-dimensional display state. For example, the rehabilitation training virtual scene is a virtual scene of reaching height in the right hand training virtual scene, and the virtual scene of reaching height in the right hand training virtual scene includes at least a small ball model and a wall model of reaching height (a plurality of ball holes are provided at different heights of the wall model of reaching height to accommodate the small ball model). In the virtual scene of reaching height in the right hand training virtual scene, the user needs to pick up the small ball and place it in one of the ball holes in the wall model of reaching height to complete the rehabilitation training of reaching height for the right hand.

[0051] For another example, the rehabilitation training virtual scene is a kitchen virtual scene in the two-handed training virtual scene, and the kitchen virtual scene in the two-handed training virtual scene includes at least a bread model, a toaster model, a bread shovel model, a hamburger sauce model, a vegetable and hamburger meat model, a hamburger box model, etc. In the kitchen virtual scene in the two-handed training virtual scene, the user needs to first pick up the bread and put it into the toaster, cover the toaster, start the bread baking, and after baking, use a bread shovel to scoop the bread out of the toaster, squeeze hamburger sauce in the bread layer, place vegetables and hamburger meat in the bread layer to make a hamburger, and put the hamburger into the hamburger box. The kitchen operation rehabilitation training is completed by completing the above operations.

[0052] The gesture recognition device 30 is used to create a user hand model based on the hand image set if a user hand image set is detected and collected, and map the user hand model to the rehabilitation training virtual scene.

[0053] In this embodiment, when the user performs rehabilitation training for the training target model in the rehabilitation training virtual scene, the gesture recognition device in the desktop three-dimensional interactive all-in-one machine (in specific implementation, a Leap Motion device including two infrared cameras and three LED infrared lights can be used, which is a gesture recognition device that combines sensors and infrared cameras and can be used to track and identify the movement of hands and fingertips) can first detect and collect at least one image of the user's hand to form a hand image set. Then, a user hand model is created based on the hand image set. Finally, the user hand model is mapped to the rehabilitation training virtual scene. It can be seen that through the above process, the user's hand image can be quickly detected and mapped to the rehabilitation training virtual scene where the training target model is located, and the user's hand is displayed as a three-dimensional model.

[0054] In one embodiment, the gesture recognition device 30 is specifically used for: If it is detected that a user's hand enters the interaction area corresponding to the gesture recognition device, a preset hand image acquisition strategy is obtained; Based on the hand image acquisition strategy, images of the user's hands in the interaction area are acquired to obtain the hand image set; Based on a preset three-dimensional reconstruction strategy, a three-dimensional hand virtual model is reconstructed for the hand image set in the rehabilitation training virtual scene to obtain the user hand model, and the user hand model is mapped to the rehabilitation training virtual scene.

[0055] In this embodiment, before the user needs to operate the training target model in the selected rehabilitation training virtual scene, Figure 4 and Figure 5 As shown, first, after the user wears 3D glasses and sees the interactive area 31 corresponding to the gesture recognition device 30 (for example, a rectangular interactive area), the user's hands need to move to the interactive area 31. After that, the hand image acquisition strategy is obtained (for example, multiple hand images are acquired according to a preset acquisition cycle, more specifically, the preset acquisition cycle is set to 0.1s, 0.2s, 0.3s, etc.), and the binocular camera in the gesture recognition device 30 is used to acquire images of the user's hands in the interactive area 31 based on the hand image acquisition strategy to obtain the hand image set. Finally, based on the camera data and three-dimensional reconstruction strategy corresponding to the hand image set, a user hand model of the user's hands in the three-dimensional space of the rehabilitation training virtual scene (that is, the three-dimensional space corresponding to the real physical world viewed in the 3D display of the 3D display integrated machine after the user wears 3D glasses) is created, and the user hand model is mapped to the rehabilitation training virtual scene.

[0056] After the above-mentioned user hand model mapping operation is completed, various hand movements of the user's hands in the interactive area will drive the same movements of the user's hand model in the rehabilitation training virtual scene. For example, if the user performs actions such as pinching, releasing, pushing, turning, clicking, and confirming, the same movements of the user's hand model in the rehabilitation training virtual scene will be driven.

[0057] In one embodiment, the gesture recognition device 30 is further specifically used for: If it is detected that the user's hand is captured and gesture recognized in the interaction area corresponding to the gesture recognition device and corresponds to the preset gesture recognition result, a hand image capture instruction is generated to trigger the execution of the step of capturing the image of the user's hand in the interaction area based on the hand image capture strategy to obtain the hand image set.

[0058] In this embodiment, after the gesture recognition device is initialized, it detects for the first time that the user's hand has entered the interactive area. Therefore, in order to more accurately trigger the image acquisition operation of the user's hand, it is necessary to prompt the user's hand to make a specified gesture (for example, the system voice prompt can be played through the speaker of the 3D display all-in-one machine to prompt the user to make an OK gesture with the right hand). When the user's right hand makes a specified gesture in the interactive area (such as the OK gesture listed in the above example), it means that the gesture recognition device can perform subsequent hand image acquisition and user hand model creation operations. Afterwards, the user can quickly open the right hand and still stay in the interactive area like the left hand, and no longer make the right hand to make the specified gesture. It can be seen that by using the above-mentioned method of triggering the generation of hand image acquisition instructions with a specified gesture, it is avoided that the user needs to use complex operations to trigger the gesture recognition device to perform subsequent hand image acquisition and user hand model creation operations.

[0059] In one embodiment, the reconstructing a three-dimensional hand virtual model of the hand image set in the rehabilitation training virtual scene based on a preset three-dimensional reconstruction strategy to obtain the user hand model, and mapping the user hand model to the rehabilitation training virtual scene includes: Performing image preprocessing on each hand image in the hand image set based on a preset image preprocessing strategy to obtain a preprocessed hand image corresponding to each hand image, and forming a preprocessed hand image set; Based on a preset hand joint point recognition model, key hand nodes are identified for each preprocessed hand image in the preprocessed hand image set to obtain simplified hand node models corresponding to each preprocessed hand image; Based on the image information of each preprocessed hand image in the preprocessed hand image set and the corresponding simplified hand node model, the user hand model is determined, and the user hand model is mapped to the rehabilitation training virtual scene; wherein the user hand model is a three-dimensional hand virtual model corresponding to the hand image set and used to be displayed in the rehabilitation training virtual scene.

[0060] In this embodiment, when performing image preprocessing on each hand image in the hand image set based on the image preprocessing strategy, taking the processing of a hand image as an example, the image preprocessing strategy is used to process the hand image mainly to eliminate background noise (such as performing edge detection through the Canny operator, Roberts operator, etc. to eliminate background noise in the hand image and highlight the hand and fingers), thereby obtaining a corresponding preprocessed hand image. In the hand image set, each hand image and its corresponding preprocessed hand image retain depth information so that they can be mapped to a three-dimensional space later.

[0061] Afterwards, the hand joint point recognition model (specifically, a hand gesture estimation model based on a Deep Map depth image) is used to identify the key hand nodes of each pre-processed hand image in the pre-processed hand image set. For example, taking the processing of a pre-processed hand image as an example, the hand joint point recognition model is used to identify the key hand nodes of the pre-processed hand image (for example, the pre-processed hand image is Figure 5 21 key nodes of the hand model are identified in the left hand image and the right hand image respectively, as shown in the following figure: Figure 7 The simplified node model of the hand shown ( Figure 7 The rectangular box in the figure represents the interactive area 31).

[0062] Finally, based on the image information of each preprocessed hand image in the preprocessed hand image set (the image information mainly includes the depth of field information of each pixel point in the preprocessed hand image) and the corresponding simplified hand node model, the three-dimensional spatial coordinates of each key node of the hand model in the simplified hand node model in the three-dimensional space corresponding to the rehabilitation training virtual scene are determined, thereby realizing the mapping of the user hand model to the rehabilitation training virtual scene, and displaying the user's hand in the real physical world as a three-dimensional hand virtual model in the three-dimensional space corresponding to the rehabilitation training virtual scene. Because in the preprocessed hand image set, a user hand sub-model can be determined by combining the image information of a preprocessed hand image and the corresponding simplified hand node model, multiple preprocessed hand images can correspond to multiple user hand sub-models.

[0063] After the user hand sub-models are determined based on the image information of multiple pre-processed hand images and the corresponding simplified hand node models, at least one user hand sub-model can be finally used as the user hand model in the following way: if it is determined that the image information of each pre-processed hand image in the pre-processed hand image set and the user hand sub-models determined by the corresponding simplified hand node models are the same, then the user hand sub-model corresponding to the last pre-processed hand image in the pre-processed hand image set is used as the user hand model. That is, when it is determined that the image information of each pre-processed hand image in the pre-processed hand image set and the user hand sub-models determined by the corresponding simplified hand node models are the same, it means that the user keeps both hands at a fixed position in the interaction area for the gesture recognition device to collect hand images and create a user hand model. It can be seen that based on the above process, the user's hands are quickly mapped from the real physical world to the three-dimensional space of the virtual scene of rehabilitation training.

[0064] In one embodiment, the reconstructing a three-dimensional hand virtual model of the hand image set in the rehabilitation training virtual scene based on a preset three-dimensional reconstruction strategy to obtain the user hand model, and mapping the user hand model to the rehabilitation training virtual scene includes: Based on the reference coordinate information corresponding to the interactive area; Obtaining image information of the i-th pre-processed hand image in the pre-processed hand image set; wherein the initial value of i is 1; Based on the image information of the i-th preprocessed hand image, the simplified hand node model corresponding to the i-th preprocessed hand image and the reference coordinate information, determine the i-th user hand sub-model corresponding to the simplified hand node model corresponding to the i-th preprocessed hand image in the rehabilitation training virtual scene; Increment i by 1 to update the value of i; If it is determined that i does not exceed N, then return to the step of obtaining the image information of the i-th pre-processed hand image in the pre-processed hand image set; wherein N represents the total number of pre-processed hand images included in the pre-processed hand image set; If it is determined that i exceeds N, any one of the first to Nth user hand sub-models is used as the user hand model, and the user hand model is mapped to the rehabilitation training virtual scene.

[0065] In this embodiment, creating a user hand sub-model corresponding to a pre-processed hand image in the pre-processed hand image set is taken as an example, and more specifically, creating a user hand sub-model corresponding to the Nth pre-processed hand image is taken as an example. After user hand sub-models are created corresponding to the 1st pre-processed hand image to the N-1th pre-processed hand image, when creating the Nth user hand sub-model based on the image information of the Nth pre-processed hand image, the simplified hand node model corresponding to the Nth pre-processed hand image and the reference coordinate information, the reference coordinate information at least includes three-dimensional spatial coordinate information corresponding to the four vertices of the interaction area, and after combining the image information based on the Nth pre-processed hand image (wherein the depth of field information of each pixel point in the pre-processed hand image is known) and the spatial relative position information of each hand model key node in the simplified hand node model corresponding to the Nth pre-processed hand image compared to the four vertices in the interaction area, each hand model key node in the simplified hand node model corresponding to the Nth pre-processed hand image can be mapped to the three-dimensional space corresponding to the rehabilitation training virtual scene, thereby obtaining the Nth user hand sub-model (which can be regarded as a three-dimensional hand simplified node model mapped in the three-dimensional space). At this time, in order to determine the user hand model more quickly, any user hand sub-model from the first user hand sub-model to the Nth user hand sub-model can be directly used as the user hand model, and the user hand model can be mapped to the rehabilitation training virtual scene.

[0066] The gesture recognition device 30 is further used to recognize the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene to obtain a current operation recognition result if the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene is detected.

[0067] In this embodiment, since the rendering and display of the training target model have been completed in the rehabilitation training virtual scene (the training target model includes multiple target rehabilitation training sub-models, and the multiple target rehabilitation training sub-models are displayed in an initial combination or separation under the corresponding scene type of the rehabilitation training virtual scene), and the creation and display of the user hand model have been completed, the user's current operation gesture data for the training target model in the rehabilitation training virtual scene can be dynamically detected in real time based on the gesture recognition device. Afterwards, the current operation gesture data can still be recognized based on the gesture posture estimation model of the Deep Map depth image in the gesture recognition device to obtain the current operation recognition result, for example, the current operation recognition result is grabbing, pushing, turning, etc.

[0068] The 3D display all-in-one machine 10 is further used to perform model operation on the training target model based on the current operation recognition result to obtain a current target operation model.

[0069] In this embodiment, after the current operation recognition result is determined in the gesture recognition device, the user hand model is driven to synchronously perform model operations on the training target model, thereby updating the training target model and obtaining the current target operation model. It can be seen that through the above method, the corresponding model operations and model updates of the training target model in the rehabilitation training virtual scene based on the user's real-time operation are realized.

[0070] In one embodiment, the 3D display integrated machine 10 is specifically used for: Obtaining a current target operation sub-model targeted by the current operation recognition result in the training target model; The model position and state of the current target operation sub-model are updated based on the model operation corresponding to the current operation recognition result to update the training target model and obtain the current target operation model.

[0071] In this embodiment, when the current operation recognition result corresponding to the current operation gesture data is detected by the gesture recognition device, the user hand model can also be driven to move to the corresponding three-dimensional space position in the rehabilitation training virtual scene based on the current operation gesture data. If the user hand model at this time has an overlapping area with the target operation sub-model in the training target model, it means that the user needs to use the target operation sub-model as the current target operation sub-model and perform corresponding model operations, such as grabbing, pushing, rotating, etc. on the current target operation sub-model, and then specifically update the model position and state of the current target operation sub-model based on the model operation corresponding to the current operation recognition result, so as to update the training target model and obtain the current target operation model. It can be seen that the operation performed by the user's hand on the training target model in the interactive area in the real physical world can be synchronized to the target operation model in a timely manner, and the model state in the three-dimensional space can be updated in a timely manner.

[0072] The gesture recognition device is also used to, if the next operation gesture data is detected, use the next operation gesture data as the current operation gesture data, use the current target operation model as the training target model, and return to execute the step of identifying the current operation gesture data if the gesture recognition device detects the user's current operation gesture data for the training target model in the rehabilitation training virtual scene to obtain the current operation recognition result, until the gesture recognition device stops detecting when it fails to detect the next operation gesture data or detects an operation end instruction.

[0073] In this embodiment, if the gesture recognition device detects the next operation gesture data, it means that the user still needs to perform corresponding model operations on the training target model in the rehabilitation training virtual scene. At this time, the next operation gesture data is first completed as the current operation gesture data, and the current target operation model is used as the update operation of the training target model, and then the execution returns to step S130.

[0074] Only when the gesture recognition device does not detect the next operation gesture data (for example, the gesture recognition device does not detect the user's hand image for a duration exceeding the first preset waiting time, such as the first preset waiting time of 30s, 45s, 60s, etc., it is deemed that the gesture recognition device does not detect the next operation gesture data), or detects an operation end instruction (for example, there is an end operation virtual button in a designated area of ​​the rehabilitation training virtual scene, and the user drives the user hand model to click the end operation virtual button, or the user issues a voice control instruction and is recognized as an operation end instruction, or the user drives the user hand model to make a designated gesture such as making a fist with both hands and determines the corresponding operation end instruction), the gesture recognition device can enter the standby state and end the detection and recording of the user's complete operation process for the training target model. For example, when the technical solution of the present application is applied to the intelligent training process of upper limb function assessment during specific implementation, the rehabilitation trainee can operate a more immersive and interactive three-dimensional virtual operation based on a desktop three-dimensional interactive all-in-one machine.

[0075] In one embodiment, the gesture recognition device 30 is further used for: If the next operation gesture data is not detected or the operation end instruction is detected, a recorded video of the complete operation process for the training target model is obtained and saved to a local preset storage area.

[0076] In this embodiment, the 3D display integrated machine can record the complete operation process of the user on the training target model in the rehabilitation training virtual scene, and obtain the complete operation process video data corresponding to the training target model and save it locally. Afterwards, other users (for example, when the previous user is a student role and the other user is a teacher role) can play and review the complete operation process video data locally multiple times, and make text comments or ratings on the user's complete operation process on the training target model.

[0077] It can be seen that the embodiment of the device can quickly capture the user's hand image based on the gesture recognition device and create a corresponding user hand model in the three-dimensional space corresponding to the virtual scene of rehabilitation training. Then, the user's actual hand movements are used to drive the user hand model to perform corresponding operations on the training target model in the virtual scene of rehabilitation training, which can effectively simulate complex hand movements and realize corresponding virtual rehabilitation training.

[0078] The above-mentioned gesture interaction device based on the desktop three-dimensional interactive all-in-one machine can be implemented in the form of a computer program. The computer program can be used in a computer system such as Fig. 9 Runs on the computer device shown.

[0079] See also Fig.10 , Fig.10 The schematic block diagram of a computer device provided by an embodiment of the present invention is shown in FIG. The computer device integrates any one of the gesture interaction devices based on a desktop three-dimensional interactive all-in-one machine provided by an embodiment of the present invention.

[0080] See also Fig.10 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .

[0081] The storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 can execute the above-mentioned gesture interaction method based on the desktop three-dimensional interactive all-in-one machine.

[0082] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device.

[0083] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned gesture interaction method based on the desktop three-dimensional interactive all-in-one machine.

[0084] The network interface 405 is used to communicate with other devices over the network. Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0085] The processor 402 is used to run the computer program 4032 stored in the memory to implement the above-mentioned gesture interaction method based on the desktop three-dimensional interactive all-in-one machine.

[0086] It should be understood that in the embodiment of the present invention, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0087] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer device to implement the process steps of the embodiment of the above method.

[0088] Therefore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the above-mentioned gesture interaction method based on a desktop three-dimensional interactive all-in-one machine.

[0089] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0090] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0091] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0092] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0094] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A gesture interaction method based on a desktop three-dimensional interactive all-in-one machine, applied to a desktop three-dimensional interactive all-in-one machine, characterized in that: The desktop three-dimensional interactive all-in-one machine includes a 3D display all-in-one machine, 3D glasses and a gesture recognition device, and the gesture recognition device is communicatively connected with the 3D display all-in-one machine; the gesture interaction method based on the desktop three-dimensional interactive all-in-one machine includes: The 3D display integrated machine responds to the rehabilitation training instruction, obtains the training target model corresponding to the rehabilitation training instruction and displays it in three dimensions in the rehabilitation training virtual scene, so that the user can view the training target model through 3D glasses; If the gesture recognition device detects and collects a set of hand images of the user, a user hand model is created based on the hand image set, and the user hand model is mapped to the rehabilitation training virtual scene; If the gesture recognition device detects the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene, the gesture recognition device recognizes the current operation gesture data to obtain a current operation recognition result; The 3D display integrated machine performs a model operation on the training target model based on the current operation recognition result to obtain a current target operation model; If the gesture recognition device detects the next operation gesture data, the next operation gesture data is used as the current operation gesture data, the current target operation model is used as the training target model, and the step of returning to execute if the gesture recognition device detects the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene, identifying the current operation gesture data to obtain the current operation recognition result is performed until the gesture recognition device stops detecting when it fails to detect the next operation gesture data or detects an operation end instruction.

2. The method according to claim 1, characterized in that If the gesture recognition device detects and collects a hand image set of the user, a user hand model is created based on the hand image set, and the user hand model is mapped to the rehabilitation training virtual scene, including: If it is detected that a user's hand enters the interaction area corresponding to the gesture recognition device, a preset hand image acquisition strategy is obtained; Based on the hand image acquisition strategy, images of the user's hands in the interaction area are acquired to obtain the hand image set; Based on a preset three-dimensional reconstruction strategy, a three-dimensional hand virtual model is reconstructed for the hand image set in the rehabilitation training virtual scene to obtain the user hand model, and the user hand model is mapped to the rehabilitation training virtual scene.

3. The method according to claim 2, characterized in that After the step of acquiring a preset hand image acquisition strategy if it is detected that a user hand enters the interaction area corresponding to the gesture recognition device, and before the step of acquiring an image of the user hand in the interaction area based on the hand image acquisition strategy to obtain the hand image set, the method further includes: If it is detected that the user's hand is captured and gesture recognized in the interaction area corresponding to the gesture recognition device and corresponds to the preset gesture recognition result, a hand image capture instruction is generated to trigger the execution of the step of capturing the image of the user's hand in the interaction area based on the hand image capture strategy to obtain the hand image set.

4. The method according to claim 2, characterized in that: The method of reconstructing a three-dimensional hand virtual model of the hand image set in the rehabilitation training virtual scene based on a preset three-dimensional reconstruction strategy to obtain the user hand model, and mapping the user hand model to the rehabilitation training virtual scene includes: Performing image preprocessing on each hand image in the hand image set based on a preset image preprocessing strategy to obtain a preprocessed hand image corresponding to each hand image, and forming a preprocessed hand image set; Based on a preset hand joint point recognition model, key hand nodes are identified for each preprocessed hand image in the preprocessed hand image set to obtain simplified hand node models corresponding to each preprocessed hand image; Based on the image information of each preprocessed hand image in the preprocessed hand image set and the corresponding simplified hand node model, the user hand model is determined, and the user hand model is mapped to the rehabilitation training virtual scene; wherein the user hand model is a three-dimensional hand virtual model corresponding to the hand image set and used to be displayed in the rehabilitation training virtual scene.

5. The method according to claim 4, characterized in that The determining the user hand model based on the image information of each preprocessed hand image in the preprocessed hand image set and the corresponding simplified hand node model, and mapping the user hand model to the rehabilitation training virtual scene, comprises: Based on the reference coordinate information corresponding to the interactive area; Obtaining image information of the i-th pre-processed hand image in the pre-processed hand image set; wherein the initial value of i is 1; Based on the image information of the i-th preprocessed hand image, the simplified hand node model corresponding to the i-th preprocessed hand image and the reference coordinate information, determine the i-th user hand sub-model corresponding to the simplified hand node model corresponding to the i-th preprocessed hand image in the rehabilitation training virtual scene; Increment i by 1 to update the value of i; If it is determined that i does not exceed N, then return to the step of obtaining the image information of the i-th pre-processed hand image in the pre-processed hand image set; wherein N represents the total number of pre-processed hand images included in the pre-processed hand image set; If it is determined that i exceeds N, any one of the first to Nth user hand sub-models is used as the user hand model, and the user hand model is mapped to the rehabilitation training virtual scene.

6. The method according to claim 1, characterized in that The performing a model operation on the training target model based on the current operation recognition result to obtain a current target operation model includes: Obtaining a current target operation sub-model targeted by the current operation recognition result in the training target model; The model position and state of the current target operation sub-model are updated based on the model operation corresponding to the current operation recognition result to update the training target model and obtain the current target operation model.

7. The method according to claim 1, characterized in that After the step of performing a model operation on the training target model based on the current operation recognition result to obtain a current target operation model, the method further includes: If the gesture recognition device does not detect the next operation gesture data or detects the operation end instruction, the complete operation process recording video for the training target model is obtained and saved to a local preset storage area.

8. A gesture interaction device based on a desktop three-dimensional interactive all-in-one machine, configured on the desktop three-dimensional interactive all-in-one machine, characterized in that: The desktop three-dimensional interactive all-in-one machine includes a 3D display all-in-one machine, 3D glasses and a gesture recognition device, and the gesture recognition device is communicatively connected with the 3D display all-in-one machine; The 3D display integrated machine is used to respond to the rehabilitation training instruction, obtain the training target model corresponding to the rehabilitation training instruction and display it in three dimensions in the rehabilitation training virtual scene, so that the user can view the training target model through 3D glasses; The gesture recognition device is used to create a user hand model based on the hand image set if a user hand image set is detected and collected, and map the user hand model to the rehabilitation training virtual scene; The gesture recognition device is further configured to recognize the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene to obtain a current operation recognition result if the current operation gesture data of the user for the training target model in the rehabilitation training virtual scene is detected; The 3D display integrated machine is further used to perform a model operation on the training target model based on the current operation recognition result to obtain a current target operation model; The gesture recognition device is also used to, if the next operation gesture data is detected, use the next operation gesture data as the current operation gesture data, use the current target operation model as the training target model, and return to execute the step of identifying the current operation gesture data if the gesture recognition device detects the user's current operation gesture data for the training target model in the rehabilitation training virtual scene to obtain the current operation recognition result, until the gesture recognition device stops detecting when it fails to detect the next operation gesture data or detects an operation end instruction.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the gesture interaction method based on the desktop three-dimensional interactive all-in-one machine as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the gesture interaction method based on a desktop three-dimensional interactive all-in-one machine as described in any one of claims 1 to 7 can be implemented.

Images