Gesture recognition interaction technology

Abstract

The invention provides a gesture recognition interaction technology, gesture actions are converted into system instructions by capturing and recognizing gestures for operating panel objects such as a sand table, and specific gesture instructions are interpreted and converted into operation signals of the sand table based on sand table instruction information interaction, so that the sand table operation is visually executed. Deviation is provided for gestures of information system manipulation. Meanwhile, according to the scheme, two different mapping logics of the corresponding relation between the specific gesture instruction and the sand table operation instruction signal are provided, fitting operation is provided for the gesture sequence, and continuous operation execution on the same operation object is facilitated. Moreover, gesture memory is carried out based on feedback evaluation, and the gesture is bound with a corresponding fitting instruction, so that an out-of-band gesture recognition strategy is provided, and repeated gesture recognition is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of new generation information technology, and in particular relates to a gesture recognition interaction technology. Background Art

[0002] In the process of information system operation, especially intuitive visual information systems such as sand tables and large electronic screens, commonly used operation methods include mouse clicking, keyboard operation, hand drawing, gesture operation, etc. Among them, gesture operation has received more and more attention due to its convenience, and related technologies have also flourished.

[0003] With the widespread application of computers, human-computer interaction (HCI) has become an important part of people's daily lives. The ultimate goal of human-computer interaction is to achieve natural communication between people and machines. The initial gesture recognition mainly used machine equipment to directly detect this gesture recognition research in line with the development needs. However, due to the angles and spatial positions of the joints of the hands and arms themselves, these devices are mostly connected to the computer system and the user through wired technology, so that the user's gesture information is transmitted to the recognition system completely and accurately. Typical devices include data gloves. Data gloves are composed of multiple sensor devices, through which information such as the position of the user's hand and the direction of the fingers can be transmitted to the computer system. Although data gloves can provide good detection effects, it is expensive to apply them in common fields.

[0004] Later, optical markers replaced data gloves. Wearing optical markers on the hands can transmit the changes of hand position and fingers to the system screen through infrared rays. This method can also provide good results, but still requires more complex equipment.

[0005] Although the intervention of external devices has improved the accuracy and stability of gesture recognition, it has obscured the natural expression of gestures. For this reason, visual gesture recognition has emerged. Visual gesture recognition refers to the recognition of gestures by processing image sequences containing gestures captured by video acquisition devices through computer vision technology. A professional research team has reviewed the research status and applications of visual gesture recognition from three aspects: gesture modeling, gesture analysis and gesture recognition.

[0006] Generally speaking, during gesture operation, the system needs to recognize the gesture, for example, through corresponding circuits or information collection to confirm the gesture. Gesture recognition is the process of classifying the trajectory (or point) in the model parameter space into a subset of the space, which includes static gesture recognition and dynamic gesture recognition. Dynamic gesture recognition can eventually be converted into static gesture recognition. From the perspective of the technical implementation of gesture recognition, common gesture recognition methods mainly include: template matching method, neural network method and hidden Markov model method.

[0007] Conventional gesture recognition methods include selecting image detection. For example, operators can upload corresponding gesture images to the system to carry out common gesture detection. The system will analyze the uploaded images, identify the common gestures included in them, and display information or recognize gesture results so that operators and information systems can clearly understand the corresponding operation methods of gestures.

[0008] However, current gesture recognition is only a simple gesture distinction, without considering the continuous association between gestures, and no in-depth research has been conducted on the adaptive learning improvement of gesture recognition accuracy. In the field of gesture recognition technology, the recognition accuracy and accuracy of gestures need to be strengthened, and there is insufficient research on the coordinated recognition of multiple gestures within a specific time. At the same time, in the gesture recognition process, the system often relies on a specific recognition model, which is not adequate for differentiated recognition needs and recognition needs in specific scenarios. There is also no effective expansion of customized processing and operation preferences for system operations, and there are still many deficiencies.

[0009] In order to solve the above-mentioned problems in the prior art, the present invention proposes a gesture recognition interaction technology, which realizes the conversion of gesture actions to system commands by capturing and identifying gestures for operating panel objects such as sandboxes, and interprets specific gesture commands based on sandbox command information interaction, and converts them into sandbox operation signals, thereby visually executing sandbox operations and providing deviations for gesture-based information system control. At the same time, the scheme of the present invention provides two different mapping logics for the correspondence between specific gesture commands and sandbox operation command signals, and provides fitting operations for gesture sequences, which facilitates the continuous operation of the same operation object. Moreover, the present invention also performs gesture memory based on feedback evaluation and binds it to the corresponding fitting instructions, thereby providing an out-of-band gesture recognition strategy to avoid repeated gesture recognition. Summary of the invention

[0010] The present invention aims to provide a gesture recognition interaction technology that is superior to the prior art.

[0011] In order to achieve the above object, the technical solution of the present invention is as follows: A gesture recognition interaction system, comprising: An operation panel object, which is an object panel for gesture operation, and is used to receive gesture operation instructions and present gesture operation results on the operation panel object; The operation panel object is the specific implementation and display panel of the gesture operation. The operation panel object connects the gesture capture and recognition module and the sandbox instruction information interaction module, and interacts with the feedback learning module. The operation panel object displays the gesture operations supported by the system, the operation interface, and the system changes and feedback after the operation.

[0012] Gesture capture and recognition module, which performs motion capture and command recognition on gesture operations; The gesture capture and recognition module is one of the main functional modules of the system, which is used to realize the action shooting and motion capture of each gesture in the gesture operation and convert it into corresponding gesture instructions through the system's built-in processing and send it to the sandbox instruction information interaction module.

[0013] Specifically, the gesture capture and recognition module can use a three-dimensional holographic camera to capture gestures, and keep the three-dimensional holographic camera shooting gestures for a period of time to form a three-dimensional spatial three-dimensional gesture image, and establish a spatial coordinate system based on the spatial three-dimensional gesture image, and provide a matching coordinate information structure containing the coordinates of each finger contact point and the coordinates of the palm midpoint for the gesture through the relative coordinate positions of each finger and the palm, so as to classify specific gestures. In the gesture recognition process, the coordinate information structure of the corresponding gesture is input into the recognition model, clustered by, for example, the K-means algorithm, and the specific instruction set to which the gesture belongs is recognized through the clustering result.

[0014] A sandbox instruction information interaction module, used to receive the specific gesture instruction recognized by the gesture capture and recognition module, convert it into sandbox instruction information, and perform sandbox interaction based on the instruction; The sandbox instruction information interaction module is connected to the gesture capture and recognition module, and is used to perform instruction conversion after receiving a specific gesture instruction recognized by the gesture capture and recognition module, to form an operation panel object operation instruction recognizable by the system, that is, sandbox instruction information, and run the instruction.

[0015] The feedback learning module is used to receive operation feedback of the command call, and perform adaptive learning of the command based on the feedback information to generate gesture preference settings.

[0016] The feedback learning module receives the feedback selected or filled in by the operator on the operation panel object or the information feedback component of other modules, and inputs the above feedback into the training process of the big data gesture recognition model to carry out adaptive AI training and learning after classification.

[0017] After multiple command executions, the feedback learning module receives information about the operator's satisfaction with the command execution results, and determines whether the recognition and execution of the command fully conform to the operator's command operation intention. If so, the feedback of the command is positive; if not or the deviation is large, it is negative feedback. Based on multiple learnings of positive and negative feedback, the gesture recognition accuracy is adjusted, and gesture preference settings exclusive to the operator are generated.

[0018] Preferably, the operation panel object is a sandbox.

[0019] Preferably, the gesture capture and recognition module at least includes: A gesture image capturing submodule, wherein the gesture image capturing submodule uses a three-dimensional holographic camera to capture gesture actions ACThand; The gesture recognition submodule uses a big data gesture recognition model based on the gesture action ACThand to identify the gesture action ACThand, and recognizes the gesture operation as a specific gesture instruction according to the action identification result.

[0020] Preferably, the sandbox instruction information interaction module includes at least an instruction conversion submodule, an instruction fitting submodule, and an instruction calling submodule, wherein: The command conversion submodule converts the gesture command conversion mapping relationship set into the corresponding sandbox operation command signal; The instruction fitting submodule receives a plurality of specific gesture instructions within a first time period; performs instruction fitting based on a continuity algorithm between the specific gesture instructions to obtain a fitting instruction; The instruction calling submodule receives the fitting instruction, and calls the corresponding operation function of the operation panel object based on the fitting instruction, executes the sandbox operation and presents it visually.

[0021] Preferably, the feedback learning module at least includes: The instruction feedback submodule is used to receive feedback and evaluation from the operator on the visualization of the sandbox operation and determine the operation fidelity, specifically: the instruction conversion and instruction fitting process with correct feedback results is entered into the big data gesture recognition model AI adaptive learning positive training database; the instruction conversion and instruction fitting process with incorrect feedback results is entered into the big data gesture recognition model AI adaptive learning negative training database; Adaptive AI learning submodule, which performs adaptive AI training and learning of big data gesture recognition models based on positive training database and negative training database; The gesture preference submodule directly memorizes gesture operations that are repeated a large number of times and evaluated as correct by feedback, binds them to the corresponding fitting instructions, and provides an out-of-band gesture recognition strategy to avoid repeated recognition of preferred gestures.

[0022] Preferably, the big data gesture recognition model is used to perform gesture recognition based on a gesture three-dimensional image recognition pre-training model, perform clustering recognition processing on the gesture action ACThand, and output a recognition result that is recognized as a specific gesture instruction according to the action clustering result of the clustering algorithm; The big data gesture recognition model also performs adaptive AI training and learning based on a positive training database and a negative training database, thereby improving recognition accuracy.

[0023] Preferably, the gesture instruction conversion mapping relationship set records the correspondence between specific gesture instructions preset by the system and sandbox operation instruction signals, and based on the correspondence, the specific gesture instructions are converted into sandbox operation instruction signals.

[0024] Preferably, in addition to recording the correspondence between specific gesture commands preset by the system and sandbox operation command signals, the gesture command conversion mapping relationship set also stores a custom gesture command setting module, which is used to copy the conversion mapping relationship of specific gesture commands to custom sandbox operation command signals.

[0025] At the same time, the present invention also proposes a gesture recognition interaction device, including a memory and a processor, and the device is used to execute the corresponding functions of the gesture recognition interaction technology as described above through the processor.

[0026] At the same time, the present invention also proposes a computer-readable storage medium, which stores a program executable by a processor, characterized in that the program executable by the processor is used to perform the corresponding functions of the gesture recognition interaction technology described above when executed by the processor.

[0027] At the same time, the present invention also proposes a computer program product, which includes computer instructions, and the computer instructions execute corresponding functions of the gesture recognition interaction technology as described above when executed by a processor.

[0028] The present invention proposes a gesture recognition interaction technology, which realizes the conversion of gesture actions to system commands by capturing and identifying gestures of operating panel objects such as sandboxes, and interprets specific gesture commands based on sandbox command information interaction and converts them into sandbox operation signals, thereby visually executing sandbox operations and providing deviations for gesture-based information system control. At the same time, the scheme of the present invention provides two different mapping logics for the correspondence between specific gesture commands and sandbox operation command signals, and provides fitting operations for gesture sequences, which facilitates the continuous operation of the same operation object. Moreover, the present invention also performs gesture memory based on feedback evaluation and binds it to the corresponding fitting instructions, thereby providing an out-of-band gesture recognition strategy to avoid repeated gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a basic system structure diagram of the gesture recognition interaction technology shown in the present invention; Figure 2 is a schematic diagram of one of the embodiments of the gesture recognition interaction technology and its specific gesture capture and recognition module shown in the present invention; Figure 3 It is a schematic diagram of one embodiment of a sandbox instruction information interaction module of a gesture recognition interaction technology shown in the present invention; Figure 4 It is one of the embodiments of the gesture recognition interaction technology feedback learning module claimed in the present invention; Figure 5 It is a schematic diagram of a preferred embodiment of the hand gesture recognition interaction technology instruction processing chain flow chart shown in the present invention. DETAILED DESCRIPTION

[0030] Several embodiments and beneficial effects of the gesture recognition interaction technology claimed in the present invention are described in detail below to facilitate a more detailed review and analysis of the present invention.

[0031] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0032] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0034] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0035] It should be understood that although the terms first, second, etc. may be used to describe the method and corresponding device in the embodiments of the present invention, these keywords should not be limited to these terms. These terms are only used to distinguish keywords from each other. For example, without departing from the scope of the embodiments of the present invention, a template, etc. may also be referred to as a target plate.

[0036] It should be understood that although the terms first, second, etc. may be used to describe the method and corresponding device in the embodiments of the present invention, these keywords should not be limited to these terms. These terms are only used to distinguish the keywords from each other. For example, without departing from the scope of the embodiments of the present invention, the first time period, etc. may also be referred to as the second time period.

[0037] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0038] As the instruction manual Figure 1-5 As shown, it is one of the embodiments of the gesture recognition interaction technology and its specific included modules claimed by the present invention. The gesture recognition interaction technology claimed by the present invention includes: An operation panel object, wherein the operation panel object is an object panel for gesture operation, and is used to receive gesture operation instructions and present gesture operation results on the operation panel object; The operation panel object is the specific implementation and display panel of the gesture operation. The operation panel object connects the gesture capture and recognition module and the sandbox instruction information interaction module, and interacts with the feedback learning module. The operation panel object displays the gesture operations supported by the system, the operation interface, and the system changes and feedback after the operation.

[0039] Gesture capture and recognition module, which performs motion capture and command recognition on gesture operations; The gesture capture and recognition module is one of the main functional modules of the system, which is used to realize the action shooting and motion capture of each gesture in the gesture operation and convert it into corresponding gesture instructions through the system's built-in processing and send it to the sandbox instruction information interaction module.

[0040] Specifically, the gesture capture and recognition module can use a three-dimensional holographic camera to capture gestures, and keep the three-dimensional holographic camera shooting gestures for a period of time to form a three-dimensional spatial three-dimensional gesture image, and establish a spatial coordinate system based on the spatial three-dimensional gesture image, and provide a matching coordinate information structure containing the coordinates of each finger contact point and the coordinates of the palm midpoint for the gesture through the relative coordinate positions of each finger and the palm, so as to classify specific gestures. In the gesture recognition process, the coordinate information structure of the corresponding gesture is input into the recognition model, clustered by, for example, the K-means algorithm, and the specific instruction set to which the gesture belongs is recognized through the clustering result.

[0041] A sandbox instruction information interaction module, used to receive the specific gesture instruction recognized by the gesture capture and recognition module, convert it into sandbox instruction information, and perform sandbox interaction based on the instruction; The sandbox instruction information interaction module is connected to the gesture capture and recognition module, and is used to perform instruction conversion after receiving a specific gesture instruction recognized by the gesture capture and recognition module, to form an operation panel object operation instruction recognizable by the system, that is, sandbox instruction information, and run the instruction.

[0042] The feedback learning module is used to receive operation feedback of the command call, and perform adaptive learning of the command based on the feedback information to generate gesture preference settings.

[0043] The feedback learning module receives the feedback selected or filled in by the operator on the operation panel object or the information feedback component of other modules, and inputs the above feedback into the training process of the big data gesture recognition model to carry out adaptive AI training and learning after classification.

[0044] After multiple command executions, the feedback learning module receives information about the operator's satisfaction with the command execution results, and determines whether the recognition and execution of the command fully conform to the operator's command operation intention. If so, the feedback of the command is positive; if not or the deviation is large, it is negative feedback. Based on multiple learnings of positive and negative feedback, the gesture recognition accuracy is adjusted, and gesture preference settings exclusive to the operator are generated.

[0045] As a superimposable preferred embodiment, the operation panel object is a sandbox.

[0046] As a superimposable preferred embodiment, the gesture capture and recognition module at least includes: A gesture image capturing submodule, wherein the gesture image capturing submodule uses a three-dimensional holographic camera to capture gesture actions ACThand; The gesture recognition submodule uses a big data gesture recognition model based on the gesture action ACThand to identify the gesture action ACThand, and recognizes the gesture operation as a specific gesture instruction according to the action identification result.

[0047] As a superimposable preferred embodiment, the sandbox instruction information interaction module at least includes an instruction conversion submodule, an instruction fitting submodule, and an instruction calling submodule, wherein: The command conversion submodule converts the gesture command conversion mapping relationship set into the corresponding sandbox operation command signal; As a superimposable preferred embodiment, the command conversion submodule converts the gesture command conversion mapping relationship set to the corresponding sandbox operation command signal based on the gesture command conversion mapping relationship set, specifically: the gesture command conversion mapping relationship set records the correspondence between the specific gesture commands preset by the system and the sandbox operation command signals, and based on the correspondence, converts the specific gesture commands into sandbox operation command signals. For example, the gesture command conversion mapping relationship set records that the swing gesture - waving twice corresponds to the sandbox operation command signal "remove sandbox tag point"; the tap gesture - 1 time corresponds to the sandbox operation command signal "add sandbox tag point" and so on, which will not be repeated here. At the same time, in addition to recording the correspondence between the specific gesture commands preset by the system and the sandbox operation command signals, the gesture command conversion mapping relationship set also stores a custom gesture command setting module, which is used to copy the conversion mapping relationship of the specific gesture command to the custom sandbox operation command signal. For example, an operator copies the "add sandbox tag point" from a tap gesture-1 time to a click gesture-double-click, which provides an out-of-band gesture operation method for "adding sandbox tag points". During the operation of this specific person, a specific method of click gesture-double-click is provided for "adding sandbox tag points".

[0048] The instruction fitting submodule receives a plurality of specific gesture instructions within a first time period; performs instruction fitting based on a continuity algorithm between the specific gesture instructions to obtain fitting instructions.

[0049] As a superimposable preferred embodiment, the instruction fitting submodule receives multiple specific gesture instructions within a first time period; based on the continuity algorithm between specific gesture instructions, executes instruction fitting to obtain fitting instructions, specifically: sets the gesture operation interval to be longer than a specific time length as the fitting gesture interruption point, and the previous and subsequent points are different fitting gestures. For example, the operator performs 5 different gesture operations in succession and interrupts for 3S. If the gesture operation interval is 2S, then the other gesture operations after the interruption are determined to be the next group rather than the current group of gestures to be fitted; based on the continuity of the gesture, determine whether the gesture to be fitted is a continuous gesture for the same operation object on the sandbox through a continuity algorithm. If so, define the fitting gesture as a continuous operation for the same operation object on the sandbox to obtain a fitting instruction; if not, only process the first gesture in the gesture to be fitted.

[0050] Among them, based on the continuity of the gesture, a continuity algorithm is used to determine whether the fitting gesture is a continuous gesture for the same operation object on the sandbox, specifically: based on the pause time threshold between continuous gestures, determine whether it is a continuous gesture.

[0051] The instruction calling submodule receives the fitting instruction, and calls the corresponding operation function of the operation panel object based on the fitting instruction, executes the sandbox operation and presents it visually.

[0052] As a superimposable preferred embodiment, the feedback learning module at least includes: The instruction feedback submodule is used to receive feedback and evaluation from the operator on the visualization of the sandbox operation and determine the operation fidelity, specifically: the instruction conversion and instruction fitting process with correct feedback results is entered into the big data gesture recognition model AI adaptive learning positive training database; the instruction conversion and instruction fitting process with incorrect feedback results is entered into the big data gesture recognition model AI adaptive learning negative training database; Adaptive AI learning submodule, which performs adaptive AI training and learning of big data gesture recognition models based on positive training database and negative training database; The gesture preference submodule directly memorizes gesture operations that are repeated a large number of times and evaluated as correct by feedback, binds them to the corresponding fitting instructions, and provides an out-of-band gesture recognition strategy to avoid repeated recognition of preferred gestures.

[0053] As a superimposable preferred embodiment, the big data gesture recognition model is used to perform gesture recognition based on a gesture three-dimensional image recognition pre-training model, perform clustering recognition processing on the gesture action ACThand, and output a recognition result that is recognized as a specific gesture instruction based on the action clustering result of the clustering algorithm; The big data gesture recognition model also performs adaptive AI training and learning based on a positive training database and a negative training database, thereby improving recognition accuracy.

[0054] As a superimposable preferred embodiment, the gesture instruction conversion mapping relationship set records the correspondence between specific gesture instructions preset by the system and sandbox operation instruction signals, and based on the correspondence, converts the specific gesture instructions into sandbox operation instruction signals.

[0055] As a superimposable preferred embodiment, the gesture command conversion mapping relationship set not only records the correspondence between the specific gesture commands preset by the system and the sandbox operation command signals, but also saves a custom gesture command setting module, and the custom gesture command setting module is used to copy the conversion mapping relationship of the specific gesture commands to the custom sandbox operation command signals.

[0056] At the same time, the present invention also proposes a gesture recognition interaction device, including a memory and a processor, and the device is used to execute the corresponding functions of the gesture recognition interaction technology as described above through the processor.

[0057] At the same time, the present invention also proposes a computer-readable storage medium, which stores a program executable by a processor, characterized in that the program executable by the processor is used to perform the corresponding functions of the gesture recognition interaction technology described above when executed by the processor.

[0058] At the same time, the present invention also proposes a computer program product, which includes computer instructions, and the computer instructions execute corresponding functions of the gesture recognition interaction technology as described above when executed by a processor.

[0059] The present invention proposes a gesture recognition interaction technology, which realizes the conversion of gesture actions to system commands by capturing and identifying gestures of operating panel objects such as sandboxes, and interprets specific gesture commands based on sandbox command information interaction and converts them into sandbox operation signals, thereby visually executing sandbox operations and providing deviations for gesture-based information system control. At the same time, the scheme of the present invention provides two different mapping logics for the correspondence between specific gesture commands and sandbox operation command signals, and provides fitting operations for gesture sequences, which facilitates the continuous operation of the same operation object. Moreover, the present invention also performs gesture memory based on feedback evaluation and binds it to the corresponding fitting instructions, thereby providing an out-of-band gesture recognition strategy to avoid repeated gesture recognition.

[0060] In all the above embodiments, in order to realize some special data transmission and read / write function requirements, the above method operation process and its corresponding device can add devices, modules, components, hardware, pin connections or memory, processor differences to expand functions.

[0061] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the methods, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0062] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the method steps is only a logical or functional division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0063] The units described as the various steps of the method and the separated components of the device may or may not be logically or physically separated, and may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0064] In addition, the method steps and their implementations and functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0065] The above method and device can be implemented as an integrated unit in the form of a software functional unit, which can be stored in a computer-readable storage medium. The above software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), NVRAM, disk or optical disk, etc., which can store program code.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0067] It should be noted that the above embodiments are only used to more clearly explain and illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gesture recognition interaction technology, comprising: An operation panel object, wherein the operation panel object is an object panel for gesture operation, and is used to receive gesture operation instructions and present gesture operation results on the operation panel object; Gesture capture and recognition module, which performs motion capture and command recognition on gesture operations; A sandbox instruction information interaction module, used to receive the specific gesture instruction recognized by the gesture capture and recognition module, convert it into sandbox instruction information, and perform sandbox interaction based on the instruction; The feedback learning module is used to receive operation feedback of the command call, and perform adaptive learning of the command based on the feedback information to generate gesture preference settings.

2. The gesture recognition interaction technology as claimed in claim 1, characterized in that: The operation panel object is a sandbox.

3. The gesture recognition interaction technology as claimed in claim 2, characterized in that: The gesture capture and recognition module at least includes: A gesture image capturing submodule, wherein the gesture image capturing submodule uses a three-dimensional holographic camera to capture gesture actions ACThand; The gesture recognition submodule uses a big data gesture recognition model based on the gesture action ACThand to identify the gesture action ACThand, and identifies the gesture operation as a specific gesture instruction according to the action identification result.

4. The gesture recognition interaction technology as claimed in claim 2, characterized in that: The sandbox instruction information interaction module at least includes an instruction conversion submodule, an instruction fitting submodule, and an instruction calling submodule, wherein: The command conversion submodule converts the gesture command conversion mapping relationship set into the corresponding sandbox operation command signal; The instruction fitting submodule receives a plurality of specific gesture instructions within a first time period; performs instruction fitting based on a continuity algorithm between the specific gesture instructions to obtain a fitting instruction; The instruction calling submodule receives the fitting instruction, and calls the corresponding operation function of the operation panel object based on the fitting instruction, executes the sandbox operation and presents it visually.

5. The gesture recognition interaction technology as claimed in claim 2, characterized in that: The feedback learning module at least includes: The instruction feedback submodule is used to receive feedback and evaluation from the operator on the visualization of the sandbox operation and determine the operation fidelity, specifically: the instruction conversion and instruction fitting process with correct feedback results is entered into the big data gesture recognition model AI adaptive learning positive training database; the instruction conversion and instruction fitting process with incorrect feedback results is entered into the big data gesture recognition model AI adaptive learning negative training database; Adaptive AI learning submodule, which performs adaptive AI training and learning of big data gesture recognition models based on positive training database and negative training database; The gesture preference submodule directly memorizes gesture operations that are repeated a large number of times and evaluated as correct by feedback, binds them to the corresponding fitting instructions, and provides an out-of-band gesture recognition strategy to avoid repeated recognition of preferred gestures.

6. The gesture recognition interaction technology as claimed in claim 5, characterized in that: The big data gesture recognition model is used to perform gesture recognition based on the gesture three-dimensional image recognition pre-training model, perform clustering recognition processing on the gesture action ACThand, and output the recognition result of the specific gesture instruction according to the action clustering result of the clustering algorithm; The big data gesture recognition model also performs adaptive AI training and learning based on a positive training database and a negative training database, thereby improving recognition accuracy.

7. The gesture recognition interaction technology as claimed in claim 6, characterized in that: The gesture instruction conversion mapping relationship set records the correspondence between specific gesture instructions preset by the system and sandbox operation instruction signals, and based on the correspondence, converts the specific gesture instructions into sandbox operation instruction signals.

8. The gesture recognition interaction technology as claimed in claim 7, characterized in that: In addition to recording the correspondence between the specific gesture commands preset by the system and the sandbox operation command signals, the gesture command conversion mapping relationship set also stores a custom gesture command setting module, which is used to copy the conversion mapping relationship of the specific gesture commands to the custom sandbox operation command signals.

9. A gesture recognition interaction device, comprising a memory and a processor, wherein the device is used to execute a function corresponding to the gesture recognition interaction technology according to any one of claims 1 to 8 through the processor.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to perform the corresponding function of the gesture recognition interaction technology as described in any one of claims 1-8 when executed by the processor.

Images