Gesture interaction method and system applied to large scene and computer program product

Abstract

The invention discloses a gesture interaction method and system applied to a large scene and a computer program product, and the method comprises the steps: obtaining a three-dimensional coordinate of a user viewpoint through a positioning module, building a viewpoint coordinate system, determining a three-dimensional coordinate of the user viewpoint in an actual physical scene through a viewpoint tracking module, and carrying out the recognition of the three-dimensional coordinate of the user viewpoint in the actual physical scene; then completing mapping of each hand skeleton node in the virtual scene in a mapping interaction module so as to create a virtual hand in the virtual scene, and updating the virtual hand to obtain gesture information of the virtual hand; and finally, determining an interaction response instruction corresponding to the virtual hand gesture information in the display feedback module, and executing the interaction response instruction to realize man-machine interaction. According to the gesture interaction method, the user can freely move during interaction, the degree of freedom of gesture interaction in a large scene is improved, and the position data of the hand skeleton nodes of the user are accurately obtained through multiple coordinate system conversion, so that intuitive bare-hand interaction is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of gesture recognition and human-computer interaction, and in particular relates to a gesture interaction method, system and computer program product applied to large scenes. Background Art

[0002] In the current digital information age, with the increase and enrichment of content information types and expression styles, display terminals have developed from handheld terminal screens and desktop display terminal screens to wall-mounted LED screens, and even LED screens covering the entire facade of a building. In cutting-edge applications such as virtual reality, in order to obtain better visual presentation effects, higher requirements are usually placed on the size, resolution, and viewing angle range of display devices. With the application trend of larger and larger display screens and higher and higher resolutions, the traditional mouse interaction has become increasingly difficult to meet the user's application needs for ultra-large screens. This is because one of the prerequisites for the traditional mouse to work is that the user needs to accurately locate the position of the mouse pointer, and it is relatively difficult to find and track the mouse coordinates through the human eye on a large screen. In addition, changing the observation viewpoint position in a large physical space scene (large scene) is also extremely inconvenient for the observer user holding the mouse. Therefore, in a large scene, a new interaction method is needed to replace the traditional mouse for human-computer interaction.

[0003] However, various applications in large scenarios, whether used for content expression for integrated media or for experience creation for users, face two core problems: First, in human-computer interaction in large scenarios, users' activities are often severely restricted, and they can only stay in a specific position or move within a small range. This makes it impossible for users to move their observation points (viewpoints) freely when interacting. Moreover, it is difficult to realize the concept of "what you see is what you interact" in interactions in large scenarios, which makes it difficult to achieve intuitive bare-hand interaction within the user's line of sight, which is also a major problem.

[0004] Therefore, there is an urgent need to provide a gesture interaction method, system and computer program product that can be applied in large scenes to ensure that interaction can still be performed when the observation point moves freely and can perform intuitive bare-hand interaction. Summary of the invention

[0005] The purpose of the present invention is to provide a gesture interaction method, system and computer program product applied to large scenes, so as to solve the above-mentioned problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a gesture interaction method applied to a large scene, comprising:

[0008] Acquire the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and establish a viewpoint coordinate system based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene, wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system, and the first coordinate origin of the viewpoint coordinate system is the three-dimensional coordinates of the user's viewpoint in the actual physical scene;

[0009] Based on the viewpoint coordinate system, calibrate the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system;

[0010] Based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system, the three-dimensional coordinates of each hand bone node of the user in the actual physical scene are calibrated;

[0011] Mapping the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene to create a virtual hand in the virtual scene, wherein the virtual hand includes a plurality of virtual bone nodes, and each virtual bone node corresponds to each hand bone node one by one;

[0012] Acquire displacement information of the user's viewpoint and motion information of each hand bone node of the user, and update each virtual bone node in the virtual hand according to the displacement information of the user's viewpoint and the motion information of each hand bone node of the user, so as to obtain virtual hand gesture information based on each updated virtual bone node;

[0013] Based on the preset designated gesture information, an interactive response instruction corresponding to the virtual hand gesture information is determined, and the interactive response instruction is executed to realize human-computer interaction.

[0014] In a possible design, an interactive display terminal is provided in the actual physical scene, and a first target camera is provided at the interactive display terminal;

[0015] Among them, obtaining the three-dimensional coordinates of the user's viewpoint in the actual physical scene includes:

[0016] Acquire the position information of the first target camera, and determine the position of the first target camera as the second coordinate origin according to the position information of the first target camera;

[0017] According to the second coordinate origin, the actual scene coordinate system is established;

[0018] The position information of the user's viewpoint is obtained, and the position information of the user's viewpoint is mapped into the actual scene coordinate system to determine the three-dimensional coordinates of the user's viewpoint position in the actual physical scene.

[0019] In a possible design, based on the viewpoint coordinate system, the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system are calibrated, including:

[0020] Acquire user's hand image in real time;

[0021] Based on the user's hand image, the user's hand bone node data is calculated based on the YOLOv8 algorithm;

[0022] Based on the data of each hand bone node of the user, the viewpoint coordinates of each hand bone node of the user are calibrated in the viewpoint coordinate system.

[0023] In a possible design, the user performs gesture interaction based on interactive glasses, wherein the interactive glasses are provided with a second target camera facing the user's viewpoint;

[0024] Among them, based on the data of each hand bone node of the user, the viewpoint coordinates of each hand bone node of the user are calibrated in the viewpoint coordinate system, including:

[0025] From the user's hand bone node data, select any hand bone node as the third coordinate origin;

[0026] According to the third coordinate origin, a hand tracking coordinate system is established, wherein the hand tracking coordinate system selects the direction from the back of the hand to the palm as the positive direction of the Z axis, the direction from the palm to the finger as the positive direction of the X axis, and the direction from the palm to the index finger as the positive direction of the Y axis;

[0027] Based on the data of each hand skeleton node of the user, the hand tracking coordinates of each hand skeleton node of the user are calibrated in the hand tracking coordinate system;

[0028] Get the intrinsic parameter matrix of the second target camera;

[0029] Establishing a first rigid body transformation matrix, wherein the first rigid body transformation matrix is ​​used to represent the transformation relationship between the hand tracking coordinate system and the viewpoint coordinate system;

[0030] Based on the intrinsic parameter matrix of the second target camera and the first rigid body transformation matrix, the hand tracking coordinates of each hand bone node of the user in the hand tracking coordinate system are converted into the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system.

[0031] In a possible design, an interactive display terminal is provided in the actual physical scene, and a first target camera is provided at the interactive display terminal;

[0032] Among them, based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system, the three-dimensional coordinates of each hand bone node of the user in the actual physical scene are calibrated, including:

[0033] Get the intrinsic parameter matrix of the first target camera;

[0034] Based on the viewpoint coordinate system, establishing a second rigid body transformation matrix, wherein the second rigid body transformation matrix is ​​used to represent the conversion relationship between the viewpoint coordinate system and the actual scene coordinate system;

[0035] Based on the intrinsic parameter matrix of the first target camera and the second rigid body transformation matrix, the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system are converted into the three-dimensional coordinates of each hand bone node of the user in the actual physical scene.

[0036] In a possible design, the three-dimensional coordinates of each hand bone node of the user in the actual physical scene are mapped to the virtual scene to create a virtual hand in the virtual scene, including:

[0037] Establish a virtual scene coordinate system;

[0038] Mapping the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene coordinate system to form the virtual scene coordinates of each hand bone node of the user;

[0039] Using the virtual scene coordinates of each hand bone node of the user, each virtual bone node is created in the virtual scene, wherein the virtual scene coordinates of each hand bone node of the user correspond one-to-one to the coordinate position of each virtual bone node in the virtual scene coordinate system;

[0040] A virtual hand is created in the virtual scene according to each hand bone node.

[0041] In a possible design, each virtual bone node is updated in real time according to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, and virtual hand gesture information is obtained through each virtual bone node in the virtual scene, including:

[0042] Obtain the displacement information of the user's viewpoint and the motion information of each hand bone node of the user in real time;

[0043] According to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the three-dimensional coordinates of each hand bone node of the user in the actual physical scene after real-time update are obtained;

[0044] Based on the real-time updated three-dimensional coordinates of the user's viewpoint in the actual physical scene and the real-time updated three-dimensional coordinates of each hand bone node of the user in the actual physical scene, a real-time updated virtual hand is obtained in the virtual scene;

[0045] According to each virtual skeleton node in the virtual hand updated in real time, the virtual hand gesture information is obtained.

[0046] In a possible design, based on preset designated gesture information, determining an interactive response instruction corresponding to the virtual hand gesture information, and executing the interactive response instruction to achieve human-computer interaction, includes:

[0047] Acquire a designated gesture information library, wherein the designated gesture information library includes a plurality of designated gesture information, and each designated gesture information corresponds to an interactive response instruction;

[0048] Based on the virtual hand gesture information, the designated gesture information corresponding to the virtual hand gesture information is filtered out from the designated gesture information library;

[0049] Execute the interactive response instruction corresponding to the designated gesture information corresponding to the virtual hand gesture information to realize human-computer interaction.

[0050] In a second aspect, the present invention provides a gesture interaction system applied to a large scene, which is used to implement the gesture interaction method applied to a large scene as described in the first aspect or any possible design of the first aspect, and is characterized by comprising:

[0051] A positioning module, used to obtain the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and use the three-dimensional coordinates of the user's viewpoint in the actual physical scene as the first coordinate origin to establish a viewpoint coordinate system, wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system;

[0052] A viewpoint tracking module, used to obtain the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and to establish a viewpoint coordinate system based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and also used to calibrate the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system based on the viewpoint coordinate system; wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system, and the first coordinate origin of the viewpoint coordinate system is the three-dimensional coordinates of the user's viewpoint in the actual physical scene;

[0053] A mapping interaction module, used to map the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene, so as to create a virtual hand in the virtual scene, obtain the displacement information of the user's viewpoint and the action information of each hand bone node of the user, and update each virtual bone node in the virtual hand according to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, so as to obtain virtual hand gesture information based on each updated virtual bone node; wherein the virtual hand includes a plurality of virtual bone nodes, and each virtual bone node corresponds to each hand bone node one by one;

[0054] The display feedback module is used to determine the interactive response instruction corresponding to the virtual hand gesture information based on the preset designated gesture information, and execute the interactive response instruction to achieve human-computer interaction.

[0055] In a third aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the gesture interaction method applied to large scenes as described in the first aspect or any possible design of the first aspect.

[0056] Beneficial effects: The present invention provides a gesture interaction method, system and computer program product for large scenes, wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are first obtained, and the three-dimensional coordinates of the user's viewpoint in the actual physical scene are used as the first coordinate origin to establish a viewpoint coordinate system, and then the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system are calibrated, and the three-dimensional coordinates in the actual physical scene are determined, and then the mapping of each hand bone node of the user in the virtual scene is completed to create a virtual hand in the virtual scene, and the virtual hand is updated to obtain virtual hand gesture information; finally, based on the preset specified gesture information, the interactive response instruction corresponding to the virtual hand gesture information is determined, and the interactive response instruction is executed to realize human-computer interaction. This gesture interaction method allows the user to move freely in the actual physical scene when interacting, which improves the user's freedom of gesture interaction in a large scene; and through multiple determinations and conversions of the coordinate system and coordinate positions, the position data of the user's hand bone nodes can be accurately obtained, which is convenient for the user to perform intuitive bare-hand interaction and realize the concept of "what you see is interaction". BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of the gesture interaction method applied to a large scene in Example 1;

[0058] Figure 2 This is a functional structure diagram of the gesture interaction system applied to large scenes in Example 2. DETAILED DESCRIPTION

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0060] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.

[0061] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0062] Example:

[0063] like Figure 1 As shown, this embodiment provides a gesture interaction method applied to a large scene, including the following steps:

[0064] S100. Obtain the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and use the three-dimensional coordinates of the user's viewpoint in the actual physical scene as the first coordinate origin to establish a viewpoint coordinate system, wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system;

[0065] Specifically, in a possible design, in step S100, an interactive display terminal is provided in the actual physical scene, and a first target camera is provided at the interactive display terminal;

[0066] Among them, obtaining the three-dimensional coordinates of the user's viewpoint in the actual physical scene includes:

[0067] S1001. Obtaining position information of a first target camera, and determining the position of the first target camera as a second coordinate origin according to the position information of the first target camera;

[0068] S1002. Establishing an actual scene coordinate system according to the second coordinate origin;

[0069] S1003. Obtain the position information of the user's viewpoint, and map the position information of the user's viewpoint to the actual scene coordinate system to determine the three-dimensional coordinates of the user's viewpoint in the actual physical scene.

[0070] Among them, it should be noted that the first target camera is set at the display terminal to obtain the user's viewpoint position in the large scene; and the actual scene coordinate system is a three-dimensional coordinate system, which can also be considered as an actual world coordinate system, which selects the position of the display terminal (first target camera) as the coordinate origin (second coordinate origin), and the direction from bottom to top is the positive direction of the Z axis, and the X-axis and Y-axis selection can be targeted according to the actual usage scenario; and the viewpoint coordinate system selects the position of the user's viewpoint as the coordinate origin (first coordinate origin), the direction of the user's line of sight is the positive direction of the Z axis, the direction from left to right is the positive direction of the X axis, and the direction from bottom to top is the positive direction of the Y axis.

[0071] S200. Based on the viewpoint coordinate system, calibrate the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system;

[0072] Specifically, in a possible design, in step S200, based on the viewpoint coordinate system, the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system are calibrated, including:

[0073] S2001. Real-time acquisition of user hand image;

[0074] S2002. Based on the user's hand image, calculate the user's hand bone node data based on the YOLOv8 algorithm;

[0075] S2003. Based on the data of each hand bone node of the user, calibrate the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system.

[0076] Among them, the YOLOv8 algorithm is the latest version of the YOLO (You Only Look Once) series of real-time target detection algorithms. YOLOv8 uses a deep convolutional neural network (such as Darknet-53) as the backbone network to extract features from the input image. This neural network contains multiple convolutional layers and pooling layers, which can effectively extract key information from the image; and based on the YOLOv8 algorithm's more lightweight network architecture, it can significantly improve its reasoning speed by reducing redundant calculations and optimizing the network structure. This enables the YOLOv8 algorithm to achieve real-time detection with lower computing resources, which ensures the real-time and continuity of the interaction when used in real-time gesture interaction; and after training, the YOLOv8 algorithm can accurately identify 21 3D hand key points (hand bone nodes) in the detected hand area.

[0077] It should be noted that the YOLOv8 algorithm is open source (Ultralytics opened it on January 10, 2023), and the YOLOv8 code is also easy to understand and use, and detailed documentation and sample codes are provided in its official database. Therefore, a person of ordinary skill in the art can obtain and use the YOLOv8 algorithm without creative labor, and use it to construct the gesture interaction method in this embodiment.

[0078] Specifically, in a possible design, in step S2003, the user performs gesture interaction based on the interactive glasses, wherein the interactive glasses are provided with a second target camera facing the user's viewpoint;

[0079] Among them, based on the data of each hand bone node of the user, the viewpoint coordinates of each hand bone node of the user are calibrated in the viewpoint coordinate system, including:

[0080] S20031. Select any hand bone node from the user's hand bone node data as the third coordinate origin;

[0081] S20032. According to the third coordinate origin, a hand tracking coordinate system is established, wherein the hand tracking coordinate system selects the direction from the back of the hand to the palm as the positive direction of the Z axis, the direction from the palm to the finger as the positive direction of the X axis, and the direction from the palm to the index finger as the positive direction of the Y axis;

[0082] S20033. Based on the data of each hand skeleton node of the user, the hand tracking coordinates of each hand skeleton node of the user are calibrated in the hand tracking coordinate system;

[0083] S20034. Obtain the intrinsic parameter matrix of the second target camera;

[0084] S20035. Establish a first rigid body transformation matrix, wherein the first rigid body transformation matrix is ​​used to represent the conversion relationship between the hand tracking coordinate system and the viewpoint coordinate system;

[0085] S20036. Based on the intrinsic parameter matrix of the second target camera and the first rigid body transformation matrix, the hand tracking coordinates of each hand bone node of the user in the hand tracking coordinate system are converted into the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system.

[0086] Wherein, the hand tracking coordinate system is a three-dimensional coordinate system, and the second target camera is located at the user's viewpoint, and is used to obtain the precise position of each hand bone node under the user's viewpoint. The internal parameter matrix of the second target camera mentioned here is a 3x3 matrix for describing the internal parameters of the camera, and the first rigid body transformation matrix is ​​used to represent the conversion relationship between the hand tracking coordinate system and the viewpoint coordinate system, which includes a rotation matrix and a displacement vector. The internal parameter matrix of the second target camera and the first rigid body transformation matrix are introduced to calculate the hand tracking coordinate system, which can be converted into the viewpoint coordinate system, and the hand tracking coordinates of each hand bone node of the user in the hand tracking coordinate system are converted into the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system. In this design of the present embodiment, a new coordinate system, the hand tracking coordinate system, is introduced again, so as to be able to more accurately identify and represent the position of each hand bone node, so that when performing real-time updating, the action information of each hand bone node can be more accurately obtained to ensure the accuracy of gesture recognition.

[0087] S300. Based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system, calibrate the three-dimensional coordinates of each hand bone node of the user in the actual physical scene;

[0088] Specifically, in a possible design, in step S300, an interactive display terminal is provided in the actual physical scene, and a first target camera is provided at the interactive display terminal;

[0089] Among them, based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system, the three-dimensional coordinates of each hand bone node of the user in the actual physical scene are calibrated, including:

[0090] S3001. Obtain the intrinsic parameter matrix of the first target camera;

[0091] S3002. Based on the viewpoint coordinate system, establish a second rigid body transformation matrix, wherein the second rigid body transformation matrix is ​​used to represent the conversion relationship between the viewpoint coordinate system and the actual scene coordinate system;

[0092] S3003. Based on the intrinsic parameter matrix of the first target camera and the second rigid body transformation matrix, the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system are converted into the three-dimensional coordinates of each hand bone node of the user in the actual physical scene.

[0093] S400. Mapping the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene to create a virtual hand in the virtual scene, wherein the virtual hand includes a plurality of virtual bone nodes, and each virtual bone node corresponds to each hand bone node one by one;

[0094] Specifically, in a possible design, in step S400, the three-dimensional coordinates of each hand bone node of the user in the actual physical scene are mapped to the virtual scene to create a virtual hand in the virtual scene, including:

[0095] S4001. Establish a virtual scene coordinate system;

[0096] S4002. Mapping the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene coordinate system to form the virtual scene coordinates of each hand bone node of the user;

[0097] S4003. Using the virtual scene coordinates of each hand bone node of the user, each virtual bone node is created in the virtual scene, wherein the virtual scene coordinates of each hand bone node of the user correspond one to one with the coordinate position of each virtual bone node in the virtual scene coordinate system;

[0098] S4004. Create a virtual hand in the virtual scene according to each hand skeleton node.

[0099] The virtual scene coordinate system is a three-dimensional coordinate system, and the coordinates in the virtual scene coordinate system are in a one-to-one mapping relationship with the coordinates in the actual scene coordinate system;

[0100] S500. Obtain the displacement information of the user's viewpoint and the motion information of each hand bone node of the user, and update each virtual bone node in the virtual hand according to the displacement information of the user's viewpoint and the motion information of each hand bone node of the user, so as to obtain virtual hand gesture information based on each updated virtual bone node;

[0101] Specifically, in a possible design, in step S500, each virtual bone node is updated in real time according to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, and virtual hand gesture information is obtained through each virtual bone node in the virtual scene, including:

[0102] S5001. Obtain the displacement information of the user's viewpoint and the motion information of each hand bone node of the user in real time;

[0103] S5002. According to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the three-dimensional coordinates of each hand bone node of the user in the actual physical scene after real-time update are obtained;

[0104] S5003. Based on the real-time updated three-dimensional coordinates of the user's viewpoint in the actual physical scene and the real-time updated three-dimensional coordinates of each hand bone node of the user in the actual physical scene, a real-time updated virtual hand is obtained in the virtual scene;

[0105] S5004. Obtain virtual hand gesture information according to each virtual skeleton node in the virtual hand that is updated in real time.

[0106] S600. Acquire virtual hand gesture information, compare it with designated gesture information, and provide real-time feedback of the comparison result to achieve human-computer interaction.

[0107] Specifically, in a possible design, in step S600, based on the preset designated gesture information, determining the interactive response instruction corresponding to the virtual hand gesture information, and executing the interactive response instruction to achieve human-computer interaction, including:

[0108] S6001. Acquire a specified gesture information library, wherein the specified gesture information library includes a number of specified gesture information, and each specified gesture information corresponds to an interactive response instruction;

[0109] S6002. Based on the virtual hand gesture information, the designated gesture information corresponding to the virtual hand gesture information is filtered out from the designated gesture information library;

[0110] S6003. Execute the interactive response instruction corresponding to the designated gesture information corresponding to the virtual hand gesture information to realize human-computer interaction.

[0111] It should be noted that the designated gesture information library is preset before the user performs human-computer interaction, and several of the designated gesture information correspond to preset interaction response instructions. The designated gesture information corresponding to the virtual hand gesture information is screened out from the designated gesture information library, and the interaction response instructions corresponding to the designated gesture information corresponding to the virtual hand gesture information are executed to achieve the purpose of human-computer interaction; multiple designated gesture information can be preset and correspond to different interaction response instructions respectively, which can expand the scope of use of the gesture interaction method in this embodiment and enhance the user experience.

[0112] Among them, the preset of the specified gesture information needs to determine the gesture features first, and the basic elements of the gesture features can be selected as the movement speed, relative distance and relative angle of the hand bone node. Define the position coordinates of each hand bone node relative to the palm as its direction vector, and use the direction vector of the fingertip bone node as the direction vector of the root finger. The relative distance and relative angle between two bone nodes can be calculated by the direction vectors of the two bone nodes, and the movement speed can be calculated by the direction vector of the same bone node at different times. In addition, the selection of gesture features needs to take into account the user's habits and the distinction between gesture features. When different users make the same type of gestures with the same semantics, due to different physiological structures or inconsistent speeds of action completion, it is impossible to achieve complete consistency of gestures after dataization, and even the same user cannot reproduce the same gesture completely. Therefore, in order to improve the consistency of subsequent comparisons, after collecting the hand bone node data, it is necessary to select appropriate feature elements for feature space construction, so that gestures that want to express the same semantics are clustered into one category to improve the practicality of gesture recognition.

[0113] This gesture interaction method adds the function of viewpoint positioning on the basis of determining gesture interaction. This viewpoint is the observation point of the user performing human-computer interaction. When the user moves or turns his head in the actual physical scene, his viewpoint will change accordingly. Viewpoint positioning exists to deal with this situation. When the user's viewpoint moves, it will immediately obtain the displacement information of the user's viewpoint, update the viewpoint position, and update the established viewpoint coordinate system. This allows the user to move freely within a fairly large range during human-computer interaction, greatly improving the user's freedom of gesture interaction in large scenes.

[0114] Moreover, in large scenes, it is often difficult to capture the position of the user's bare hands through the positioning camera of the display terminal, and it is even more difficult to identify the user's hand bone nodes, which makes gesture interaction in large scenes difficult to achieve. The gesture interaction method provided in this embodiment can transform the difficult problem of identifying the user's hand bone nodes into a simple coordinate system conversion problem by establishing a viewpoint coordinate system. This is because the viewpoint coordinate system is established based on the user's viewpoint. When the user performs human-computer interaction, a high-precision camera is placed at the viewpoint position, which can perform high-precision positioning of the user's hand bone nodes in real time, and complete the coordinate position calibration in the viewpoint coordinate system to obtain its viewpoint coordinates, and then transform the viewpoint coordinate system to obtain the three-dimensional coordinates of each hand bone node of the user in the actual physical scene, thereby achieving the purpose of accurately obtaining the position data of the user's hand bone nodes, and then continuously updating and mapping it to the virtual scene, ultimately achieving the effect of bare-hand interaction and realizing the concept of "what you see is interaction".

[0115] Embodiment 2:

[0116] This embodiment provides a gesture interaction system applied to large scenes, which is used to implement any one of the embodiments 1 or any possible design of the gesture interaction method applied to large scenes in the embodiment 1, and is characterized by comprising:

[0117] A positioning module, used to obtain the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and use the three-dimensional coordinates of the user's viewpoint in the actual physical scene as the first coordinate origin to establish a viewpoint coordinate system, wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system;

[0118] A viewpoint tracking module is used to obtain the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and to establish a viewpoint coordinate system based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and is also used to calibrate the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system based on the viewpoint coordinate system; wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system, and the first coordinate origin of the viewpoint coordinate system is the three-dimensional coordinates of the user's viewpoint in the actual physical scene;

[0119] A mapping interaction module is used to map the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene to create a virtual hand in the virtual scene, obtain the displacement information of the user's viewpoint and the action information of each hand bone node of the user, and update each virtual bone node in the virtual hand according to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, so as to obtain virtual hand gesture information based on each updated virtual bone node; wherein the virtual hand includes a plurality of virtual bone nodes, and each virtual bone node corresponds to each hand bone node one by one;

[0120] The display feedback module is used to determine the interactive response instruction corresponding to the virtual hand gesture information based on the preset designated gesture information, and execute the interactive response instruction to realize human-computer interaction.

[0121] Through the coordinated work of various modules, the user's position can be freely moved during human-computer interaction in large scenes, and precise and intuitive bare-hand interaction can be achieved.

[0122] Embodiment 3:

[0123] This embodiment provides a computer program product, including a computer program or instructions, wherein the computer program or instructions, when executed by a computer, implements a gesture interaction method applied to a large scene. The computer here may refer to a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0124] It should be understood that specific details are provided in the above description to facilitate a complete understanding of the example embodiments. However, it should be understood by those of ordinary skill in the art that the example embodiments can be implemented without these specific details. For example, the system can be shown in a block diagram to avoid making the example unclear with unnecessary details. In other examples, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid making the example embodiments unclear.

[0125] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A gesture interaction method applied to large scenes, characterized in that: include: Acquire the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and establish a viewpoint coordinate system based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene, wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system, and the first coordinate origin of the viewpoint coordinate system is the three-dimensional coordinates of the user's viewpoint in the actual physical scene; Based on the viewpoint coordinate system, calibrate the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system; Based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system, the three-dimensional coordinates of each hand bone node of the user in the actual physical scene are calibrated; Mapping the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene to create a virtual hand in the virtual scene, wherein the virtual hand includes a plurality of virtual bone nodes, and each virtual bone node corresponds to each hand bone node one by one; Acquire displacement information of the user's viewpoint and motion information of each hand bone node of the user, and update each virtual bone node in the virtual hand according to the displacement information of the user's viewpoint and the motion information of each hand bone node of the user, so as to obtain virtual hand gesture information based on each updated virtual bone node; Based on the preset designated gesture information, an interactive response instruction corresponding to the virtual hand gesture information is determined, and the interactive response instruction is executed to realize human-computer interaction.

2. The gesture interaction method applied to large scenes according to claim 1 is characterized in that ,An interactive display terminal is set in the actual physical scene, and a first target camera is set at the interactive display terminal; Among them, obtaining the three-dimensional coordinates of the user's viewpoint in the actual physical scene includes: Acquire the position information of the first target camera, and determine the position of the first target camera as the second coordinate origin according to the position information of the first target camera; According to the second coordinate origin, the actual scene coordinate system is established; The position information of the user's viewpoint is obtained, and the position information of the user's viewpoint is mapped into the actual scene coordinate system to determine the three-dimensional coordinates of the user's viewpoint position in the actual physical scene.

3. The gesture interaction method applied to large scenes according to claim 1, characterized in that: Based on the viewpoint coordinate system, the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system are calibrated, including: Acquire user's hand image in real time; Based on the user's hand image, the data of each hand bone node of the user is calculated based on the YOLOv8 algorithm; Based on the data of each hand bone node of the user, the viewpoint coordinates of each hand bone node of the user are calibrated in the viewpoint coordinate system.

4. The gesture interaction method applied to large scenes according to claim 3 is characterized in that: The user performs gesture interaction based on the interactive glasses, wherein the interactive glasses are provided with a second target camera facing the user's viewpoint; Among them, based on the data of each hand bone node of the user, the viewpoint coordinates of each hand bone node of the user are calibrated in the viewpoint coordinate system, including: From the user's hand bone node data, select any hand bone node as the third coordinate origin; According to the third coordinate origin, a hand tracking coordinate system is established, wherein the hand tracking coordinate system selects the direction from the back of the hand to the palm as the positive direction of the Z axis, the direction from the palm to the finger as the positive direction of the X axis, and the direction from the palm to the index finger as the positive direction of the Y axis; Based on the data of each hand skeleton node of the user, the hand tracking coordinates of each hand skeleton node of the user are calibrated in the hand tracking coordinate system; Get the intrinsic parameter matrix of the second target camera; Establishing a first rigid body transformation matrix, wherein the first rigid body transformation matrix is ​​used to represent the transformation relationship between the hand tracking coordinate system and the viewpoint coordinate system; Based on the intrinsic parameter matrix of the second target camera and the first rigid body transformation matrix, the hand tracking coordinates of each hand bone node of the user in the hand tracking coordinate system are converted into the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system.

5. The gesture interaction method applied to large scenes according to claim 1, characterized in that: An interactive display terminal is arranged in the actual physical scene, and a first target camera is arranged at the interactive display terminal; Among them, based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system, the three-dimensional coordinates of each hand bone node of the user in the actual physical scene are calibrated, including: Get the intrinsic parameter matrix of the first target camera; Based on the viewpoint coordinate system, establishing a second rigid body transformation matrix, wherein the second rigid body transformation matrix is ​​used to represent the conversion relationship between the viewpoint coordinate system and the actual scene coordinate system; Based on the intrinsic parameter matrix of the first target camera and the second rigid body transformation matrix, the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system are converted into the three-dimensional coordinates of each hand bone node of the user in the actual physical scene.

6. The gesture interaction method applied to large scenes according to claim 5, characterized in that: Mapping the 3D coordinates of each hand bone node of the user in the actual physical scene to the virtual scene to create a virtual hand in the virtual scene, including: Establish a virtual scene coordinate system; Mapping the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene coordinate system to form the virtual scene coordinates of each hand bone node of the user; Using the virtual scene coordinates of each hand bone node of the user, each virtual bone node is created in the virtual scene, wherein the virtual scene coordinates of each hand bone node of the user correspond one-to-one to the coordinate position of each virtual bone node in the virtual scene coordinate system; A virtual hand is created in the virtual scene according to each hand bone node.

7. The gesture interaction method applied to large scenes according to claim 1, characterized in that: According to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, each virtual bone node in the virtual hand is updated to obtain virtual hand gesture information based on each updated virtual bone node, including: Obtain the displacement information of the user's viewpoint and the motion information of each hand bone node of the user in real time; According to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, the three-dimensional coordinates of the user's viewpoint in the actual physical scene and the three-dimensional coordinates of each hand bone node of the user in the actual physical scene after real-time update are obtained; Based on the real-time updated three-dimensional coordinates of the user's viewpoint in the actual physical scene and the real-time updated three-dimensional coordinates of each hand bone node of the user in the actual physical scene, a real-time updated virtual hand is obtained in the virtual scene; According to each virtual skeleton node in the virtual hand updated in real time, the virtual hand gesture information is obtained.

8. The gesture interaction method applied to large scenes according to claim 1, characterized in that: Based on preset designated gesture information, determining an interactive response instruction corresponding to the virtual hand gesture information, and executing the interactive response instruction to achieve human-computer interaction, including: Acquire a designated gesture information library, wherein the designated gesture information library includes a plurality of designated gesture information, and each designated gesture information corresponds to an interactive response instruction; Based on the virtual hand gesture information, the designated gesture information corresponding to the virtual hand gesture information is filtered out from the designated gesture information library; Execute the interactive response instruction corresponding to the designated gesture information corresponding to the virtual hand gesture information to realize human-computer interaction.

9. A gesture interaction system applied to large scenes, used to implement the gesture interaction method applied to large scenes as claimed in any one of claims 1 to 8, characterized in that: include: A positioning module, used to obtain the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and use the three-dimensional coordinates of the user's viewpoint in the actual physical scene as the first coordinate origin to establish a viewpoint coordinate system, wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system; A viewpoint tracking module, used to obtain the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and to establish a viewpoint coordinate system based on the three-dimensional coordinates of the user's viewpoint in the actual physical scene, and also used to calibrate the viewpoint coordinates of each hand bone node of the user in the viewpoint coordinate system based on the viewpoint coordinate system; wherein the three-dimensional coordinates of the user's viewpoint in the actual physical scene are determined by the actual scene coordinate system, and the first coordinate origin of the viewpoint coordinate system is the three-dimensional coordinates of the user's viewpoint in the actual physical scene; A mapping interaction module, used to map the three-dimensional coordinates of each hand bone node of the user in the actual physical scene to the virtual scene, so as to create a virtual hand in the virtual scene, obtain the displacement information of the user's viewpoint and the action information of each hand bone node of the user, and update each virtual bone node in the virtual hand according to the displacement information of the user's viewpoint and the action information of each hand bone node of the user, so as to obtain virtual hand gesture information based on each updated virtual bone node; wherein the virtual hand includes a plurality of virtual bone nodes, and each virtual bone node corresponds to each hand bone node one by one; The display feedback module is used to determine the interactive response instruction corresponding to the virtual hand gesture information based on the preset designated gesture information, and execute the interactive response instruction to achieve human-computer interaction.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the gesture interaction method applied to a large scene as claimed in any one of claims 1 to 8 is implemented.

Images