A high-quality human-computer interaction method and device based on posture matching

Abstract

The application discloses a high-quality human-computer interaction method and device based on pose matching. The method comprises the following steps: establishing a human-computer interaction pose library of a mixed reality cockpit environment, and obtaining camera space pose information through camera space calibration operation; calculating six-degree-of-freedom information of a user's wrist rigid body by using a camera motion capture system according to the camera space pose information, and determining geometric feature information of the user's hand; calculating position information of a fingertip area under a preset coordinate system according to the six-degree-of-freedom information of the user's wrist rigid body and the geometric feature information of the user's hand; and judging an interaction state of a collision detection bounding box according to the position information. The application solves the technical problem that the interaction data set collected in the prior art is not necessarily continuous interaction action behavior, and the role of the data set is not to drive human motion, so that if the data set is used to drive digital human motion, the search difficulty of a specific interaction pose is increased, and the continuity of human motion in time and space cannot be ensured.

Description

TECHNICAL FIELD

[0001] The application provides a high-quality human-computer interaction method suitable for a mixed reality cockpit scene, matches the motion behavior of a real hand with an existing gesture in a hand motion gesture database, so as to drive the motion of a digital person by real motion capture data or artificially synthesized data, interact, improve the human-computer interaction precision, and improve the natural performance capability. The application belongs to the technical field of virtual reality and mixed reality. BACKGROUND

[0002] With the continuous development of intelligent technology, people increasingly use intelligent devices in life, work and study, and use intelligent technology to improve the quality of life and increase the efficiency of learning and work.

[0003] MR technology is applied to aircraft cockpit scheme evaluation: mixed reality (MR) is a visualization technology that superimposes virtual objects and real objects as a twin system, which connects the real world and the virtual world and interacts in real time, and therefore gradually attracts the attention of the aviation, aerospace, high-speed rail, ship and automobile fields, and begins to be applied to product design evaluation, complex large component manufacturing and assembly, operation training and maintenance support, etc. The aircraft cockpit is an important part of the pilot's control of the aircraft, with complex flight control and state detection interactive interfaces, so the design of the cockpit not only needs to consider the realization of the functions of the aircraft, but also needs to fully consider the accessibility, efficiency, safety and reliability of the pilot's operation in the cockpit environment. The traditional method of evaluating the quality of the cockpit design scheme is to use a full-size cockpit physical prototype, which not only has high production cost and long production time, but also is difficult to modify after production, and cannot be flexibly used for comparison, evaluation and optimization iteration between different conceptual design schemes of the aircraft cockpit. In order to improve flexibility and replace the evaluation function of the physical cockpit prototype, MR technology is used to superimpose a part of the modular experimental simulation part and the virtual cockpit prototype to form a mixed reality cockpit, striving to create a real cockpit environment, allowing experimental evaluation personnel to immerse in the environment and simulate real flight conditions, thereby achieving flexible scheme evaluation effect. In addition to greatly improving the flexibility of evaluation, the use of MR technology also reduces the cost of evaluation.

[0004] Current motion capture and human-computer interaction technologies: To enable real-time interaction between real and digital humans in MR environments, motion capture devices are used to track human motion, with real hands serving as the primary interaction target. Common hand motion capture devices include data gloves using inertial measurement unit sensors, and optical devices such as RGB / RGBD cameras and infrared cameras. Compared to data gloves, which suffer from drift, markerless RGB / RGBD cameras, which use deep learning and image processing techniques, and marker-based infrared cameras, which use motion capture, often offer more accurate and stable pose tracking and are widely used for interaction in large indoor environments. However, compared to the human body, human hands are small and flexible, possess numerous joints, and possess a large number of degrees of freedom. Furthermore, fingers exhibit self-similarity. Placing markers at every finger joint will exacerbate occlusion during interaction, leading to loss of joint position information. Furthermore, marker identification is difficult, especially when a marker disappears from the infrared camera's tracking range; online identification of its reappearance is challenging. The usual solution is to change the arrangement of infrared cameras, reduce the tracking space, and increase the number of cameras. However, this not only increases costs, but also limits the spatial tracking capabilities of motion capture equipment. Another solution is to reduce the number of hand joint markers, thereby reducing the difficulty of marker identification and alleviating occlusion problems. However, the positions of joints without markers are calculated using the IK algorithm, and their positions are not accurate. Not only are they inaccurate, but the existing heuristic, Jacobi, and Newton IK algorithms also do not have the ability to add global constraints on joint motion, resulting in joint motion that does not conform to natural human hand posture changes. In addition, once the markers are blocked, more joint position information will be lost, thereby disrupting the movement of the digital human hand, such as Figure 1 shown.

[0005] Common human-computer interaction methods in the industry: Currently, for online hand motion tracking using dense or sparse marker placement, machine learning algorithms are typically used to predict missing markers and re-mark them when they reappear within the tracking range. Dense marker placement can track complex and subtle hand movements, but it places strict demands on the number of cameras and their placement. IK algorithms have been continuously improved for sparse marker placement. While sparse marker placement reduces the constraints imposed by real-world human movements, the joint positions calculated by IK are currently inaccurate and lack global constraints. Human motion is increasingly being driven by motion capture equipment, using real-world data to solve IK problems. However, most of these technologies use RGB / RGBD depth cameras to capture images and utilize image processing techniques, such as deep learning, to achieve matching results, thereby enabling human-computer interaction. Few patents or research directly utilizes 3D datasets obtained from actual measurements to drive digital human motion, particularly for more complex hand movements, enabling real-time human-computer interaction and motion synthesis.

[0006] The closest patented technical solution: Human-computer interaction method and device based on 3D image human posture matching, such as Figure 2 As shown, this invention provides a method for identifying whether a real person is in a human-computer interaction state. First, a three-dimensional skeleton model is constructed using deep learning technology. The skeleton model is driven by real human skeleton information input to obtain human posture features. Then, the obtained posture features are compared with the three-dimensional image template posture stored in the interactive machine, that is, the interaction dataset. The loss function between the two is calculated and judged to be less than a certain threshold. Ultimately, it is determined whether the real person is interacting with the device. Its patent number is CN202110406810A.

[0007] A method and device for generating input information based on limb posture: This invention provides a method for generating input information based on limb posture. The method first records limb posture data obtained by motion sensors installed on the subject's wrists and ankles. The recorded posture data is then processed to construct a posture matching library that records the mapping relationship between specified movements and posture data. Once the posture matching library is constructed, the collected real-life limb posture data is added to a matching queue and matched with the existing posture data in the library. If a match is successful, the existing data in the library is used to drive the virtual character, and the limb data in the matching queue is removed, thereby improving memory utilization efficiency. The patent number is CN201510998896A.

[0008] The first patent mentioned above uses deep learning methods to detect the interaction between real people and machines. Therefore, the interaction dataset collected does not necessarily represent continuous interactive actions, and the purpose of the dataset is not to drive human movement. Using this dataset to drive digital human movement not only increases the difficulty of searching for specific interactive postures, but also fails to ensure the continuity of human movement in time and space. Patent 2 focuses on mining the movement characteristics of human limbs, collecting limb movement data to build a posture library, and then matching the acquired movement data with existing postures in the library. The major difference from Patent 1 lies in the data acquisition and representation methods and the complexity of the sample features. Human movement has complex morphology, while Patent 2's sample features are relatively simple, focusing only on the extremities. However, there are multiple postures with the same extremity movement characteristics. This will result in the postures matched from the library not truly reflecting the current real person's movement posture and can easily lead to discontinuities in the digital human's movement in time and space. Both Patent 1 and Patent 2 ignore the importance of hands in the interaction process. This patent not only collects the motion data of real human hands and extracts relevant features for analysis and utilization, but also considers the consistency of hand movements in time and space dimensions. Therefore, the method proposed in this patent is essentially different from the above methods.

[0009] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0010] The embodiments of the present invention provide a high-quality human-computer interaction method and device based on posture matching, which at least solves the technical problem in the existing technology that the interactive action behaviors collected by the interactive data set are not necessarily continuous, and the role of the data set is not to drive human movement. If this data set is used to drive the movement of the digital human, it will not only increase the difficulty of searching for a specific interactive posture, but also fail to ensure the continuity of human movement in time and space.

[0011] According to one aspect of an embodiment of the present invention, a high-quality human-computer interaction method based on posture matching is provided, including: establishing a human-computer interaction posture library for a mixed reality cockpit environment, and obtaining camera space posture information through camera space calibration operations; based on the camera space posture information, using a camera motion capture system to calculate the six-degree-of-freedom information of the user's wrist rigid body, and determine the geometric feature information of the user's hand; based on the six-degree-of-freedom information of the user's wrist rigid body and the geometric feature information of the user's hand, calculating the position information of the fingertip area in a preset coordinate system; and judging the interaction state of the collision detection bounding box based on the position information.

[0012] Optionally, the human-computer interaction posture library of the mixed reality cockpit environment is established, and the camera space posture information is obtained through the camera space calibration operation, including: the interphalangeal joint is regarded as 1 degree of freedom, the metacarpophalangeal joint is regarded as 2 degrees of freedom, the carpometacarpal joint is regarded as 1 degree of freedom, and the wrist joint is regarded as 6 degrees of freedom, which is determined by the rigid body of the back of the hand, thereby constructing a digital human hand kinematic model with 26 degrees of freedom consisting of 16 joints, and the kinematic parameters are recorded as According to the distribution pattern of healthy male and female hand sizes and referring to human body size data standards, gloves of different sizes were selected. Corresponding markers were arranged on the gloves according to the 16 joints and fingertip areas determined by the digital human model. Users with different hand characteristics were selected and made to wear gloves of corresponding hand sizes. No less than 10 OptiTrack motion capture devices were used to capture the postures of users operating the simulation device. The spatial positions of each joint and fingertip area of ​​the hand in each posture relative to the rigid body of the back of the hand were recorded, and the kinematic parameters were calculated.

[0013] Optionally, the position information of the fingertip area in the preset coordinate system includes: the position of the wrist is The wrist rotation angle is The position of each fingertip is The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture classification is Motion={P1,P2,P3,…,P n}、The fingertip area position in the gesture library is

[0014] Optionally, the interactive state judgment of the collision detection bounding box according to the position information includes: calculating If D is greater than a given threshold, there is no interaction; otherwise, it is in an interactive state.

[0015] According to another aspect of an embodiment of the present invention, a high-quality human-computer interaction device based on posture matching is also provided, including: an establishment module for establishing a human-computer interaction posture library for a mixed reality cockpit environment, and obtaining camera space posture information through camera space calibration operations; a first calculation module for calculating the six-degree-of-freedom information of the user's wrist rigid body based on the camera space posture information using a camera motion capture system, and determining the geometric feature information of the user's hand; a second calculation module for calculating the position information of the fingertip area in a preset coordinate system based on the six-degree-of-freedom information of the user's wrist rigid body and the geometric feature information of the user's hand; and a judgment module for performing interaction status judgment on the collision detection bounding box based on the position information.

[0016] Optionally, the establishment module includes: a first unit for treating the interphalangeal joints as 1 degree of freedom, the metacarpophalangeal joints as 2 degrees of freedom, the carpometacarpal joints as 1 degree of freedom, and the wrist joints as 6 degrees of freedom, which are determined by the rigid body of the back of the hand, thereby constructing a digital human hand kinematic model consisting of 16 joints with 26 degrees of freedom, and the kinematic parameters are recorded as The second unit is used to select gloves of different sizes according to the distribution of hand sizes of healthy men and women and refer to human body size data standards, and arrange corresponding markers on the gloves according to the 16 joints and fingertip areas determined by the digital human model; the third unit is used to select users with different hand characteristics and make them wear gloves that match the corresponding hand size. The user's posture when operating the simulation device is captured using no less than 10 OptiTrack motion capture devices, and the spatial position of each joint and fingertip area of ​​the hand in each posture relative to the rigid body of the back of the hand is recorded to calculate the kinematic parameters.

[0017] Optionally, the position information of the fingertip area in the preset coordinate system includes: the position of the wrist is The wrist rotation angle is The position of each fingertip is The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture is classified as Motion={P1, P2, P3, …, P n} and the fingertip region position in the posture library is

[0018] Optionally, the judging module comprises a judging unit configured to calculate wherein D is greater than a given threshold value, then no interaction, otherwise, in an interaction state.

[0019] According to another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, which comprises a stored program, wherein the program controls a device in which the non-volatile storage medium is located to perform a high-quality human-computer interaction method based on posture matching when the program is run.

[0020] According to another aspect of the embodiments of the present application, an electronic device is also provided, which comprises a processor and a memory; the memory stores computer readable instructions, and the processor is configured to run the computer readable instructions, wherein the computer readable instructions perform a high-quality human-computer interaction method based on posture matching when the computer readable instructions are run.

[0021] In the embodiments of the present application, a human-computer interaction posture library of a mixed reality cockpit environment is established, and camera space pose information is obtained through camera space calibration operation; six degrees of freedom information of a user's wrist rigid body is calculated by using a camera motion capture system according to the camera space pose information, and geometric feature information of the user's hand is determined; position information of a fingertip region in a preset coordinate system is calculated according to the six degrees of freedom information of the user's wrist rigid body and the geometric feature information of the user's hand; and the manner of judging the interaction state of the collision detection bounding box according to the position information solves the technical problem that the interaction action behavior collected in the prior art is not necessarily continuous, and the role of the data set is not to drive human motion, so that if the data set is used to drive digital human motion, not only the search difficulty of a specific interaction posture is increased, but also the continuity of human motion in time and space cannot be guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application. In the drawings:

[0023] Figure 1 is a virtual-real matching error diagram of a human-computer interaction process in the prior art according to the embodiments of the present application

[0024] Figure 2 is a human posture matching flowchart based on three-dimensional images in the prior art according to the embodiments of the present application

[0025] Figure 3 is a prior art four-limb posture matching diagram according to an embodiment of the present application;

[0026] Figure 4 is a principle diagram according to an embodiment of the present application;

[0027] Figure 5 is a technical flow diagram according to an embodiment of the present application;

[0028] Figure 6 is a digital human hand kinematics model diagram according to an embodiment of the present application;

[0029] Figure 7 is a posture library hierarchical diagram according to an embodiment of the present application;

[0030] Figure 8 is a world coordinate system and local coordinate system diagram according to an embodiment of the present application;

[0031] Figure 9 is a classifier working process diagram according to an embodiment of the present application;

[0032] Figure 10 is a hand local coordinate system diagram according to an embodiment of the present application;

[0033] Figure 11 is a synthetic hand posture diagram according to an embodiment of the present application;

[0034] Figure 12 is a flow diagram of a high-quality human-computer interaction method based on posture matching according to an embodiment of the present application;

[0035] Figure 13 is a structural block diagram of a high-quality human-computer interaction device based on posture matching according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] According to an embodiment of the present invention, a method embodiment of a high-quality human-computer interaction method based on posture matching is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Example 1

[0040] Figure 12 is a flow chart of a high-quality human-computer interaction method based on posture matching according to an embodiment of the present invention. Figure 12 As shown, the method includes the following steps:

[0041] Step S1202: Establish a human-computer interaction posture library for the mixed reality cockpit environment, and obtain camera space posture information through camera space calibration operations.

[0042] Step S1204: Calculate the six-degree-of-freedom information of the user's wrist rigid body based on the camera's spatial pose information using the camera motion capture system, and determine the geometric feature information of the user's hand.

[0043] Step S1206: Calculate the position information of the fingertip area in a preset coordinate system based on the six-degree-of-freedom information of the user's wrist rigid body and the geometric feature information of the user's hand.

[0044] Step S1208: determining the interaction state of the collision detection bounding box according to the position information.

[0045] Optionally, the human-computer interaction posture library of the mixed reality cockpit environment is established, and the camera space posture information is obtained through the camera space calibration operation, including: the interphalangeal joint is regarded as 1 degree of freedom, the metacarpophalangeal joint is regarded as 2 degrees of freedom, the carpometacarpal joint is regarded as 1 degree of freedom, and the wrist joint is regarded as 6 degrees of freedom, which is determined by the rigid body of the back of the hand, thereby constructing a digital human hand kinematic model with 26 degrees of freedom consisting of 16 joints, and the kinematic parameters are recorded as According to the distribution pattern of healthy male and female hand sizes and referring to human body size data standards, gloves of different sizes were selected. Corresponding markers were arranged on the gloves according to the 16 joints and fingertip areas determined by the digital human model. Users with different hand characteristics were selected and made to wear gloves of corresponding hand sizes. No less than 10 OptiTrack motion capture devices were used to capture the postures of users operating the simulation device. The spatial positions of each joint and fingertip area of ​​the hand in each posture relative to the rigid body of the back of the hand were recorded, and the kinematic parameters were calculated.

[0046] Optionally, the position information of the fingertip area in the preset coordinate system includes: the position of the wrist is The wrist rotation angle is The position of each fingertip is The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture classification is Motion={P1,P2,P3,…,P n}、The fingertip area position in the gesture library is

[0047] Optionally, the interactive state judgment of the collision detection bounding box according to the position information includes: calculating If D is greater than a given threshold, there is no interaction; otherwise, it is in an interactive state.

[0048] In an embodiment of the present invention, a rigid body, encoded with small marker balls, is placed on the back of a real person's hand to determine the position of both hands in the mixed reality cockpit space. Markers are placed on the fingertips of each finger to determine the position of the fingertips in space. Combined with the geometric information of the hand's back rigid body, the geometric outline of the real person's hand and the position of the markers in the fingertips relative to the hand's back rigid body can be determined. When a real person enters the mixed reality cockpit environment, the input geometric outline features are first used to match a model in the hand motion posture database (referred to as the posture library for the remainder of this technical briefing) with the model that most closely matches the size of the real person's hand. Each operating device in the virtual cockpit is equipped with a collision detection box. When the digital human hand is outside the collision box, it is considered non-interactive. At this time, the position of the fingertips relative to the hand's back rigid body is used as the end-effector input for the IK algorithm. When the digital human is inside the collision box, the position of the fingertips relative to the hand's back rigid body is input. Feature matching is performed for different relative fingertip positions to select the optimal posture in the posture library. If the output of the optimal pose is greater than a set threshold, the optimal pose match is considered unsuccessful. The real person is not interacting at this time, but the fingertip area positions are still input into the IK solver to drive the digital human's motion. Otherwise, the data in the pose library is used as a reference to replace the current real person's hand motion state. While the fingertip area positions provide precise positioning, the matched data undergoes IK adjustments to correspond to the current real person's hand motion changes in real time. Furthermore, the real person is required to wear a head-mounted display device and place markers on the fingertips of each finger, the extremities used for interaction. When not interacting, a local spatial coordinate system is established using the rigid body pose information of the back of the real person's hand. The positions of the fingertip markers in this local coordinate system serve as input to the IK solver, which calculates the positions of other unmarked hand joints in real time to drive the digital human's hand motion. During interaction, the optimal pose is selected from the pose library based on the real person's fingertip area position characteristics. The selected data undergoes IK optimization adjustments to match the current real hand motion state in real time.

[0049] Therefore, the specific implementation steps of the embodiment of the present invention are:

[0050] Establish a human-computer interaction posture library for the mixed reality cockpit environment: the interphalangeal joint is regarded as 1 degree of freedom, the metacarpophalangeal joint is regarded as 2 degrees of freedom, the carpometacarpal joint is regarded as 1 degree of freedom, and the wrist joint is regarded as 6 degrees of freedom, which are determined by the rigid body of the back of the hand. Thus, a digital human hand kinematic model with 26 degrees of freedom consisting of 16 joints is constructed. The kinematic parameters are recorded as

[0051] According to the distribution of hand sizes of healthy men and women, gloves of different sizes were selected based on human body size data standards. Corresponding marking points were placed on the gloves according to the 16 joints and fingertip areas determined by the digital human model.

[0052] Select users with different hand features and have them wear gloves that match their hand sizes. Use no less than 10 OptiTrack motion capture devices to capture the user's gestures when operating the simulation device. Record the spatial positions of each joint and fingertip area of ​​the hand relative to the rigid body of the back of the hand in each gesture, and calculate the kinematic parameters.

[0053] Install infrared cameras around the cockpit simulation experimental platform, calibrate the spatial position and orientation of the cameras, and obtain the spatial pose information of the cameras;

[0054] The user wears a rigid body composed of multiple marker balls and fingertips with marker points arranged on the fingertips to enter the cockpit simulation experimental platform and turn on the simulation platform. The world coordinate system established by the multi-camera system is denoted as W; the local coordinate system is established based on the six-degree-of-freedom information of the user's wrist rigid body, denoted as H.

[0055] The camera motion capture system is used to calculate the six degrees of freedom information of the user's wrist rigid body in real time, and the position information of each fingertip: the position of the wrist is The wrist rotation angle is The position of each fingertip is

[0056] calculate Determine the geometric features of the user's hand and match the model closest to the current user's hand size in the gesture library; calculate the position of the fingertip area in the local coordinate system H The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture classification is Motion={P1,P2,P3,…,P n}、The fingertip area position in the gesture library is

[0057] Determine whether the digital human enters the collision detection bounding box. If it does, calculate If D is greater than a given threshold, there is no interaction. Otherwise, it is in the interactive state, and I = 1. Y = 1 further proves that it is in the interactive state:

[0058] When the user is in an interactive state, the optimal posture is searched in Motion. The back of the hand rigid body posture information and the actual fingertip area position are used to locate the fingertip position sample farthest from the posture library. Then, the IK optimization algorithm is used to perform motion synthesis to match the current state in real time.

[0059] When the user is not in the interactive state, directly As hand model IK solver input information, so as to drive digital hand movement;

[0060] Repeat step, update human-computer interaction state: i = {1, 2, 3, 4, 5}.

[0061] In addition, the embodiment of the present application also has the following technical features: establishing a mixed reality cockpit environment human-computer interaction posture library: the establishment of the mixed reality cockpit environment human-computer interaction posture library first needs to establish the kinematics model of the real hand. The kinematics model of the hand is simplified as a motion chain with 16 joints and a total of 26 degrees of freedom hierarchical structure, and the wrist has a total of six degrees of freedom, which is determined by the six degrees of freedom information of the rigid body arranged in the back of the hand. The hand kinematics model parameters are represented as The local transformation of the child joint relative to the parent joint is represented as T(θ), and the global transformation matrix of the child joint j relative to all n parent nodes is T j = Π n T i (θ i ).

[0062] Collecting hand posture takes three approaches: the first approach uses OptiTrack motion capture equipment. In order to reduce occlusion and reflect the flexible and subtle operation of the hand, no less than 10 OptiTrack cameras are arranged around a certain type of control component to reduce the tracking space, for example, if you want to collect the pilot's rotating knob action, focus all the cameras on the area where the knob is located. According to the male and female body size data standards, gloves of different specifications are selected. According to the hand kinematics model, mark points are arranged at the joint and fingertip area positions of the glove, and a rigid body is placed in the back area. Invite male and female experimental personnel with different hand characteristics to wear gloves that match their hand size to interact with different simulators in the mixed reality cockpit environment using the right hand, and the left hand posture is obtained by mirroring the right hand, so as to collect postures that conform to the real motion of the hand, and extract the six degrees of freedom information of the wrist and the position information of the fingertip area in the camera space coordinate system and the local coordinate system of the hand as sample data. The second approach uses data gloves to collect corresponding postures, and data gloves generally collect the rotation angle information of the hand joints, so it is necessary to obtain the position, velocity and other feature information of the wrist and fingertip area as mentioned above through angle conversion, which are also sample data. The third approach is to artificially synthesize data for specific actions, for example, using a 3D engine tool to adjust the joint positions of the digital hand model to obtain different types of action sequence postures.

[0063] The poses obtained through these three approaches are represented by skinned hand models of varying sizes that meet human dimensional data standards. Furthermore, these skinned models are scaled to match the size distribution of real human hands, further expanding the library to encompass poses of varying sizes and motions, thereby increasing data diversity.

[0064] Determining the Interaction State: Due to the high accuracy of rigid body modeling, the topographical features and spatial positions of the real-world cockpit simulation components can be assumed to fully match those of the corresponding components in the virtual cockpit. Therefore, the spatial positional features of the real person's hands can be used to determine whether the hands are near the device to be operated and whether the real person intends to operate it. A collision detection bounding box is configured for the trigger device within the virtual cockpit scene. When the distance between the real person's hand and the simulation component decreases and falls below a set threshold, the bounding box's detection function is triggered, signaling a pending interaction state. The relative position of the input fingertip area is searched for the optimal pose in the library. If the result exceeds the threshold and no similar pose matches the current motion state, the real person has no intention to interact and remains in the non-interaction state. Otherwise, the non-interaction state switches to the interaction state, and the trigger signal from the simulation component further confirms that the interaction state is in effect. At this point, the IK solver is no longer used to calculate the positions of the hand joints without markers. Instead, the optimal pose data from the pose library is used to drive the digital human's motion.

[0065] Searching for the optimal pose within the bounding box: When the digital hand is within the bounding box of the triggering device, further determining whether interaction is occurring involves mapping the position of the fingertip area relative to the hand back rigid body in the current frame input to a specific pose in the corresponding classification in the pose library. This mapping process is a supervised learning process, whereby a multi-classifier is trained using sample data provided by feature information. Machine learning techniques such as the random forest classification algorithm can be used.

[0066] When a real person and a digital human are interacting, the input feature is the relative spatial position of the fingertip area. In a mixed reality cockpit environment, interaction with experimental simulation devices scattered across different spatial regions is necessary. The spatial position of the fingertip area is extremely random, making it difficult to use as a feature to distinguish different hand postures. Directly using it to train multiple classifiers or as a feature input for multiple classifiers will not easily yield correct classification results. Since the relative position of the fingertip area relative to the wrist varies, the presented hand posture also varies. Therefore, a local coordinate system is established using the six-degree-of-freedom information of the wrist rigid body. Within this coordinate system, a linear transformation is performed on the fingertip area position sample data collected when constructing the posture library and the spatial position input during interaction to obtain the relative position, which is then used to train the classifier and serve as a feature input for the classifier.

[0067] The simplest method is to use Euclidean distance. Because different hand postures are present, the sum of the distances between each fingertip and the hand's back varies. Therefore, we preprocess the sample data to calculate the sum of the distances between the fingertip and the hand's back for each posture. This sum is then used to define independent intervals. When the input fingertip falls within this interval, the gesture is considered highly likely to represent the corresponding action. We then calculate the difference between the relative positions of the fingertips and the corresponding gestures in the gesture library to further confirm whether an interaction is occurring.

[0068] The position of the hand back rigid body in the world coordinate system is w P express, The rotation matrix is ​​represented by R, which is a third-order square matrix, and the position of the fingertip area in the world coordinate system is In the local coordinate system of the wrist rigid body, it can be expressed as The conversion relationship between the two is:

[0069] f h =R -1 f w -w P

[0070] Right now

[0071]

[0072] The trained multi-classifier is used to locate the real human hand motion posture into a specific action category.

[0073] After positioning, the Euclidean distance is used to determine whether it is less than the set threshold to trigger the interaction state in order to obtain the optimal posture.

[0074]

[0075] Among them, F input is the input feature vector, F data is the feature vector in the posture library.

[0076] Hand pose synthesis and precise positioning: After obtaining the optimal operating pose in the pose library, to ensure the continuity of hand movement in time and space, motion synthesis measures adjusted by the IK algorithm are adopted, based on the matching optimal pose. The six-degree-of-freedom information of the hand back rigid body is used to achieve precise positioning of the matching pose. At this time, the position of the real person's fingertip area and the fingertip area of ​​the matching pose may not coincide. The matching pose can be considered as a rigid body and cannot respond to changes in finger movement in real time. However, the position and rotation angle of each joint are known. Therefore, the position of the real person's fingertip area and the six-degree-of-freedom information of the hand back rigid body are used as inputs to the IK solver end effector to dynamically adjust the pose.

[0077] Assume that in the process of dynamically synthesizing hand gestures, each finger of the digital human is regarded as an independent kinematic chain, and the poses of the metacarpophalangeal joint and the carpometacarpal joint are consistent with the existing gestures in the library, and the two are respectively regarded as the root nodes of the corresponding fingers. Analyzing any finger of the digital human, let the position of the tip of the digital human finger be The target position is the position of the tip of the real person's finger, denoted as

[0078] The metacarpophalangeal joint has two degrees of freedom, represented by θ1 and θ2 respectively; the interphalangeal joint has one degree of freedom, and the two interphalangeal joints are represented by θ3 and θ4 respectively, then the kinematics parameters of the digital human finger can be represented as is a function of , and is A local coordinate system is established at each joint as shown in Figure 10 , and the local rotation and translation transformation of any child node relative to the parent node is denoted as T i (θ i ).

[0079] Given the six degrees of freedom information of the metacarpophalangeal joint or the carpometacarpal joint relative to the rigid body of the back of the hand, denoted as

[0080]

[0081]

[0082] The distance between the joints is a constant value, as shown in Figure 10 , which are l1, l2, l3 respectively, let

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] Then Let

[0090] R = R x (θ x1 ) R y (θ y2 ) R z (θ z3 )

[0091] but

[0092] The objective function is constructed as follows

[0093]

[0094] The Gauss-Newton method is used to obtain the optimal solution of the objective function. The first-order approximate iterative formula is:

[0095]

[0096] in

[0097]

[0098] For hand kinematics model

[0099] θ min ≤θ i ≤θ max

[0100] The technical problem addressed by this application proposal is the matching of the movements of the digital human and the real person, especially the hand movements, when the real person interacts with the physical scene in a mixed reality cockpit scenario. Specifically, there are two states: the first is the non-interaction state, which means that the real person's hands have not touched or triggered the experimental simulation device-related devices. In this case, accurate and natural hand posture is not required, and a quick reflection of human hand movement is sufficient. The hand joint position is solved using the IK algorithm with added independent constraints; the second state is the interaction state, which means that the real person's hands have touched or triggered the experimental simulation device-related devices. In this case, not only is the digital human required to have high interaction accuracy, but the hand posture must also be natural and coordinated, consistent with the movement relationship of the real person.

[0101] The embodiments of the present invention have the following advantages and technical effects:

[0102] The method proposed in this patent can be applied not only to high-quality human-computer interaction in mixed reality cockpit scenarios, but also to other scenarios that require complex interactions. When interacting, delicate and flexible hand movements are required, while when not interacting, only rough tracking of hand movements is required. Therefore, the states are divided into two categories: interacting and non-interacting, which simplifies the analysis of the problem. The gesture library collects real motion capture data or manually adjusted and synthesized data, which drives the hand movements of digital humans more naturally and more in line with real movement conditions, thereby enhancing the sense of immersion in the mixed reality environment. Compared with the traditional IK algorithm, the problem is more difficult to solve, the motion constraint effect is better, and the application value is higher.

[0103] The method provided in the patent proposal accelerates the search for the optimal pose in the pose library through optimal pose search and motion synthesis strategy, and drives the digital human motion in real time based on the optimal pose, greatly reducing the calculation amount and improving the efficiency. The pose library has good scalability and migration, and the established pose library is a general template pose library, which can continuously add new poses, increase interactive categories, expand the data scale, and further cover most interactive behaviors and migrate to other application scenarios. By reducing the number of marking points, selecting similar poses from the pose library can reduce the dependence of the system on the tracking device, reduce the system complexity, and reduce the cost.

[0104] Through the above embodiment, the technical problem that the collected data set in the prior art is not necessarily a continuous interactive action behavior, and the role of the data set is not to drive human motion, and if the data set is used to drive digital human motion, not only increases the search difficulty of a specific interactive pose, but also cannot guarantee the continuity of human motion in time and space is solved.

[0105] Embodiment two

[0106] Figure 13 is a structural block diagram of a high-quality human-computer interaction device based on pose matching according to an embodiment of the present application, as Figure 13 shown, the device comprises:

[0107] The establishing module 1300 is configured to establish a human-computer interaction pose library of a mixed reality cockpit environment, and obtain camera space pose information through camera space calibration.

[0108] The first calculation module 1302 is configured to calculate six degrees of freedom information of a user's wrist rigid body and determine geometric feature information of the user's hand according to the camera space pose information by using a camera motion capture system.

[0109] The second calculation module 1304 is configured to calculate position information of a fingertip area in a preset coordinate system according to the six degrees of freedom information of the user's wrist rigid body and the geometric feature information of the user's hand.

[0110] The judging module 1306 is configured to judge the interactive state of the collision detection bounding box according to the position information.

[0111] Optionally, the establishing module comprises a first unit configured to regard the interphalangeal joint as one degree of freedom, regard the metacarpophalangeal joint as two degrees of freedom, regard the carpometacarpal joint as one degree of freedom, regard the wrist joint as six degrees of freedom, determine by the rigid body of the back of the hand, and thereby construct a digital human hand kinematics model with 26 degrees of freedom composed of 16 joints, and record the kinematics parameters as The second unit is used to select gloves of different sizes according to the distribution of hand sizes of healthy men and women and refer to human body size data standards, and arrange corresponding markers on the gloves according to the 16 joints and fingertip areas determined by the digital human model; the third unit is used to select users with different hand characteristics and make them wear gloves that match the corresponding hand size. The user's posture when operating the simulation device is captured using no less than 10 OptiTrack motion capture devices, and the spatial position of each joint and fingertip area of ​​the hand in each posture relative to the rigid body of the back of the hand is recorded to calculate the kinematic parameters.

[0112] Optionally, the position information of the fingertip area in the preset coordinate system includes: the position of the wrist is The wrist rotation angle is The position of each fingertip is The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture classification is Motion={P1,P2,P3,…,P n}、The fingertip area position in the gesture library is

[0113] Optionally, the judgment module includes: a judgment unit for calculating Among them, if D is greater than a given threshold, there is no interaction, otherwise, it is in an interactive state.

[0114] In an embodiment of the present invention, a rigid body, encoded with small marker balls, is placed on the back of a real person's hand to determine the position of both hands in the mixed reality cockpit space. Markers are placed on the fingertips of each finger to determine the position of the fingertips in space. Combined with the geometric information of the hand's back rigid body, the geometric outline of the real person's hand and the position of the markers in the fingertips relative to the hand's back rigid body can be determined. When a real person enters the mixed reality cockpit environment, the input geometric outline features are first used to match a model in the hand motion posture database (referred to as the posture library for the remainder of this technical briefing) with the model that most closely matches the size of the real person's hand. Each operating device in the virtual cockpit is equipped with a collision detection box. When the digital human hand is outside the collision box, it is considered non-interactive. At this time, the position of the fingertips relative to the hand's back rigid body is used as the end-effector input for the IK algorithm. When the digital human is inside the collision box, the position of the fingertips relative to the hand's back rigid body is input. Feature matching is performed for different relative fingertip positions to select the optimal posture in the posture library. If the output of the optimal pose is greater than a set threshold, the optimal pose match is considered unsuccessful. The real person is not interacting at this time, but the fingertip area positions are still input into the IK solver to drive the digital human's motion. Otherwise, the data in the pose library is used as a reference to replace the current real person's hand motion state. While the fingertip area positions provide precise positioning, the matched data undergoes IK adjustments to correspond to the current real person's hand motion changes in real time. Furthermore, the real person is required to wear a head-mounted display device and place markers on the fingertips of each finger, the extremities used for interaction. When not interacting, a local spatial coordinate system is established using the rigid body pose information of the back of the real person's hand. The positions of the fingertip markers in this local coordinate system serve as input to the IK solver, which calculates the positions of other unmarked hand joints in real time to drive the digital human's hand motion. During interaction, the optimal pose is selected from the pose library based on the real person's fingertip area position characteristics. The selected data undergoes IK optimization adjustments to match the current real hand motion state in real time.

[0115] Therefore, the specific implementation steps of the embodiment of the present invention are:

[0116] Establish a human-computer interaction posture library for the mixed reality cockpit environment: the interphalangeal joint is regarded as 1 degree of freedom, the metacarpophalangeal joint is regarded as 2 degrees of freedom, the carpometacarpal joint is regarded as 1 degree of freedom, and the wrist joint is regarded as 6 degrees of freedom, which are determined by the rigid body of the back of the hand. Thus, a digital human hand kinematic model with 26 degrees of freedom consisting of 16 joints is constructed. The kinematic parameters are recorded as

[0117] According to the distribution of hand sizes of healthy men and women, gloves of different sizes were selected based on human body size data standards. Corresponding marking points were placed on the gloves according to the 16 joints and fingertip areas determined by the digital human model.

[0118] Select users with different hand features and have them wear gloves that match their hand sizes. Use no less than 10 OptiTrack motion capture devices to capture the user's gestures when operating the simulation device. Record the spatial positions of each joint and fingertip area of ​​the hand relative to the rigid body of the back of the hand in each gesture, and calculate the kinematic parameters.

[0119] Install infrared cameras around the cockpit simulation experimental platform, calibrate the spatial position and orientation of the cameras, and obtain the spatial pose information of the cameras;

[0120] The user wears a rigid body composed of multiple marker balls and fingertips with marker points arranged on the fingertips to enter the cockpit simulation experimental platform and turn on the simulation platform. The world coordinate system established by the multi-camera system is denoted as W; the local coordinate system is established based on the six-degree-of-freedom information of the user's wrist rigid body, denoted as H.

[0121] The camera motion capture system is used to calculate the six degrees of freedom information of the user's wrist rigid body in real time, and the position information of each fingertip: the position of the wrist is The wrist rotation angle is The position of each fingertip is

[0122] calculate Determine the geometric features of the user's hand and match the model closest to the current user's hand size in the gesture library; calculate the position of the fingertip area in the local coordinate system H The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture classification is Motion={P1,P2,P3,…,P n}、The fingertip area position in the gesture library is

[0123] Determine whether the digital human enters the collision detection bounding box. If it does, calculate If D is greater than a given threshold, there is no interaction. Otherwise, it is in the interactive state, and I = 1. Y = 1 further proves that it is in the interactive state:

[0124] When the user is in an interactive state, the optimal posture is searched in Motion. The back of the hand rigid body posture information and the actual fingertip area position are used to locate the fingertip position sample farthest from the posture library. Then, the IK optimization algorithm is used to perform motion synthesis to match the current state in real time.

[0125] When the user is not in the interactive state, directly Serves as input information for the hand model IK solver to drive the movement of the digital hand;

[0126] Repeat the steps to update the human-computer interaction state: i = {1, 2, 3, 4, 5}.

[0127] In addition, the embodiment of the present invention also has the following technical features: Establishing a human-computer interaction posture library for a mixed reality cockpit environment: To establish a human-computer interaction posture library for a mixed reality cockpit environment, it is first necessary to establish a kinematic model of a real human hand. The kinematic model of the hand is simplified to a kinematic chain with 16 joints and a total of 26 degrees of freedom. The wrist has a total of six degrees of freedom, which is determined by the six degrees of freedom information of the rigid body arranged in the back of the hand area. The hand kinematic model parameters are expressed as The local transformation of the child joint relative to the parent joint is expressed as T(θ), and the global transformation matrix of the child joint j relative to all its n parent nodes is T j =∏ n T i (θ i ).

[0128] Three approaches were used to capture hand gestures. The first utilized OptiTrack motion capture equipment. To reduce occlusion and capture delicate hand movements, no fewer than ten OptiTrack cameras were positioned around a specific control component to narrow the tracking volume. For example, to capture a pilot turning a knob, all cameras were focused on the knob area. Gloves of different sizes were selected based on male and female body size standards. Based on hand kinematic models, markers were placed on the glove joints and fingertips, and a rigid body was placed on the back. Equal numbers of male and female participants with varying hand characteristics were invited to wear gloves of appropriate hand size and interact with various simulated devices in a mixed reality cockpit environment using their right hands. The left hand gestures were obtained by mirroring the right hand, thereby capturing gestures that matched real-world hand movements. From these gestures, six-degree-of-freedom information about the wrist and the position of the fingertips in both the camera space coordinate system and the local hand coordinate system were extracted as sample data. The second category uses data gloves to capture gestures. Data gloves typically collect information about the rotation angles of hand joints, so angle conversion is required to obtain characteristic information such as position and velocity related to the wrist and fingertips, as described above. This information also serves as sample data. The third category uses artificially synthesized data for specific movements. For example, using 3D engine tools to adjust the joint positions of a digital hand model to obtain different types of action sequence gestures.

[0129] The poses obtained through these three approaches are represented by skinned hand models of varying sizes that meet human dimensional data standards. Furthermore, these skinned models are scaled to match the size distribution of real human hands, further expanding the library to encompass poses of varying sizes and motions, thereby increasing data diversity.

[0130] Determining the Interaction State: Due to the high accuracy of rigid body modeling, the topographical features and spatial positions of the real-world cockpit simulation components can be assumed to fully match those of the corresponding components in the virtual cockpit. Therefore, the spatial positional features of the real person's hands can be used to determine whether the hands are near the device to be operated and whether the real person intends to operate it. A collision detection bounding box is configured for the trigger device within the virtual cockpit scene. When the distance between the real person's hand and the simulation component decreases and falls below a set threshold, the bounding box's detection function is triggered, signaling a pending interaction state. The relative position of the input fingertip area is searched for the optimal pose in the library. If the result exceeds the threshold and no similar pose matches the current motion state, the real person has no intention to interact and remains in the non-interaction state. Otherwise, the non-interaction state switches to the interaction state, and the trigger signal from the simulation component further confirms that the interaction state is in effect. At this point, the IK solver is no longer used to calculate the positions of the hand joints without markers. Instead, the optimal pose data from the pose library is used to drive the digital human's motion.

[0131] Searching for the optimal pose within the bounding box: When the digital hand is within the bounding box of the triggering device, further determining whether interaction is occurring involves mapping the position of the fingertip area relative to the hand back rigid body in the current frame input to a specific pose in the corresponding classification in the pose library. This mapping process is a supervised learning process, whereby a multi-classifier is trained using sample data provided by feature information. Machine learning techniques such as the random forest classification algorithm can be used.

[0132] When a real person and a digital human are interacting, the input feature is the relative spatial position of the fingertip area. In a mixed reality cockpit environment, interaction with experimental simulation devices scattered across different spatial regions is necessary. The spatial position of the fingertip area is extremely random, making it difficult to use as a feature to distinguish different hand postures. Directly using it to train multiple classifiers or as a feature input for multiple classifiers will not easily yield correct classification results. Since the relative position of the fingertip area relative to the wrist varies, the presented hand posture also varies. Therefore, a local coordinate system is established using the six-degree-of-freedom information of the wrist rigid body. Within this coordinate system, a linear transformation is performed on the fingertip area position sample data collected when constructing the posture library and the spatial position input during interaction to obtain the relative position, which is then used to train the classifier and serve as a feature input for the classifier.

[0133] The simplest method is to use Euclidean distance. Because different hand postures are present, the sum of the distances between each fingertip and the hand's back varies. Therefore, we preprocess the sample data to calculate the sum of the distances between the fingertip and the hand's back for each posture. This sum is then used to define independent intervals. When the input fingertip falls within this interval, the gesture is considered highly likely to represent the corresponding action. We then calculate the difference between the relative positions of the fingertips and the corresponding gestures in the gesture library to further confirm whether an interaction is occurring.

[0134] The position of the hand back rigid body in the world coordinate system is w P express, The rotation matrix is ​​represented by R, which is a third-order square matrix, and the position of the fingertip area in the world coordinate system is In the local coordinate system of the wrist rigid body, it can be expressed as The conversion relationship between the two is:

[0135] f h =R -1 f w -w P

[0136] Right now

[0137]

[0138] The trained multi-classifier is used to locate the real human hand motion posture into a specific action category.

[0139] After positioning, the Euclidean distance is used to determine whether it is less than the set threshold to trigger the interaction state in order to obtain the optimal posture.

[0140]

[0141] Among them, F input is the input feature vector, F data is the feature vector in the posture library.

[0142] Hand pose synthesis and precise positioning: After obtaining the optimal operating pose in the pose library, to ensure the continuity of hand movement in time and space, motion synthesis measures adjusted by the IK algorithm are adopted, based on the matching optimal pose. The six-degree-of-freedom information of the hand back rigid body is used to achieve precise positioning of the matching pose. At this time, the position of the real person's fingertip area and the fingertip area of ​​the matching pose may not coincide. The matching pose can be considered as a rigid body and cannot respond to changes in finger movement in real time. However, the position and rotation angle of each joint are known. Therefore, the position of the real person's fingertip area and the six-degree-of-freedom information of the hand back rigid body are used as inputs to the IK solver end effector to dynamically adjust the pose.

[0143] Assume that in the process of dynamic hand gesture synthesis, each finger of the digital human is regarded as an independent kinematic chain, and the position of the metacarpophalangeal joint and the carpometacarpal joint is always consistent with the existing posture in the library, and the two are respectively the root nodes of the corresponding fingers. Take any finger of the digital human for analysis, and assume that the position of the end area of ​​the digital human fingertip is The target position is the real fingertip area position, denoted as

[0144] The metacarpophalangeal joint has two degrees of freedom, represented by θ1 and θ2 respectively; the interphalangeal joint has one degree of freedom, represented by θ3 and θ4 respectively. The kinematic parameters of the digital human finger can be expressed as yes The function is Establish at each joint Figure 10 The local coordinate system shown in Figure 2 is the local rotation and translation transformation of any child node relative to the parent node, which is T i (θ i ).

[0145] The six degrees of freedom information of the metacarpophalangeal joint or carpometacarpal joint relative to the rigid body of the back of the hand is known, denoted as

[0146]

[0147]

[0148] The distance between joints is a constant, such as Figure 10 As shown, they are l1, l2, l3 respectively, let

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155] but make,

[0156] R=R x (θ x1 )R y (θ y2 )R z (θ z3 )

[0157] but

[0158] The objective function is constructed as follows

[0159]

[0160] The Gauss-Newton method is used to obtain the optimal solution of the objective function. The first-order approximate iterative formula is:

[0161]

[0162] in

[0163]

[0164] For hand kinematics model

[0165] θ min ≤θ i ≤θ max

[0166] The technical problem addressed by this application proposal is the matching of the movements of the digital human and the real person, especially the hand movements, when the real person interacts with the physical scene in a mixed reality cockpit scenario. Specifically, there are two states: the first is the non-interaction state, which means that the real person's hands have not touched or triggered the experimental simulation device-related devices. In this case, accurate and natural hand posture is not required, and a quick reflection of human hand movement is sufficient. The hand joint position is solved using the IK algorithm with added independent constraints; the second state is the interaction state, which means that the real person's hands have touched or triggered the experimental simulation device-related devices. In this case, not only is the digital human required to have high interaction accuracy, but the hand posture must also be natural and coordinated, consistent with the movement relationship of the real person.

[0167] The embodiments of the present invention have the following advantages and technical effects:

[0168] The method proposed in this patent can be applied not only to high-quality human-computer interaction in mixed reality cockpit scenarios, but also to other scenarios that require complex interactions. When interacting, delicate and flexible hand movements are required, while when not interacting, only rough tracking of hand movements is required. Therefore, the states are divided into two categories: interacting and non-interacting, which simplifies the analysis of the problem. The gesture library collects real motion capture data or manually adjusted and synthesized data, which drives the hand movements of digital humans more naturally and more in line with real movement conditions, thereby enhancing the sense of immersion in the mixed reality environment. Compared with the traditional IK algorithm, the problem is more difficult to solve, the motion constraint effect is better, and the application value is higher.

[0169] The method proposed in this patent proposal accelerates the search for the optimal pose in the pose library through optimal pose search and motion synthesis strategies. It then uses the optimal pose as a benchmark to drive the digital human's movement in real time, significantly reducing computational complexity and improving efficiency. The pose library is highly scalable and portable. The established pose library is a universal template pose library that can continuously add new poses, increase interaction categories, and expand the data scale to cover most interactive behaviors and migrate to other application scenarios. By reducing the number of markers and selecting similar poses from the pose library, the system's reliance on tracking devices can be reduced, reducing system complexity and lowering costs.

[0170] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, which includes a stored program, wherein when the program is run, it controls the device where the non-volatile storage medium is located to execute a high-quality human-computer interaction method based on posture matching.

[0171] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory; the memory stores computer-readable instructions, and the processor is used to run the computer-readable instructions, wherein the computer-readable instructions, when run, execute a high-quality human-computer interaction method based on posture matching.

[0172] The above embodiments solve the technical problem in the prior art that the interactive data set collected does not necessarily represent continuous interactive actions, and the role of the data set is not to drive human movement. If this data set is used to drive the movement of the digital human, it will not only increase the difficulty of searching for a specific interactive posture, but also fail to ensure the continuity of human movement in time and space.

[0173] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0174] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0176] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0177] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0179] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A high-quality human-computer interaction method based on posture matching, characterized in that: include: Establish a human-computer interaction posture library for the mixed reality cockpit environment, and obtain the camera space pose information through camera space calibration operations; Based on the camera spatial pose information, a camera motion capture system is used to calculate six degrees of freedom information of the user's wrist rigid body and determine geometric feature information of the user's hand; Calculating position information of the fingertip area in a preset coordinate system based on the six-degree-of-freedom information of the user's wrist rigid body and the geometric feature information of the user's hand; Determine the interaction state of the collision detection bounding box according to the position information; Among them, the input geometric contour features are first used to match the model closest to the size of a real human hand in the hand motion posture database; each operating device in the virtual cockpit is equipped with a collision detection box. When the digital human hand is outside the collision box, it is considered to be in a non-interactive state. At this time, the position information of the fingertip area relative to the hand back rigid body is used as the end effector input of the IK algorithm; when the digital human is inside the collision box, the position of the fingertip area relative to the hand back rigid body is input, and feature matching is performed on different relative fingertip positions to select the optimal posture in the posture library; if the result of matching the optimal posture output is greater than the set threshold, the optimal posture match is considered unsuccessful, and the real human is not interacting at this time. The fingertip area position is still input into the IK solver to drive the digital human movement; otherwise, the data in the posture library is used as a benchmark to replace the current real human hand motion state. While the fingertip area position achieves precise positioning, the matched data is adjusted by IK to correspond to the current real human hand motion changes in real time.

2. The method according to claim 1, characterized in that The process of establishing a human-machine interaction posture library for a mixed reality cockpit environment and obtaining camera space posture information through a camera space calibration operation includes: The interphalangeal joints are considered as 1 degree of freedom, the metacarpophalangeal joints as 2 degrees of freedom, the carpometacarpal joints as 1 degree of freedom, and the wrist joints as 6 degrees of freedom, which are determined by the rigid body of the back of the hand. Thus, a kinematic model of the digital human hand with 26 degrees of freedom consisting of 16 joints is constructed. The kinematic parameters are recorded as According to the distribution of hand sizes of healthy men and women, gloves of different sizes were selected based on human body size data standards. Corresponding marking points were placed on the gloves according to the 16 joints and fingertip areas determined by the digital human model. Select users with different hand features and have them wear gloves that match their hand sizes. Use no less than 10 OptiTrack motion capture devices to capture the user's gestures when operating the simulation device. Record the spatial positions of each joint and fingertip area of ​​the hand relative to the rigid body of the back of the hand in each gesture, and calculate the kinematic parameters.

3. The method according to claim 1, characterized in that The position information of the fingertip area in the preset coordinate system includes: the position of the wrist is The wrist rotation angle is The position of each fingertip is The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture classification is Motion={P1,P2,P3,…,P n }、The fingertip area position in the gesture library is 4. The method according to claim 1, wherein The determining of the interaction state of the collision detection bounding box according to the position information includes: calculate If D is greater than a given threshold, there is no interaction; otherwise, it is in an interactive state.

5. A high-quality human-computer interaction device based on posture matching, characterized in that: include: Establish a module for building a human-machine interaction posture library for the mixed reality cockpit environment and obtain the camera space pose information through camera space calibration operations; A first calculation module is used to calculate the six-degree-of-freedom information of the user's wrist rigid body based on the camera spatial pose information using a camera motion capture system, and determine the geometric feature information of the user's hand; a second calculation module, configured to calculate position information of the fingertip area in a preset coordinate system based on the six-degree-of-freedom information of the user's wrist rigid body and the geometric feature information of the user's hand; A judgment module is configured to determine the interaction state of the collision detection bounding box based on the position information. The module first uses the input geometric contour features to match a model closest to the size of a real human hand in the hand motion posture database. Each operating device in the virtual cockpit is provided with a collision detection box. When the digital human hand is outside the collision box, it is considered to be in a non-interactive state. At this time, the position information of the fingertip area relative to the hand back rigid body is used as the end effector input of the IK algorithm. When the digital human is within the collision box, the position of the fingertip area relative to the hand back rigid body is input, and feature matching is performed on different relative fingertip positions to select the optimal posture in the posture library. If the result of matching the optimal posture output is greater than a set threshold, the optimal posture match is deemed unsuccessful and the real human is not interacting at this time. The fingertip area position is still input into the IK solver to drive the digital human movement. Otherwise, the data in the posture library is used as a reference to replace the current real human hand motion state. While the fingertip area position is used to achieve precise positioning, the matched data is adjusted by IK to correspond to the current real human hand motion changes in real time.

6. The device according to claim 5, characterized in that The establishment module includes: The first unit is used to treat the interphalangeal joint as 1 degree of freedom, the metacarpophalangeal joint as 2 degrees of freedom, the carpometacarpal joint as 1 degree of freedom, and the wrist joint as 6 degrees of freedom, which are determined by the rigid body of the back of the hand. Thus, a digital human hand kinematic model with 26 degrees of freedom consisting of 16 joints is constructed. The kinematic parameters are recorded as The second unit is used to select gloves of different sizes based on the size distribution of healthy male and female hands and refer to human body size data standards. The corresponding marking points are arranged on the gloves according to the 16 joints and fingertip areas determined by the digital human model. The third unit is used to select users with different hand features and have them wear gloves that match their hand sizes. Using no fewer than 10 OptiTrack motion capture devices, the user's gestures when operating the simulation device are recorded. The spatial positions of each joint and fingertip area of ​​the hand in each gesture relative to the rigid body of the back of the hand are recorded, and the kinematic parameters are calculated.

7. The device according to claim 5, characterized in that The position information of the fingertip area in the preset coordinate system includes: the position of the wrist is The wrist rotation angle is The position of each fingertip is The trigger signal of the experimental simulation device is Y={0, 1}, the interaction state is I={0, 1}, and the input feature set is The output posture classification is Motion={P1,P2,P3,…,P n }、The fingertip area position in the gesture library is 8. The device according to claim 5, characterized in that The judgment module includes: Judgment unit, used to calculate Among them, if D is greater than a given threshold, there is no interaction, otherwise, it is in an interactive state.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to any one of claims 1 to 4.

10. An electronic device, characterized in that: The method comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method according to any one of claims 1 to 4 when executed.

Images