Mixed reality-based simulated microgravity environment hand-object grasping method and apparatus
By using mixed reality technology to detect hand posture and collision in real time, and combining visual algorithms and Coulomb friction models, the limitations of existing hand grasping methods in complex scenarios and microgravity environments are solved, and efficient and low-cost space grasping simulation training is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2023-11-14
- Publication Date
- 2026-07-28
AI Technical Summary
Existing hand-grabbing methods lack universality in complex scenarios. Gesture-based and heuristic methods require predefined information, while physics-based methods involve large computational loads and lack real-time performance, and are difficult to simulate real-world interactions in microgravity environments.
Using a mixed reality-based approach, hand posture and hand-object collision are detected in real time. Visual algorithms and Coulomb friction models are used to execute hand-object interaction actions. The rendering of the hand provides a sense of realism and reduces the reliance on complex devices.
It provides an efficient and low-cost microgravity environment simulation tool, which improves the realism and efficiency of astronaut training and reduces the difficulty and cost of training.
Smart Images
Figure CN117373313B_ABST
Abstract
Description
Technical Field
[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a method and apparatus for grasping objects in a simulated microgravity environment based on mixed reality. Background Technology
[0002] Current hand-grabbing methods primarily employ gesture-based or heuristic-based approaches. Both are simple to implement, computationally inexpensive, and less prone to object drop during grasping. However, both require predefined prior information. Gesture-based methods predefine several sets of gestures for manipulating the object, assigning specific semantics to each gesture. During grasping, static or dynamic gesture recognition is used to match the corresponding semantics, thereby controlling the object's movement. Heuristic-based methods require a series of rules based on prior information such as the object's shape to determine the grasping state. In addition, physics-based hand-grabbing methods are increasingly being used. These methods primarily achieve hand-object gripping through force calculations and physical models. In recent years, with the development of deep learning, hand-grabbing motion generation has also received widespread attention. The main application scenario for this method is when users interact with objects using a controller. Deep learning methods generate appropriate gripping actions based on the current hand and object postures. Current hand-grabbing methods often suffer from "clipping" issues. To address this problem, the God-object algorithm is commonly used.
[0003] Both gesture-based and heuristic-based grasping methods require predefined prior information. Gesture-based methods require predefined gestures, while heuristic-based methods require predefined rules for grasping objects. Both methods have limitations in complex scenarios and lack universality in object grasping. Physics-based methods suffer from high computational costs and cannot guarantee real-time system performance. Hand-grabbing action generation methods are primarily used with hand controllers and require large datasets. God-object algorithms, which rely on penetration, are prone to unrealistic interactions in microgravity environments and struggle to simulate diverse interaction methods. Summary of the Invention
[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0005] Some embodiments of this disclosure propose a method and apparatus for grasping objects in a simulated microgravity environment based on mixed reality, in order to solve one or more of the technical problems mentioned in the background section above.
[0006] In a first aspect, some embodiments of this disclosure provide a hand-object grasping method based on mixed reality in a simulated microgravity environment. The method includes: real-time detection of hand posture information to obtain hand posture information; real-time detection of collision between the hand and the object to obtain hand-object collision information; and execution of hand-object interaction actions based on the aforementioned hand posture information and the aforementioned hand-object collision information.
[0007] Secondly, some embodiments of this disclosure provide a hand-object grasping device based on mixed reality in a simulated microgravity environment. The device includes: a first detection unit configured to detect hand posture information in real time to obtain hand posture information; a second detection unit configured to detect collision between the hand and an object in real time to obtain hand-object collision information; and an execution unit configured to execute hand-object interaction actions based on the aforementioned hand posture information and the aforementioned hand-object collision information.
[0008] This disclosure primarily addresses the challenges astronauts face when training for space grasping maneuvers on the ground. The microgravity environment in space differs significantly from Earth's surface conditions, making it difficult for ground-based training to fully simulate the real-world space grasping experience. To solve this problem, this disclosure employs vision-based simulation technology. The system utilizes advanced visual algorithms to provide astronauts with a highly realistic simulation of the space grasping environment. This not only helps astronauts better understand and adapt to the dynamics of object grasping in space but also significantly improves training efficiency and effectiveness. Most importantly, this vision-based simulation method means that astronauts do not need to rely on any complex physical equipment during training, thus reducing training costs and difficulty. In summary, the purpose of this disclosure is to provide astronauts with an efficient and practical ground-based grasping simulation training tool, enabling them to better prepare for space missions and ensure safe and accurate execution of grasping operations in a realistic space environment. Attached Figure Description
[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.
[0010] Figure 1 This is a flowchart of some embodiments of the hand-grabbing method based on mixed reality in a simulated microgravity environment according to the present disclosure;
[0011] Figure 2This is a schematic diagram of the structure of some embodiments of the hand-grabbing device based on mixed reality and simulating microgravity environment according to the present disclosure;
[0012] Figure 3 These are schematic diagrams of scenarios suitable for implementing the mixed reality-based simulated microgravity environment hand grasping method of this disclosure. Detailed Implementation
[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0014] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.
[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0018] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0019] Figure 1 A flow 100 of some embodiments of a hand-grabbing method for simulating microgravity environments based on mixed reality according to this disclosure is shown. This hand-grabbing method for simulating microgravity environments based on mixed reality includes the following steps:
[0020] Step 101: Real-time detection of hand posture information to obtain hand posture information.
[0021] In some embodiments, the execution subject (e.g., a computing device) of the hand grasping method based on mixed reality simulated microgravity environment can detect the hand's posture information in real time and obtain the hand posture information.
[0022] Here, the aforementioned hand posture information may include, but is not limited to: hand position information, direction description information, finger bending angle information, and palm shape information. The aforementioned hand position information can refer to the spatial coordinates of the hand. For example, the aforementioned hand position information can refer to three-dimensional coordinates (x, y, z). The aforementioned x, y, and z can refer to the spatial coordinates of the hand. The aforementioned direction description information can refer to the direction vector information of the hand. For example, the aforementioned direction description information can refer to three-dimensional coordinates (a, b, c). The aforementioned a, b, and c can refer to the pointing coordinates of the fingers. The aforementioned finger bending angle information can refer to the degree of bending of each joint of the fingers. The aforementioned palm shape information can refer to the palm shape information expressed through the angles of the finger joints and the gaps between the fingers. For example, the aforementioned palm shape information can refer to the open shape.
[0023] As an example, the aforementioned execution entity can detect hand posture information in real time through a hand tracking module to obtain hand posture information. This hand tracking module can refer to a Leap Motion controller.
[0024] Optionally, the aforementioned executing entity can detect hand posture information in real time through the following steps to obtain hand posture information:
[0025] The first step is to create a skeletal hand model for the virtual hand, which includes joints and bones.
[0026] Here, the aforementioned skeletal hand model can refer to a 3D model of the skeletal hand.
[0027] The second step involves using motion-sensing control devices to acquire hand posture data and performing real-time hand posture detection through binocular vision algorithms.
[0028] Here, the aforementioned motion control device may refer to a Leap Motion controller. The aforementioned hand posture data may refer to the hand posture information data.
[0029] The third step is to perform mean filtering on the above hand posture data to obtain hand posture information.
[0030] Step 102: Real-time detection of collisions between the hand and the object to obtain the hand-object collision information.
[0031] In some embodiments, the aforementioned execution entity can detect the collision between the hand and the object in real time to obtain the hand-object collision situation.
[0032] As an example, the aforementioned execution entity can use a collision detection module to detect collisions between the hand and an object in real time, thus obtaining the hand-object collision status. This collision detection module can refer to a 3D model composed of multiple triangular facets. The collision detection module primarily uses a ray collision detection algorithm, employing rays to calculate whether a collision will occur with an object in that direction within a time step.
[0033] Optionally, the aforementioned executing entity can detect the collision between the hand and the object in real time through the following steps to obtain the hand-object collision status:
[0034] The first step is to determine the resultant velocity of the hand and the object using the continuous collision detection method, thereby obtaining the relative velocity of the hand and the object on the normal.
[0035] Here, the aforementioned continuous collision detection method can refer to single-point collision detection.
[0036] As an example, the aforementioned executing entity can... Obtain the relative velocity between the hand and the object along the normal. Here, v represents the relative velocity between the hand and the object along the normal. c1 and c2 represent the relative velocities of the hand and the object, respectively. This indicates the collision speed of the hand when it collides with an object. This represents the collision velocity of the object when the hand collides with it. n represents the normal vector. This indicates transpose. This represents the transpose of the normal vector. This represents the resultant velocity of the hand and the object.
[0037] The second step is to determine the direction of relative motion between the hand and the object based on the relative velocity mentioned above.
[0038] Optionally, the aforementioned executing entity can determine the relative motion direction between the hand and the object based on the aforementioned relative velocity through the following steps:
[0039] The first sub-step, in response to determining that the relative velocity is greater than a preset velocity threshold, determines the relative motion direction between the hand and the object as moving closer to each other.
[0040] Here, the aforementioned preset speed threshold can refer to 0.
[0041] The second sub-step, in response to determining that the relative speed is less than or equal to a preset speed threshold, determines the direction of relative motion between the hand and the object as moving away from each other.
[0042] The third step is to detect the collision between the hand and the object based on the aforementioned relative motion direction.
[0043] As an example, the aforementioned executing entity can determine that a collision between the hand and the object is imminent in response to determining that the relative motion direction is towards each other. Conversely, it can determine that a collision between the hand and the object will not occur in response to determining that the relative motion direction is away from each other.
[0044] Step 103: Based on the above hand posture information and the above hand-object collision situation, perform the hand-object interaction action.
[0045] In some embodiments, the execution entity may perform hand-object interaction actions based on the hand posture information and the hand-object collision situation.
[0046] As an example, the aforementioned execution entity performs hand-object interaction actions based on the aforementioned hand posture information and the aforementioned hand-object collision situation through the hand-object interaction module. The aforementioned hand-object interaction module can refer to the Coulomb friction model.
[0047] Optionally, the aforementioned executing entity can perform hand-object interaction actions based on the aforementioned hand posture information and the aforementioned hand-object collision situation through the following steps:
[0048] The Coulomb friction model is used to perform grasping between the hand and the object, thereby performing hand-object interaction.
[0049] Optionally, the aforementioned executing entity can utilize the Coulomb friction model to perform a grasping action between the hand and the object through the following steps:
[0050] The first step is to apply pressure and friction to the hand and the object, based on the Coulomb friction model described above.
[0051] The second step is to determine the type of friction force mentioned above by using the friction cone.
[0052] The third step is to determine that the type of friction is kinetic friction and that the relative motion between the hand and the object is in opposite directions.
[0053] The fourth step is to determine that the type of friction force is static friction force, and then compare the friction force with a preset critical value to obtain the comparison result.
[0054] The fifth step is to determine that the frictional force is equal to a preset critical value in response to the above comparison results, and to determine that the relative motion between the hand and the object is stationary.
[0055] Here, the aforementioned preset critical value can refer to a pre-set static friction coefficient value. For example, the aforementioned preset critical value could be 0.15.
[0056] Step 6: In response to the determination that the above comparison results indicate that the frictional force is less than a preset critical value, determine that the relative motion between the hand and the object is in opposite directions.
[0057] Step 7: In response to the determination that there will be overlap between the hand and the object, the rendering hand is determined by using the target hand and the object through the God object model. The target hand is not displayed in the system, while the rendering hand is displayed in the system. The target hand is used for real-time collision detection, and the rendering hand is used for visual anti-penetration and the application of the Coulomb friction model.
[0058] Here, the aforementioned God object model can refer to the God-object model. The aforementioned target hand can refer to a virtual hand. The aforementioned rendered hand can refer to a real hand.
[0059] The eighth step involves executing a grasping motion between the hand and the object, based on the direction of the relative motion between the hand and the object as described above.
[0060] Here, the rendered hand provides the user with a natural and visually realistic representation of hand movements during interaction with the object, and the displayed hand applies force to the object. The displayed hand is used to enhance the overall user experience and improve the realism of the interaction.
[0061] Optionally, after using the Coulomb friction model to perform hand-object grasping to execute hand-object interaction, the method further includes:
[0062] In response to the determination that a hand collides with an object in a microgravity environment, an impulse is applied to the object to simulate the slapping phenomenon between the hand and the object.
[0063] Here, the aforementioned impulse refers to a vector in mechanics, used to describe the change in the momentum of an object.
[0064] As an example, the aforementioned executing entity can... This is used to simulate the phenomenon of a hand striking an object. Here, m represents mass. obj The object's mass is represented by `obj`. The object itself is represented by `Δv`. obj This indicates the motion state of an object. J represents impulse. obj represents the object's identifier.
[0065] Schematic diagrams of some embodiments of the hand-grabbing method based on mixed reality in a simulated microgravity environment disclosed herein are shown below. Figure 3 As shown.
[0066] The technical details are further explained below with reference to the embodiments.
[0067] Example 1
[0068] The hand-grabbing method for simulating microgravity environments based on mixed reality disclosed in Example 1 may include the following steps:
[0069] The first step is to design a user experiment. For each participant, they will complete the task using both regular-shaped objects (such as cubes) and special-shaped objects (such as flat or elongated objects), and record the time taken to complete the task and the number of failures.
[0070] The second step, for all users, is to first familiarize themselves with the movement of the virtual hand in the virtual environment using objects of regular shapes, and then use the virtual hand to grasp virtual objects. The system starts timing when it starts up.
[0071] Third, after the user grabs the object, they first move it to target point 1 in the lower left corner. When the object reaches target point 1, a timer starts at target point 1, and the user needs to hold the object there for 3 seconds. If the object leaves the user's hand during this time, the task is considered a failure, and one failure is recorded, returning to step two to start again. Otherwise, proceed to step four.
[0072] Fourth step: After completing step three, the user must continue to hold the object and move it to target point 2 directly above. When the object reaches target point 2, a timer starts at target point 2, and the user must hold the object there for 3 seconds. If the object leaves the user's hand during this time, the task is considered a failure, and one failure is recorded, returning to step two to start again. Otherwise, proceed to step five.
[0073] Fifth, after completing step four, the user must continue to hold the object and move it to target point 3 directly above. When the object reaches target point 3, a timer starts at target point 3, and the user must hold the object there for 3 seconds. If the object leaves the user's hand during this time, the task is considered a failure, and one failure is recorded, requiring the user to return to step one and start again. Otherwise, proceed to step six.
[0074] Step 6: Once the user completes step 5, the task is considered complete. At this point, the system stops timing and records the time it took for the user to complete the task.
[0075] Step 7: After the user completes the task using regular objects, they can then complete the task using other specially shaped objects. Repeat steps 1-6 to record the time taken to complete the task and the number of failures when using other objects.
[0076] Step 8: Count the number of times users failed to complete the task and the time it took to complete the task when using different objects, and perform analysis of variance to calculate the significance level of task completion.
[0077] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a hand-grabbing method based on mixed reality in a simulated microgravity environment. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.
[0078] like Figure 2 As shown, a hand-grabbing device 200 based on mixed reality in a simulated microgravity environment, according to some embodiments, includes: a first detection unit 201, a second detection unit 202, and an execution unit 203. The first detection unit 201 is configured to detect the hand's posture information in real time to obtain hand posture information; the second detection unit 202 is configured to detect the collision between the hand and an object in real time to obtain hand-object collision information; and the execution unit 203 is configured to perform hand-object interaction actions based on the aforementioned hand posture information and hand-object collision information.
[0079] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.
[0080] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A hand-grabbing method based on mixed reality in a simulated microgravity environment, including: Real-time detection of hand posture information to obtain hand posture information includes: establishing a skeletal hand model for a virtual hand, wherein the skeletal hand model includes joints and bones; acquiring hand posture data using a motion-sensing control device, and performing real-time hand posture detection through a binocular vision algorithm; performing mean filtering on the hand posture data to obtain hand posture information, wherein the hand posture information includes: finger bending angle information and palm shape information, wherein the finger bending angle information refers to the degree of bending of each joint of the fingers, and the palm shape information refers to the palm shape information expressed by the angle of the finger joints and the gap between the fingers; Real-time detection of collisions between the hand and objects, obtaining information on hand-object collision details; Based on the hand posture information and the hand-object collision situation, perform hand-object interaction actions, including: performing a grasping action between the hand and the object using a Coulomb friction model; in response to determining that there will be overlap between the hand and the object, determining and rendering the hand using the target hand and the object through a God object model, wherein the target hand is not displayed in the system and is used for real-time collision detection, while the rendered hand is displayed in the system and is used for visual anti-penetration and the application of the Coulomb friction model; and performing a grasping action between the hand and the object based on the direction of the relative motion between the rendered hand and the hand and the object.
2. The method according to claim 1, wherein, The real-time detection of collisions between the hand and the object, and the acquisition of hand-object collision information, includes: Based on the continuous collision detection method, the resultant velocity of the hand and the object is determined, and the relative velocity of the hand and the object on the normal is obtained. Based on the relative velocity, determine the direction of relative motion between the hand and the object; Based on the relative motion direction, the collision between the hand and the object is detected, and the hand-object collision situation is obtained.
3. The method according to claim 2, wherein, Determining the relative motion direction between the hand and the object based on the relative velocity includes: In response to determining that the relative speed is greater than a preset speed threshold, the relative motion direction between the hand and the object is determined to be that they are moving closer to each other; In response to determining that the relative speed is less than or equal to a preset speed threshold, the relative motion direction between the hand and the object is determined to be moving away from each other.
4. The method according to claim 1, wherein, The method of using the Coulomb friction model to perform hand-object grasping to execute hand-object interaction also includes: According to the Coulomb friction model, pressure and friction are applied between the hand and the object; The type of frictional force is determined by the friction cone; In response to determining that the type of frictional force is kinetic friction, the relative motion between the hand and the object is determined to be in opposite directions; In response to determining that the type of frictional force is static friction, the frictional force is compared with a preset critical value to obtain a comparison result; In response to determining that the comparison result indicates that the frictional force is equal to a preset critical value, the relative motion between the hand and the object is determined to be stationary. In response to determining that the comparison result indicates that the frictional force is less than a preset critical value, it is determined that the relative motion between the hand and the object is in opposite directions.
5. The method according to claim 1, wherein, After performing the grasping action between the hand and the object using the Coulomb friction model to execute the hand-object interaction action, the method further includes: In response to the determination that a hand collides with an object in a microgravity environment, an impulse is applied to the object to simulate the slapping phenomenon between the hand and the object.
6. A hand-grabbing device based on mixed reality to simulate microgravity environment, comprising: The first detection unit is configured to detect hand posture information in real time and obtain hand posture information, including: establishing a skeletal hand model for a virtual hand, wherein the skeletal hand model includes joints and bones; acquiring hand posture data using a motion-sensing control device, and performing real-time hand posture detection through a binocular vision algorithm; performing mean filtering on the hand posture data to obtain hand posture information, wherein the hand posture information includes: finger bending angle information and palm shape information, wherein the finger bending angle information refers to the degree of bending of each joint of the fingers, and the palm shape information refers to the palm shape information expressed by the angle of the finger joints and the gap between the fingers; The second detection unit is configured to detect the collision between the hand and the object in real time and obtain the hand-object collision information. An execution unit is configured to perform hand-object interaction actions based on the hand posture information and the hand-object collision situation, including: performing a grasping action between the hand and the object using a Coulomb friction model; in response to determining that an overlap will occur between the hand and the object, determining a rendered hand using a God object model based on the target hand and the object, wherein the target hand is not displayed in the system and is used for real-time collision detection, while the rendered hand is displayed in the system and is used for visual anti-penetration and the application of the Coulomb friction model; and performing a grasping action between the hand and the object based on the direction of the relative motion between the rendered hand and the hand and the object.