An immersive virtual simulation interactive system

Abstract

The application discloses an immersive virtual simulation interaction system, comprising a wearing terminal and a virtual interaction platform, wherein the wearing terminal and the virtual interaction platform are connected, and the wearing terminal is a VR wearing terminal, and relates to the technical field of virtual interaction. The immersive virtual simulation interaction system adjusts the size of the picture displayed after the user wears the wearing terminal through the setting of a user adaptation unit, so that the high fitting degree of the picture displayed in the wearing terminal and the picture observed by the user in the actual environment is ensured, the simulation of the real environment outside is realized through the assembly of a plurality of parts, the virtual display of the assembly is realized through the feature association, and when the individual parts are displayed, the intelligent optimization of the parts displayed by the wearing terminal is realized through the judgment of the feature association data, so that the purpose of lightweight display is achieved.

Description

Technical Field

[0001] This invention relates to the field of virtual interaction technology, specifically to an immersive virtual simulation interaction system. Background Technology

[0002] Virtual reality (VR) technology uses simulation to create a three-dimensional virtual world, allowing users to experience the product or object's features and working principles through the simulation of their senses, such as sight, hearing, and touch. This allows users to feel as if they are actually there, without being limited by time, space, geography, or size.

[0003] When using VR glasses for virtual simulation interaction, an immersive virtual simulation interaction effect can be achieved. However, since different people see different perspectives, although VR glasses can achieve virtual simulation display, they cannot guarantee a high degree of fit between the image seen by the user wearing VR glasses and not wearing VR glasses. Furthermore, when displaying the real environment, there is a lack of structural optimization, making it impossible to achieve intelligent and lightweight display, resulting in a mediocre display effect. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides an immersive virtual simulation interaction system that solves the problems of low fidelity to real-world observations and the inability to provide intelligent and lightweight displays.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: an immersive virtual simulation interaction system, comprising a wearable terminal and a virtual interaction platform, wherein the wearable terminal is a VR wearable device, and the virtual interaction platform comprises a user adaptation unit, a database, an optimized display unit, and an action conversion unit. The user adaptation unit is used to adjust the size to fit the user's viewing angle. The database is used to store data of real external scenes composed of several parts forming an assembly. The database is connected to the optimized display unit, which is used to optimize the structure of the parts during display. The optimized display unit is connected to the action conversion unit, which is used to recognize hand gestures and trigger corresponding operation functions of the displayed parts.

[0008] The present invention is further configured such that: the user adaptation unit includes a reference input module, a perspective adjustment module, and an identity comparison module, wherein the reference input module is connected to the perspective adjustment module, and the perspective adjustment module is connected to the identity comparison module.

[0009] The present invention is further configured such that: the reference input module is used to obtain the external environment image directly in front of the user from the current perspective, and after keeping the body angle fixed, the user wears the terminal and displays the external environment image in the terminal.

[0010] The viewing angle adjustment module is used to adjust the size of the external environment image displayed in the wearable terminal so that the size of the external environment image displayed in the wearable terminal is similar to the external environment image when the wearable terminal is not worn.

[0011] The identity verification module is used to establish and store the mapping relationship between size adjustment information and user identity information.

[0012] The present invention is further configured such that: the database includes a data analysis module, an assembly storage module, a part information storage module, an association information storage module, and a single association information storage module; the data analysis module is connected to the assembly storage module; the assembly storage module is connected to both the part information storage module and the association information storage module; and the association information storage module is connected to the single association information storage module.

[0013] The present invention is further configured such that: the data analysis module is used to divide the real scene information to be simulated and displayed into several parts, and then perform feature association information of several parts according to the real scene information, and then assemble the real scene in the form of an assembly.

[0014] The assembly storage module is used to record and store assembly information in real-world scenarios;

[0015] The part information storage module is used to store part size and name information;

[0016] The associated information storage module is used to store the feature association information of several parts;

[0017] The single-feature association information storage module is used to filter single-feature association information and store it as a necessary feature.

[0018] The present invention is further configured such that: the optimized display unit includes a data filtering module, a data optimization module, and a part imaging module, wherein the data filtering module is connected to the data optimization module, and the data optimization module is connected to the part imaging module.

[0019] The present invention is further configured such that: the data filtering module is used to retrieve information about the corresponding part from the database when the user selects a part with a specified name;

[0020] The data optimization module is used to obtain the single feature association information when the corresponding part is displayed alone, remove the part that is repeated with the necessary feature from the single feature association information, optimize the remaining single feature association information, and obtain the optimized part information.

[0021] The part imaging module is used to display the optimized part information on the wearable terminal.

[0022] The present invention is further configured such that: the motion conversion unit includes a hand image acquisition module, a joint marking module, a gesture recognition module, a gesture-command reference library, and a command triggering module; the hand image acquisition module is connected to the joint marking module; the joint marking module is connected to the gesture recognition module; the gesture recognition module is connected to the gesture-command reference library; and the gesture-command reference library is connected to the command triggering module.

[0023] The present invention is further configured such that: the hand image acquisition module is a binocular vision camera installed on the wearable terminal, used to acquire hand images of the wearer of the wearable terminal;

[0024] The joint marking module is used to mark joint points on the finger joints;

[0025] The gesture recognition module is used to generate corresponding gestures based on the connection of key point marks and to perform gesture recognition.

[0026] The gesture-command lookup library is used to establish a mapping relationship between different gestures and operation functions in the virtual interaction platform, and to establish specified gesture trigger commands and gesture end commands. The gesture trigger command is used to start the gesture control operation function, and the gesture end command is used to turn off the gesture control operation function, while continuing the operation function corresponding to the gesture before the gesture end command.

[0027] The instruction triggering module is used to search for the corresponding operation function from the gesture-instruction lookup library based on the gesture recognized by the gesture recognition module, and trigger it to operate the displayed parts according to the operation function.

[0028] This invention also discloses a method for using an immersive virtual simulation interaction system, comprising the following steps:

[0029] Step 1, Viewpoint Adjustment: Acquire the external environment image directly in front of the user from the current viewpoint. After keeping the body angle fixed, put on the wearable terminal. Refer to the input module to display the external environment image on the wearable terminal. The viewpoint adjustment module adjusts the size of the external environment image displayed on the wearable terminal so that the size of the external environment image displayed on the wearable terminal matches the external environment image when the wearable terminal is not worn. Then, the identity verification module establishes a mapping relationship between the size adjustment information and the user's identity information and stores it.

[0030] Step 2, Data Analysis: The data analysis module divides the real scene information to be simulated into several parts, and then associates the feature information of several parts with the real scene information. The real scene is then assembled in the form of an assembly. The assembly storage module records and stores the real scene assembly information, the part information storage module stores the part size and name information, the association information storage module stores the feature association information of several parts, and the single association information storage module filters the single feature association information and stores it as a necessary feature.

[0031] Step 3, Optimize Display: When the user selects a part with a specified name, the data filtering module retrieves the information of the corresponding part from the part information storage module, the data optimization module retrieves the single feature association information of the corresponding part when it is displayed alone, removes the part that is repeated with the necessary feature from the single feature association information, optimizes the remaining single feature association information, and obtains the optimized part information. The part imaging module displays the optimized part information on the wearable terminal.

[0032] Step 4: The hand image acquisition module, acting as a binocular vision camera on the wearable terminal, acquires images of the wearer's hands. The joint marking module marks the joints of the fingers. The gesture recognition module generates corresponding gestures based on the joint markings and performs gesture recognition. The gesture-command lookup library is used to establish a mapping relationship between different gestures and operation functions in the virtual interaction platform, and to establish specified gesture trigger commands and gesture end commands. The gesture trigger command is used to start the gesture control operation function, and the gesture end command is used to close the gesture control operation function, while continuing the operation function corresponding to the gesture before the gesture end command. The command trigger module is used to search for the corresponding operation function in the gesture-command lookup library based on the gesture recognized by the gesture recognition module, and trigger it to operate the displayed parts according to the operation function.

[0033] (III) Beneficial Effects

[0034] This invention provides an immersive virtual simulation interaction system. It has the following beneficial effects:

[0035] (1) The present invention adjusts the screen size displayed after the user wears the terminal by setting the user adaptation unit, so as to ensure a high degree of fit between the screen displayed in the wearing terminal and the screen observed by the user in the actual environment. Furthermore, the invention simulates the real external environment by assembling several parts into an assembly. The invention also realizes the virtual display of the assembly by feature association. Moreover, when displaying individual parts, the invention can achieve intelligent optimization of the parts displayed in the wearing terminal by judging the feature association data, thereby achieving the purpose of lightweight display.

[0036] (2) The present invention uses the necessary feature settings stored in the single associated information storage module, combined with the screening of single feature associated information during the individual display of parts, to remove the part that is repeated with the necessary feature in the single feature associated information, and then optimizes the remaining single feature associated information. While ensuring the integrity of the parts, the optimized parts are displayed in a lightweight manner.

[0037] (3) This invention acquires hand images through a binocular vision camera and judges gestures by connecting finger joints, thereby achieving intelligent triggering of operation functions. In addition, with the setting of gesture triggering instructions and gesture ending instructions, it avoids accidental triggering of operation functions and provides effective support for users' immersive virtual simulation interaction. Attached Figure Description

[0038] Figure 1 This is a system principle block diagram of the present invention;

[0039] Figure 2 This is a system principle block diagram of the user adaptation unit of the present invention;

[0040] Figure 3 This is a system principle block diagram of the database of the present invention;

[0041] Figure 4 This invention provides an optimized system principle block diagram of the display unit.

[0042] Figure 5 This is a system principle block diagram of the action conversion unit of the present invention.

[0043] In the diagram: 1. Wearable terminal; 2. Virtual interaction platform; 3. User adaptation unit; 4. Database; 5. Optimized display unit; 6. Action conversion unit; 7. Reference input module; 8. View adjustment module; 9. Identity comparison module; 10. Data analysis module; 11. Assembly storage module; 12. Part information storage module; 13. Related information storage module; 14. Single related information storage module; 15. Data filtering module; 16. Data optimization module; 17. Part imaging module; 18. Hand image acquisition module; 19. Joint marking module; 20. Gesture recognition module; 21. Gesture-command reference library; 22. Command triggering module. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0045] Please see Figure 1-5 The present invention provides the following two technical solutions:

[0046] Example 1

[0047] An immersive virtual simulation interaction system includes a wearable terminal 1 and a virtual interaction platform 2. The wearable terminal 1 is a VR wearable device. The wearable terminal 1 and the virtual interaction platform 2 are connected. The virtual interaction platform 2 includes a database 4, an optimized display unit 5, and an action conversion unit 6. The database 4 is used to store external real scene data composed of several parts forming an assembly. Specifically, the database 4 includes a data analysis module 10, an assembly storage module 11, a part information storage module 12, an association information storage module 13, and a single association information storage module 14. The data analysis module 10 is used to divide the real scene information to be simulated and displayed into several parts, perform feature association information of several parts based on the real scene information, and assemble the real scene in the form of an assembly.

[0048] The data analysis module 10 is connected to the assembly storage module 11, which is used to record and store assembly information in real-world scenarios.

[0049] The assembly storage module 11 is connected to the part information storage module 12, which is used to store part size and name information.

[0050] The assembly storage module 11 is connected to the associated information storage module 13, which is used to store the feature association information of several parts.

[0051] The associated information storage module 13 is connected to the single associated information storage module 14. The single associated information storage module 14 is used to filter single feature associated information and store it as a necessary feature.

[0052] As a preferred solution, in order to achieve intelligent and lightweight display, the database 4 is connected to the optimized display unit 5. The optimized display unit 5 is used to optimize the structure of the parts when displaying them. Specifically, the optimized display unit 5 includes a data filtering module 15, a data optimization module 16 and a part imaging module 17. The data filtering module 15 is used to obtain the information of the corresponding part from the database 4 when the user selects a part with a specified name.

[0053] The data filtering module 15 is connected to the data optimization module 16. The data optimization module 16 is used to obtain the single feature association information when the corresponding part is displayed alone. After removing the part that is repeated with the necessary feature in the single feature association information, the remaining single feature association information is optimized to obtain the optimized part information.

[0054] The data optimization module 16 is connected to the part imaging module 17, which is used to display the optimized part information on the wearable terminal 1.

[0055] As a preferred solution, in order to ensure the user's immersive virtual simulation interaction experience, the connection between the display unit 5 and the motion conversion unit 6 is optimized. The motion conversion unit 6 is used to recognize hand movements and trigger the corresponding operation function of the displayed parts. Specifically, the motion conversion unit 6 includes a hand image acquisition module 18, a joint marking module 19, a gesture recognition module 20, a gesture-command reference library 21, and a command triggering module 22. The hand image acquisition module 18 is a binocular vision camera set on the wearing terminal 1 and is used to acquire hand images of the wearer of the wearing terminal 1.

[0056] The hand image acquisition module 18 is connected to the joint marking module 19, which is used to mark the joints of the hand fingers.

[0057] The joint marking module 19 is connected to the gesture recognition module 20. The gesture recognition module 20 is used to generate corresponding gestures based on the connection of joint point markings and to perform gesture recognition.

[0058] The gesture recognition module 20 is connected to the gesture-command lookup library 21. The gesture-command lookup library 21 is used to establish the mapping relationship between different gestures and operation functions in the virtual interaction platform 2, and to establish specified gesture trigger commands and gesture end commands. The gesture trigger command is used to start the gesture control operation function, and the gesture end command is used to close the gesture control operation function and continue the operation function corresponding to the gesture before the gesture end command.

[0059] The gesture-command lookup library 21 is connected to the command triggering module 22. The command triggering module 22 is used to search for the corresponding operation function in the gesture-command lookup library 21 based on the gesture recognized by the gesture recognition module 20, and trigger it to operate the displayed parts according to the operation function.

[0060] In this embodiment, an intelligent and lightweight display of the virtual simulation environment can be achieved.

[0061] Example 2

[0062] This embodiment is an improvement on the previous embodiment. An immersive virtual simulation interaction system is provided. The virtual interaction platform 2 includes a user adaptation unit 3. The user adaptation unit 3 is used to adjust the size to fit the user's perspective. Specifically, the user adaptation unit 3 includes a reference input module 7, a perspective adjustment module 8, and an identity comparison module 9. The reference input module 7 is used to obtain the external environment image directly in front of the user from the current perspective. After keeping the body angle fixed, the user wears the terminal 1 and displays the external environment image on the terminal 1.

[0063] The reference input module 7 is connected to the viewing angle adjustment module 8. The viewing angle adjustment module 8 is used to adjust the size of the external environment image displayed in the wearable terminal 1 so that the size of the external environment image displayed in the wearable terminal 1 is consistent with the external environment image when the wearable terminal 1 is not worn.

[0064] The viewing angle adjustment module 8 is connected to the identity comparison module 9. The identity comparison module 9 is used to establish and store the mapping relationship between size adjustment information and user identity information.

[0065] The advantage of Example 2 over Example 1 is that it can ensure a high degree of fit between the virtual simulation environment obtained by the user and the real environment obtained when not wearing the terminal 1.

[0066] The usage of an immersive virtual simulation interactive system includes the following steps:

[0067] Step 1, Viewpoint Adjustment: Obtain the external environment image directly in front of the user from the current viewpoint. After keeping the body angle fixed, wear the wearing terminal 1. Refer to the input module 7 to display the external environment image on the wearing terminal 1. The viewpoint adjustment module 8 adjusts the size of the external environment image displayed on the wearing terminal 1 so that the size of the external environment image displayed on the wearing terminal 1 matches the external environment image when the wearing terminal 1 is not worn. Then, the identity verification module 9 establishes a mapping relationship between the size adjustment information and the user's identity information and stores it.

[0068] Step 2, Data Analysis: The data analysis module 10 divides the real scene information to be simulated into several parts, and then performs feature association information of several parts based on the real scene information. The real scene is then assembled in the form of an assembly. The assembly storage module 11 records and stores the real scene assembly information. The part information storage module 12 stores the part size and name information. The association information storage module 13 stores the feature association information of several parts. The single association information storage module 14 filters the single feature association information and stores it as a necessary feature.

[0069] Step 3, Optimize Display: When the user selects a part with a specified name, the data filtering module 15 obtains the information of the corresponding part from the part information storage module 12, the data optimization module 16 obtains the single feature association information when the corresponding part is displayed alone, removes the part that is repeated with the necessary feature from the single feature association information, optimizes the remaining single feature association information, obtains the optimized part information, and the part imaging module 17 displays the optimized part information in the wearable terminal 1.

[0070] Step 4: The hand image acquisition module 18, acting as a binocular vision camera mounted on the wearable terminal 1, acquires hand images of the wearer. The joint marking module 19 marks the joints of the hand fingers. The gesture recognition module 20 generates corresponding gestures based on the joint markings and performs gesture recognition. The gesture-command lookup library 21 is used to establish a mapping relationship between different gestures and operation functions in the virtual interaction platform 2, and to establish specified gesture trigger commands and gesture end commands. The gesture trigger command is used to start the gesture control operation function, and the gesture end command is used to close the gesture control operation function and continue the operation function corresponding to the gesture before the gesture end command. The command trigger module 22 is used to search for the corresponding operation function in the gesture-command lookup library 21 based on the gesture recognized by the gesture recognition module 20, and trigger it to operate the displayed parts according to the operation function.

[0071] As an extension, this application can also be compatible with multiple device terminals, such as digital screens, mobile phone terminals, and computer terminals, in addition to the wearable terminal 1.

Claims

1. An immersive virtual simulation interaction system, comprising a wearable terminal (1) and a virtual interaction platform (2), wherein the wearable terminal (1) and the virtual interaction platform (2) are connected, characterized in that: The wearable terminal (1) is a VR wearable terminal. The virtual interaction platform (2) includes a user adaptation unit (3), a database (4), an optimization display unit (5), and an action conversion unit (6). The user adaptation unit (3) is used to adjust the size to fit the user's perspective. The database (4) is used to store data of real external scenes composed of several parts forming an assembly. The database (4) is connected to the optimization display unit (5). The optimization display unit (5) is used to optimize the structure of the parts when displaying them. The optimization display unit (5) is connected to the action conversion unit (6). The action conversion unit (6) is used to recognize hand gestures and trigger the corresponding operation functions of the displayed parts.

2. The immersive virtual simulation interaction system according to claim 1, characterized in that: The user adaptation unit (3) includes a reference input module (7), a perspective adjustment module (8), and an identity comparison module (9). The reference input module (7) is connected to the perspective adjustment module (8), and the perspective adjustment module (8) is connected to the identity comparison module (9).

3. The immersive virtual simulation interaction system according to claim 2, characterized in that: The reference input module (7) is used to obtain the external environment image directly in front of the user from the current perspective. After keeping the body angle fixed, the user wears the terminal (1) and displays the external environment image in the terminal (1). The viewing angle adjustment module (8) is used to adjust the size of the external environment image displayed in the wearable terminal (1) so that the size of the external environment image displayed in the wearable terminal (1) fits the external environment image when the wearable terminal (1) is not worn. The identity verification module (9) is used to establish and store the mapping relationship between size adjustment information and user identity information.

4. The immersive virtual simulation interaction system according to claim 3, characterized in that: The database (4) includes a data analysis module (10), an assembly storage module (11), a part information storage module (12), an association information storage module (13), and a single association information storage module (14). The data analysis module (10) is connected to the assembly storage module (11). The assembly storage module (11) is connected to the part information storage module (12) and the association information storage module (13) respectively. The association information storage module (13) is connected to the single association information storage module (14).

5. The immersive virtual simulation interaction system according to claim 4, characterized in that: The data analysis module (10) is used to divide the real scene information that needs to be simulated and displayed into several parts, and then perform feature association information of several parts based on the real scene information, and then assemble the real scene in the form of an assembly. The assembly storage module (11) is used to record and store assembly information in real-world scenarios; The part information storage module (12) is used to store part size and name information; The associated information storage module (13) is used to store the feature association information of several parts; The single association information storage module (14) is used to filter single feature association information and store it as a necessary feature.

6. The immersive virtual simulation interaction system according to claim 5, characterized in that: The optimized display unit (5) includes a data filtering module (15), a data optimization module (16), and a part imaging module (17). The data filtering module (15) is connected to the data optimization module (16), and the data optimization module (16) is connected to the part imaging module (17).

7. An immersive virtual simulation interaction system according to claim 6, characterized in that: The data filtering module (15) is used to retrieve information about the corresponding part from the database (4) when the user selects a part with a specified name; The data optimization module (16) is used to obtain the single feature association information when the corresponding part is displayed alone, remove the part that is repeated with the necessary feature in the single feature association information, optimize the remaining single feature association information, and obtain the optimized part information. The part imaging module (17) is used to display the optimized part information on the wearable terminal (1).

8. The immersive virtual simulation interaction system according to claim 7, characterized in that: The action conversion unit (6) includes a hand image acquisition module (18), a joint marking module (19), a gesture recognition module (20), a gesture-command reference library (21), and a command triggering module (22). The hand image acquisition module (18) is connected to the joint marking module (19), the joint marking module (19) is connected to the gesture recognition module (20), the gesture recognition module (20) is connected to the gesture-command reference library (21), and the gesture-command reference library (21) is connected to the command triggering module (22).

9. An immersive virtual simulation interaction system according to claim 8, characterized in that: The hand image acquisition module (18) is a binocular vision camera set on the wearing terminal (1) to acquire hand images of the wearer of the wearing terminal (1); The joint marking module (19) is used to mark the joints of the fingers; The gesture recognition module (20) is used to generate corresponding gestures based on the connection of key point marks and to perform gesture recognition; The gesture-command reference library (21) is used to establish the mapping relationship between different gestures and the operation functions in the virtual interaction platform (2), and to establish specified gesture triggering instructions and gesture ending instructions. The gesture triggering instructions are used to start the gesture control operation function, and the gesture ending instructions are used to close the gesture control operation function and continue the operation function corresponding to the gesture before the gesture ending instructions. The instruction triggering module (22) is used to search for the corresponding operation function from the gesture-instruction lookup library (21) based on the gesture recognized by the gesture recognition module (20), and trigger it to operate the displayed parts according to the operation function.

10. An immersive virtual simulation interaction system according to claim 9, characterized in that: The method of using the system includes the following steps: Step 1, Viewpoint Adjustment: Obtain the external environment image directly in front of the user from the current viewpoint. After keeping the body angle fixed, wear the wearing terminal (1). Refer to the input module (7) to display the external environment image in the wearing terminal (1). The viewpoint adjustment module (8) adjusts the size of the external environment image displayed in the wearing terminal (1) so that the size of the external environment image displayed in the wearing terminal (1) matches the external environment image when the wearing terminal (1) is not worn. Then the identity comparison module (9) establishes the mapping relationship between the size adjustment information and the user identity information and stores it. Step 2, Data Analysis: The data analysis module (10) divides the real scene information to be simulated into several parts, and then performs feature association information of several parts based on the real scene information. The real scene is then assembled in the form of an assembly. The assembly storage module (11) records and stores the real scene assembly information. The part information storage module (12) stores the part size and name information. The association information storage module (13) stores the feature association information of several parts. The single association information storage module (14) filters the single feature association information and stores it as a necessary feature. Step 3, Optimize the display: When the user selects a part with a specified name, the data filtering module (15) obtains the information of the corresponding part from the part information storage module (12), the data optimization module (16) obtains the single feature association information when the corresponding part is displayed alone, removes the part that is repeated with the necessary feature from the single feature association information, optimizes the remaining single feature association information, obtains the optimized part information, and the part imaging module (17) displays the optimized part information in the wearable terminal (1); Step 4: The hand image acquisition module (18) is a binocular vision camera set on the wearable terminal (1) to acquire the hand image of the wearer of the wearable terminal (1). The joint marking module (19) marks the joints of the hand. The gesture recognition module (20) generates the corresponding gesture based on the joint marking and performs gesture recognition. The gesture-instruction lookup library (21) is used to establish the mapping relationship between different gestures and the operation functions in the virtual interaction platform (2), and to establish the specified gesture triggering instruction and gesture ending instruction. The gesture triggering instruction is used to start the gesture control operation function, and the gesture ending instruction is used to close the gesture control operation function and continue the operation function corresponding to the gesture before the gesture ending instruction. The instruction triggering module (22) is used to search for the corresponding operation function in the gesture-instruction lookup library (21) based on the gesture recognized by the gesture recognition module (20) and trigger it to operate the displayed parts according to the operation function.

Images