Virtual reality interaction method and apparatus, system, storage medium, and electronic device

Through the combination of posture tracking equipment and lidar components, the full-body posture information collection and accurate mapping of virtual objects in the virtual reality system is achieved, solving the problem that virtual reality experience is limited by single-person scenarios in the existing technology, and improving the Virtual reality experience.

WO2024244812A1PCT designated stage expired Publication Date: 2024-12-05BOE TECHNOLOGY GROUP CO LTD +2

Patent Information

Application Number
PCT/CN2024/088888
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-29
Filing Date
2024-04-19
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

The existing virtual reality system cannot achieve full-body tracking, resulting in the virtual reality experience being limited to single-person scenes, and the virtual reality experience being unable to maximize the virtual reality experience.

Method used

By using the attitude tracking device to collect user's body posture information, combine lidar components and inertial measurement unit components, obtain user's full-body posture information, and accurately map the user's virtual objects in a virtual reality environment to achieve full-body interaction.

Benefits of technology

It improves the accuracy of the interactive actions of virtual objects in the virtual reality environment, enhances the user experience, realizes full-body tracking and participates in the interaction of the virtual environment, and promotes the improvement of the virtual reality experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088888_05122024_PF_FP_ABST
    Figure CN2024088888_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates in particular to a virtual reality interaction method and apparatus, a system, a storage medium, and an electronic device. The method comprises: acquiring body posture information of a user in a real environment by using a posture tracking device, the posture tracking device comprising a motion detection unit arranged corresponding to the position of target skeletal muscle, and the motion detection unit being used for determining the joint motion angle of a corresponding joint; acquiring first position information of the head of the user in the real environment by using a laser radar assembly, and acquiring head posture information of the head of the user in the real environment by using an inertial measurement unit assembly; and configuring the body posture information, the head position information, and the head posture information as full-body pose information of the user, so as to, according to the full-body pose information, map a virtual object corresponding to the user in a virtual reality environment. According to the present solution, the full-body pose information of the user may be accurately acquired, and interaction in the same virtual reality environment between multiple users in different real scenes is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Virtual reality interaction method and device, system, storage medium, and electronic device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority from application number 202310620062.1, application date May 29, 2023, and invention title: Virtual Reality Interaction Method and Device, System, Storage Medium, and Electronic Device. The entire contents of this Chinese patent application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of virtual reality technology, and in particular to a virtual reality interaction method, a virtual reality interaction device, a virtual reality system, a storage medium, and an electronic device. Background Art

[0004] With the rapid development of virtual reality (VR) technology, it has found widespread application in exhibitions, virtual training, e-sports, industrial simulation, and other fields. Taking VR gaming as an example, existing VR systems generally track the head and hands through input devices, but cannot achieve full-body tracking and incorporate tracking results into virtual environment interactions. Furthermore, current VR systems primarily focus on single-player scenarios, failing to maximize the virtual reality experience.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field.

[0006] Summary of the Invention

[0007] According to the present disclosure, a virtual reality interaction method is provided, the method comprising:

[0008] Collecting body posture information of the user in a real environment using a posture tracking device; wherein the posture tracking device includes a motion detection unit provided corresponding to the target skeletal muscle position, the motion detection unit being used to determine the joint motion angle of the corresponding joint;

[0009] Using a laser radar component to collect first position information of the user's head in the real environment; and using an inertial measurement unit component to collect head posture information of the user's head in the real environment;

[0010] The body posture information, head position information, and head posture information are configured as the user's full-body posture information, so as to be used for mapping a virtual object corresponding to the user in a virtual reality environment according to the full-body posture information.

[0011] In some exemplary embodiments, the motion detection unit includes: a skin tension strain gauge and an electromyographic signal electrode for respectively calculating the joint motion angle.

[0012] In some exemplary embodiments, the method further comprises:

[0013] The magnitude and direction of skeletal muscle stress data are collected through skin tension strain gauges;

[0014] Collect corresponding electromyographic current data through electromyographic signal electrodes;

[0015] The preset data table is queried according to the electromyographic current data and the magnitude and direction data of the skeletal muscle stress to determine the joint motion angle of the corresponding joint.

[0016] In some exemplary embodiments, the method includes:

[0017] Acquire a user image of a user, and identify the user image to acquire a corresponding body region image;

[0018] The body region image is calibrated to obtain length parameters corresponding to each joint.

[0019] In some exemplary embodiments, the method includes:

[0020] The whole-body posture information corresponding to the user is sent to the server side, so that the server side displays the virtual object of the user based on the whole-body posture information in the virtual reality environment.

[0021] In some exemplary embodiments, the method includes:

[0022] configuring a user virtual space for a virtual object corresponding to the user in the virtual reality environment;

[0023] A virtual object of the user is displayed in the user virtual space based on the full-body posture information.

[0024] In some exemplary embodiments, the method further comprises:

[0025] Calibrate the sub-coordinate system corresponding to the user virtual space and the virtual reality environment coordinate system to determine a coordinate transformation matrix between the sub-coordinate system corresponding to the user virtual space and the virtual reality environment coordinate system;

[0026] Based on the coordinate transformation matrix, and according to the full body posture information of the virtual object corresponding to the user in the user virtual space, the full body posture information of the virtual object corresponding to the user in the virtual reality environment coordinates is determined.

[0027] In some exemplary embodiments, the lidar assembly and the inertial measurement unit assembly are disposed on a VR helmet worn by a user.

[0028] According to a second aspect of the present disclosure, a virtual reality interaction device is provided, comprising:

[0029] A posture tracking device for collecting body posture information of a user in a real environment; wherein the posture tracking device includes a motion detection unit provided corresponding to the target skeletal muscle position, the motion detection unit being used to determine the joint motion angle of the corresponding joint;

[0030] A laser radar component is used to collect the first position information of the user's head in the real environment;

[0031] An inertial measurement unit component, used to collect head posture information of the user's head in the real environment;

[0032] A full-body posture calculation module is used to configure the body posture information, head position information, and head posture information into the user's full-body posture information, so as to map the virtual object corresponding to the user in a virtual reality environment according to the full-body posture information.

[0033] According to a third aspect of the present disclosure, a virtual reality interaction system is provided, comprising:

[0034] A user terminal device is used to collect user data, wherein the user data includes full-body posture information of the user in a real environment; wherein the full-body posture information includes head position information, head posture information, and body posture information;

[0035] The server side is used to obtain user data of multiple users, and according to the mapping relationship between each user and the virtual object in the virtual reality environment, display the virtual object corresponding to the user based on the full-body posture information in the virtual reality environment.

[0036] According to a fourth aspect of the present disclosure, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the virtual reality interaction method described above is implemented.

[0037] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0038] processor; and

[0039] a memory for storing executable instructions of the processor;

[0040] Wherein, the processor is configured to implement the above-mentioned virtual reality interaction method by executing the executable instructions.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0043] FIG1 schematically shows a schematic diagram of an exemplary system architecture applying the technical solution of an embodiment of the present invention in an exemplary embodiment of the present disclosure;

[0044] FIG2 schematically illustrates a virtual reality interaction method according to an exemplary embodiment of the present disclosure;

[0045] FIG3 schematically illustrates a user wearing a VR helmet according to an exemplary embodiment of the present disclosure;

[0046] FIG4 schematically shows a wearing diagram of a posture tracking device in an exemplary embodiment of the present disclosure;

[0047] FIG5 schematically illustrates a mapping relationship between a real space coordinate system and a user virtual space coordinate system in an exemplary embodiment of the present disclosure;

[0048] FIG6 schematically illustrates a schematic diagram of a hand tracking coordinate system conversion in an exemplary embodiment of the present disclosure;

[0049] FIG7 schematically shows a system block diagram in an exemplary embodiment of the present disclosure;

[0050] FIG8 schematically shows a schematic diagram of coordinate transformation for multi-user interaction in an exemplary embodiment of the present disclosure;

[0051] FIG9 schematically shows a diagram of a virtual reality interaction device in an exemplary embodiment of the present disclosure;

[0052] FIG10 schematically shows a diagram of a virtual reality interaction system in an exemplary embodiment of the present disclosure;

[0053] FIG11 schematically shows a composition diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0055] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] In related technologies, virtual reality applications are primarily focused on single-player scenarios, which cannot maximize the VR experience. Conventional VR systems can only track the head and hands, but in real life, our entire body can interact with the surrounding environment. Therefore, achieving full-body tracking and incorporating tracking results into virtual environment interactions can greatly enhance the VR experience.

[0057] In view of the shortcomings and deficiencies of the existing technology, this example embodiment provides a virtual reality interaction method that can be applied to the application scenarios of virtual reality interaction. Referring to FIG2 , the provided virtual reality interaction method may include:

[0058] Step S11, using a posture tracking device to collect body posture information of the user in a real environment; wherein the posture tracking device includes a motion detection unit set corresponding to the distribution of human skeletal muscles, and the motion detection unit is used to determine the joint motion angle of the corresponding joint;

[0059] Step S12, using a laser radar component to collect first position information of the user's head in the real environment; and using an inertial measurement unit component to collect head posture information of the user's head in the real environment;

[0060] Step S13: configuring the body posture information, head position information, and head posture information into full-body posture information of the user, so as to map a virtual object corresponding to the user in a virtual reality environment according to the full-body posture information.

[0061] The virtual reality interaction method provided in this example embodiment, by setting up a posture tracking device, can use multiple motion detection units to collect the joint movement angles of each joint of the user's torso, so that the body posture information can be determined based on the joint movement angles; at the same time, the laser radar component is used to accurately collect the user's first position information in the real environment, and the inertial measurement unit component is used to collect the head posture information of the user's head in the real environment, so as to obtain the user's full-body posture information; using this full-body posture information, the user's virtual object movements can be accurately mapped in the virtual reality environment, thereby improving the accuracy of the virtual object's interactive movements in the virtual reality environment, and then improving the user's usage experience.

[0062] Below, each step of the virtual reality interaction method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0063] In step S11, a posture tracking device is used to collect body posture information of the user in a real environment; wherein, the posture tracking device includes a motion detection unit set corresponding to the target skeletal muscle distribution, and the motion detection unit is used to determine the joint motion angle of the corresponding joint.

[0064] In this exemplary embodiment, the motion detection unit includes: a skin tension strain gauge and an electromyographic signal electrode for respectively calculating the joint motion angle.

[0065] In this example embodiment, the method also includes: collecting the magnitude and direction data of skeletal muscle stress through a skin tension strain gauge; collecting the corresponding electromyographic current data through an electromyographic signal electrode; and querying a preset data table based on the electromyographic current data and the magnitude and direction data of skeletal muscle stress to determine the joint motion angle of the corresponding joint.

[0066] Specifically, on the user terminal side, a posture tracking device can be provided for each user. This posture tracking device can be a wearable system; for example, it can include a tights with motion detection units arranged on the tights corresponding to the distribution of skeletal muscles in the human body. The target skeletal muscles mentioned above can be all or part of the skeletal muscles selected from the human body.

[0067] In some embodiments, when accurate posture information of each joint of the user needs to be obtained, motion detection units can be placed at each skeletal muscle location, or at the main skeletal muscle locations of the human torso, corresponding to the distribution of skeletal muscles of the human body. Of course, in some scenarios, if only brief posture information of the main torso of the human body is needed, motion detection units can be placed at pre-selected torso locations and skeletal muscle locations corresponding to joints; for example, motion detection units can be placed at the skeletal muscle locations corresponding to the elbow joint, wrist joint, knee joint, and ankle joint.

[0068] Alternatively, to obtain more accurate joint motion angle data, multiple detection units can be deployed at a single motion joint. For example, referring to FIG4 , a first elbow joint motion detection unit 41 and a second elbow joint motion detection unit 42 can be deployed at the elbow joint. The motion detection unit primarily comprises a skin tension strain gauge 401 and an electromyographic signal electrode 402. The motion detection unit may include a microcontroller unit (MCU), which can be connected to the electromyographic signal electrode and the skin tension strain gauge via a bandpass amplifier. The MCU can be used to perform A / D conversion on the received signals and calculate the joint motion angle. Specifically, during skeletal muscle contraction or relaxation, the skin tension strain gauge deforms due to the contraction or relaxation of the skin, generating pressure or tension. The magnitude of the force generated varies with the degree of contraction or relaxation of the skeletal muscle. Therefore, the magnitude and direction of the force (pressure or tension) can be used to determine the motion of the skeletal muscle and, consequently, the motion angle of the corresponding joint. EMG electrodes work similarly to skin tension strain gauges. When skeletal muscle contracts, it generates an EMG current, which changes with the degree of muscle contraction. Using EMG electrodes to detect this current, the degree of skeletal muscle contraction can be determined, and the corresponding joint movement angle can be derived from the current. Skin tension strain gauges and EMG electrodes appear to have the same effect, but in reality, EMG signals are only generated on the contracting side of the muscle, not on the relaxing side. However, skin tension strain gauges still generate tension when the muscle relaxes.

[0069] In order to obtain accurate joint angle information, data collection can be performed in advance to record the joint angle sizes corresponding to the force size and the electromyographic current size, and a data table can be established based on the collected data. For the currently obtained electromyographic current and skin stress values, the corresponding joint angle data can be determined by querying the data table. If the corresponding electromyographic current value or stress size value does not exist in the data table, the joint angle corresponding to the electromyographic current value and stress size value with the smallest numerical error can be selected from the data table as the final joint angle data. In this way, the motion angles of the joints in various parts of the human body can be obtained by arranging the motion detection units in various parts of the human body, and then the whole body posture can be obtained. When multiple groups of motion detection units are set at the same joint, multiple groups of joint angle data can be obtained, and the final joint angle data can be obtained by calculating the average value.

[0070] In step S12, the laser radar component is used to collect first position information of the user's head in the real environment; and the inertial measurement unit component is used to collect head posture information of the user's head in the real environment.

[0071] In this example embodiment, the laser radar component and the inertial measurement unit component are arranged on a VR helmet worn by the user.

[0072] In this example embodiment, when users use VR devices to interact with virtual reality, each user can pre-select a real space of a preset size in the real environment, so that the user's posture data in the real space can be collected. Specifically, for each user, when using VR devices to interact with virtual reality, a real space of a uniform size can be pre-selected in the real environment as the user's experience space in the real environment. In the application scenario where multiple people interact with the same virtual scene in different real environments, each user needs to find an experience space in his or her own real space, select a fixed position in the experience space, and calibrate the coordinate system of the experience space based on the fixed position, so as to unify the scale standards among the users. Since the length and width of the real space selected by each user are consistent, correspondingly, a user virtual space of the same size and area can be mapped to each user's virtual object in the virtual reality environment, which will be more conducive to tracking the real-time posture of the user in the entire virtual reality environment.

[0073] Specifically, as shown in Figure 3, a VR helmet includes VR glasses 32. A multi-line laser radar 31 can be installed on an existing VR helmet to achieve 360-degree omnidirectional tracking of the user's head. Current VR systems primarily use binocular cameras for vSLAM to track the user's head position, but are unable to achieve 360-degree omnidirectional tracking. This solution, however, uses a multi-line laser radar to determine the coordinates of the user's head in the real environment relative to a geodesic reference point. Furthermore, the advantage of using a laser radar instead of a binocular camera is that no matter how much the user rotates relative to the reference point, the laser radar can directly capture the user's current position relative to the reference point. This is because the laser radar detects the distance to obstacles in the surrounding horizontal direction at a constant frequency in real time, allowing for faster and more direct calculation of the user's position in the reference coordinates. In existing solutions, when the binocular camera is facing away from the reference point, the user's rotation angle relative to the reference point can only be calculated using the IMU, indirectly deriving the user's head position information. This indirect calculation inevitably leads to greater errors. In addition, multi-line LiDAR can be used to obtain information about the height of the user's head. It can detect surrounding horizontal and vertical obstacles in real time, and thus determine the head position coordinates (x, y, z) of the user wearing the VR headset. This allows the VR headset to track the user's head position. Combined with the VR headset's IMU component, the user's head posture coordinates (α, β, γ) can be obtained in real time. These are the rotation angles around the x, y, and z axes, respectively. This allows tracking the user's head posture information. The multi-line LiDAR combined with the IMU can obtain the six-degree-of-freedom pose coordinates of the user's head.

[0074] In step S13, the body posture information, head position information, and head posture information are configured as full-body posture information of the user, so as to be used for mapping a virtual object corresponding to the user in a virtual reality environment according to the full-body posture information.

[0075] In this example implementation, pose information generally includes the position and pose information of an object within a coordinate system. For each user, the first position information of the head within the experience space, collected by the LiDAR assembly, can be used as the user's full-body position information. Furthermore, the head pose information and body pose information can be used as the user's full-body pose information. Based on the currently collected body pose information, head position information, and head pose information, the user's full-body pose information can be configured.

[0076] For terminal devices, after collecting full-body posture information, the data can be sent to the virtual reality server; thus, the virtual objects of each user can be mapped in the virtual reality environment established on the server side.

[0077] In this example embodiment, the method includes: sending the full-body posture information corresponding to the user to the server, so that the server displays the virtual object of the user based on the full-body posture information in the virtual reality environment.

[0078] In this example implementation, reference is made to FIG1 , which illustrates a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present invention can be applied. As shown in FIG1 , the system architecture may include terminal devices (e.g., 101 and 102 in FIG1 ), a network 104, and a server 103. The terminal devices are intelligent VR devices worn by users, including VR helmets, VR controllers, body posture detection devices, and other VR devices, used to immerse users in a virtual reality environment. Network 104 is used as a medium to provide a communication link between the terminal devices and server 105. Network 104 may include various connection types, such as wired communication links, wireless communication links, and so on. It should be understood that the number of terminal devices, networks, and servers shown in FIG1 is merely illustrative. Any number of terminal devices, networks, and servers may be present depending on implementation needs. For example, server 103 may be a server cluster consisting of multiple servers. For example, the content of the aforementioned virtual reality interaction method may be implemented on the terminal device side. The full-body posture information collected by the user side may be sent to the server side, where the virtual reality environment and each user's virtual objects are established, and the movements of the virtual objects are controlled based on the full-body posture information.

[0079] In this example embodiment, the method includes: configuring a user virtual space for a virtual object corresponding to the user in the virtual reality environment;

[0080] A virtual object of the user is displayed in the user virtual space based on the full-body posture information.

[0081] Specifically, in a virtual reality environment, a user virtual space can be mapped to each user's real space, and virtual objects corresponding to the user can be displayed in the user virtual space. The coordinate system conversion relationship between the user's user virtual space and the virtual reality system can also be determined.

[0082] As shown in Figure 5, based on the scenario requirements, a corresponding user virtual space is divided for each user within the virtual reality environment coordinate system XYZ. A reference point is selected for each user virtual space and fixed at a fixed position in the virtual display environment. Based on this reference point, a sub-coordinate system for that user virtual space is established, such as the coordinate systems X0Y0Z0, X1Y1Z1, X2Y2Z2, and X3Y3Z3. The position of each sub-coordinate system relative to the virtual environment coordinate system XYZ is fixed. This allows the position of each user's virtual object in the virtual reality environment relative to the reference point (coordinate system XOY) to be determined. The mapping process in Figure 5 only illustrates the mapping of the three degrees of freedom (position).

[0083] In this example embodiment, the method includes: calibrating the sub-coordinate system corresponding to the user virtual space and the virtual reality environment coordinate system to determine a coordinate transformation matrix between the sub-coordinate system corresponding to the user virtual space and the virtual reality environment coordinate system;

[0084] Based on the coordinate transformation matrix, and according to the full body posture information of the virtual object corresponding to the user in the user virtual space, the full body posture information of the virtual object corresponding to the user in the virtual reality environment coordinates is determined.

[0085] Specifically, the head-mounted device worn by the user can track the dynamic position of the user's head (e.g., coordinate system x0oy0z0) relative to a reference point in the virtual reality space (e.g., coordinate system X0Y0Z0). The motion detection unit can capture the rotation angles of the user's joints throughout their body. This allows the user's full-body posture to be mapped to a virtual object within the virtual environment, allowing the user to be mapped into the virtual display environment. As the user's movements change, the dynamic changes of the virtual objects relative to the virtual reality space can also be captured, allowing each user's virtual space to be mapped into the same virtual reality environment.

[0086] For the position and posture of a virtual object in a virtual reality environment, taking the tracking of a user's hand posture relative to the virtual space as an example, as shown in Figure 6, it is necessary to determine the relative coordinates of the coordinate system x5o5y5 relative to the coordinate system XOY. For example:

[0087] in, It is the coordinate transformation matrix of coordinate system x5o5y5 relative to coordinate system XOY.

[0088] For example, the transformation matrix The calculation process may include:

[0089] in, is the coordinate transformation matrix of θ1 relative to o4; cθ1 is cosθ1; sθ1 is

[0090] sinθ1; l1 is the length of the upper arm; l2 is the length of the forearm; θ1 is the rotation angle of the shoulder joint; θ2 is the rotation angle of the elbow joint; l1 and l2 are obtained by taking pictures of the human body, and θ1 and θ2 are obtained by the motion detection unit.

[0091] In this example implementation, the method includes: acquiring a user image of a user, identifying the user image to acquire a corresponding body region image; and calibrating the body region image to acquire length parameters corresponding to each joint.

[0092] For example, referring to FIG7 , taking two users in different real-world locations as an example, when using the virtual reality interactive system, the users can first upload their full-body photos to the terminal device or server. This is used to calculate the length information between the body's joints, which serves as the basis for kinematic calculations. The motion detection unit obtains the rotation angles of the joints at all parts of the body. Combined with the IMU and lidar, the relative pose coordinates of the user's body relative to the real-world reference point can be obtained. The wearable system's AP virtualizes the real scene and the user. Because this virtualized scene only includes the processing of a single user, a local space virtual image is obtained. The AP then transmits the local space virtual image to a remote virtual reality reconstruction server via the network. The virtual reality reconstruction server then reconstructs the local space virtual images sent by all users into a global space virtual scene based on the experience scene selected by the player. The global space virtual scene is then distributed to each user's AP for virtual reality display. In this way, each user can see the virtual images of other users in the virtual reality display. Through this interactive mode, users in different real-world locations can jointly participate in the interactive experience of some multiplayer virtual games.

[0093] For example, referring to Figure 8, using a multiplayer soccer game as an example, in a virtual reality environment, a corresponding user virtual space can be created for each user's virtual object; this virtual reality scene can correspond to each user's real space. In the initial state, the initial position of the user's virtual object can be displayed in the user virtual space. The soccer ball is a virtual object in the virtual environment. To control the toe of the corresponding virtual object to kick the virtual soccer ball, the user must know the specific position of the virtual character's foot in the virtual environment. Of course, the position of the virtual soccer ball in the virtual environment is known. This solution can track the user's entire body posture, so the movement state of the corresponding virtual object can be controlled by the user's entire body posture, for example, to implement kicking actions or goalkeeper defensive movements. Furthermore, the force of the kick can be varied based on the amplitude of the user's leg swing. This force allows the virtual soccer ball to move between the user virtual spaces of different users in the virtual environment, thus implementing a passing action. This process enables multiplayer interaction in the virtual scene.

[0094] The virtual reality interaction method provided by the embodiment of the present disclosure realizes real-time and accurate collection of the user's head position information, head posture information, and body posture information in the real space by assembling a laser radar on the VR helmet and the user wearing a posture tracking device, thereby realizing 6DOF tracking of the whole body posture. The realization of full-body tracking and incorporating the tracking results into the interaction of the virtual environment greatly promotes the improvement of the virtual reality experience. By creating corresponding virtual objects for each user in the virtual reality environment, each user's real space is mapped to the corresponding user virtual space, so that users in different locations can interact in the same virtual reality environment.

[0095] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0096] Furthermore, referring to FIG9 , this example embodiment also provides a virtual reality interaction device 90 that can be applied to a virtual reality user terminal device; the device includes: a posture tracking device 901, a laser radar component 902, an inertial measurement unit component 903, and a full-body posture calculation module 904. Among them:

[0097] The posture tracking device 901 can be used to collect body posture information of the user in a real environment; wherein, the posture tracking device includes a motion detection unit set corresponding to the target skeletal muscle position, and the motion detection unit is used to determine the joint motion angle of the corresponding joint.

[0098] The laser radar component 902 can be used to collect first position information of the user's head in the real environment.

[0099] The inertial measurement unit component 903 can be used to collect head posture information of the user's head in a real environment.

[0100] The whole-body posture calculation module 904 can be used to configure the body posture information, head position information, and head posture information into the user's whole-body posture information, so as to map the virtual object corresponding to the user in the virtual reality environment according to the whole-body posture information.

[0101] Furthermore, referring to FIG10 , a virtual reality interaction system is also provided in the embodiment of this example. The system includes: a server 11 and a user terminal device 12 .

[0102] The user terminal device 12 can be used to collect user data, and the user data includes the user's full body posture information in the real environment; wherein the full body posture information includes head position information, head posture information, and body posture information.

[0103] The server end 11 can be used to obtain user data of multiple users, and according to the mapping relationship between each user and the virtual object in the virtual reality environment, display the virtual object corresponding to the user based on the full-body posture information in the virtual reality environment.

[0104] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0105] FIG. 11 shows a schematic diagram of an electronic device suitable for implementing an embodiment of the present invention.

[0106] It should be noted that the electronic device 1000 shown in FIG11 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0107] As shown in Figure 11, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. For example, the central processing unit 101 can execute the steps shown in Figure 12 to implement the above-mentioned pedestrian detection method. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, ROM 1002 and RAM 1003 are connected to each other via a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0108] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed.

[0109] In particular, according to an embodiment of the present invention, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a storage medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009 and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the various functions defined in the system of the present application are performed.

[0110] Specifically, the electronic device can be a smart device such as a server, tablet computer, or laptop computer, and can execute the aforementioned method for managing IoT device interactions applied to a proxy server or IoT platform. Alternatively, the electronic device can be an IoT device and can execute the aforementioned method for managing IoT device interactions applied to an IoT device.

[0111] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units involved in the embodiments of the present invention may be implemented in software or hardware, and the units described may also be provided in a processor. In some cases, the names of these units do not limit the units themselves.

[0114] It should be noted that, as another aspect, the present application also provides a storage medium, which can be included in an electronic device or can exist independently without being installed in the electronic device. The above-mentioned storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement each step of the method applied to a proxy server, an IoT platform, or an IoT device.

[0115] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0116] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0117] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A virtual reality interaction method, characterized in that: The method comprises: Using a posture tracking device to collect body posture information of the user in a real environment; wherein the posture tracking device includes a motion detection unit arranged corresponding to the position of the target skeletal muscle, and the motion detection unit is used to determine the joint motion angle of the corresponding joint; Using a laser radar component to collect first position information of the user's head in the real environment; and using an inertial measurement unit component to collect head posture information of the user's head in the real environment; The body posture information, head position information, and head posture information are configured as the whole-body posture information of the user, so as to be used for mapping a virtual object corresponding to the user in a virtual reality environment according to the whole-body posture information.

2. The virtual reality interaction method according to claim 1, characterized in that: The motion detection unit comprises: a skin tension strain gauge and an electromyographic signal electrode for respectively calculating the joint motion angle.

3. The virtual reality interaction method according to claim 2, characterized in that: The method further comprises: The magnitude and direction data of skeletal muscle stress are collected through skin tension strain gauges; Collect corresponding electromyographic current data through electromyographic signal electrodes; The preset data table is queried according to the electromyographic current data, the magnitude and direction data of the skeletal muscle stress, and the joint motion angle of the corresponding joint is determined.

4. The virtual reality interaction method according to claim 1, characterized in that: The method comprises: Acquire a user image of a user, and identify the user image to acquire a corresponding body region image; The body region image is calibrated to obtain length parameters corresponding to each joint.

5. The virtual reality interaction method according to claim 1, characterized in that: The method comprises: The whole body posture information corresponding to the user is sent to the server side, so that the server side displays the virtual object of the user based on the whole body posture information in the virtual reality environment.

6. The virtual reality interaction method according to claim 5, characterized in that: The method comprises: configuring a user virtual space for a virtual object corresponding to the user in the virtual reality environment; A virtual object of the user is displayed in the user virtual space based on the full-body posture information.

7. The virtual reality interaction method according to claim 6, characterized in that: The method further comprises: Calibrate the sub-coordinate system corresponding to the user virtual space and the virtual reality environment coordinate system to determine a coordinate transformation matrix between the sub-coordinate system corresponding to the user virtual space and the virtual reality environment coordinate system; Based on the coordinate transformation matrix, and according to the full body posture information of the virtual object corresponding to the user in the user virtual space, the full body posture information of the virtual object corresponding to the user in the virtual reality environment coordinates is determined.

8. The virtual reality interaction method according to claim 1, characterized in that: The laser radar component and the inertial measurement unit component are arranged on a VR helmet worn by a user.

9. A virtual reality interactive device, characterized in that: The device comprises: A posture tracking device, used to collect body posture information of a user in a real environment; wherein the posture tracking device includes a motion detection unit arranged corresponding to the position of the target skeletal muscle, and the motion detection unit is used to determine the joint motion angle of the corresponding joint; A laser radar component is used to collect the first position information of the user's head in the real environment; An inertial measurement unit component, used to collect head posture information of the user's head in the real environment; A full body posture calculation module is used to configure the body posture information, head position information, and head posture information into the full body posture information of the user, so as to map the virtual object corresponding to the user in a virtual reality environment according to the full body posture information.

10. A virtual reality interactive system, characterized in that: The system comprises: A user terminal device, used to collect user data, wherein the user data includes the user's full body posture information in a real environment; wherein the full body posture information includes head position information, head posture information, and body posture information; The server side is used to obtain user data of multiple users, and according to the mapping relationship between each user and the virtual object in the virtual reality environment, display the virtual object corresponding to the user based on the full body posture information in the virtual reality environment.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the virtual reality interaction method according to any one of claims 1 to 8 is implemented.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the virtual reality interaction method according to any one of claims 1 to 8 by executing the executable instructions.

Citation Information

Patent Citations

  • Fusion positioning system and method based on optical tracking and inertial tracking

    CN111947650A

  • Motion capture system and method based on laser large-space positioning and optical inertia complementation

    CN112256125A

  • Whole body posture tracking and touch device and virtual reality system

    CN115202471A

  • Virtual display apparatus and virtual display method

    CN116009687A

  • Method for optical-inertial tracking of a user

    WO2021211002A1

Cited By

  • Interaction control method and system based on virtual reality

    CN120704516A

  • Simulation control method and device, equipment and storage medium

    CN121143679A