Method for interacting with a shared virtual scene, controller, handheld device and virtual system

Abstract

The application provides an interactive method and controller for sharing a virtual scene, a handheld device and a virtual system, and solves the technical problem that in the prior art, when multiple users use AR technology together, free interaction of the multiple users cannot be realized, thereby reducing the experience of the users. The interactive method for sharing a virtual scene by multiple users provided by the application virtually creates an entity interaction scene model from an entity environment in which any one of the multiple users is located, the multiple users share the entity interaction scene model, and 6DOF data of handheld devices of all the users in the entity environment is mapped into the entity interaction scene model, so as to enhance the interaction and AR experience of the participants. The entity environment in which any one of the users is located is virtually created into an entity interaction scene model, and each user interacts in the entity interaction scene model, so that each of the multiple users can realize free interaction without being in a specific real scene, and the experience of the users is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the virtual technology field, and in particular to an interaction method and a controller for sharing a virtual scene, a handheld device and a virtual system. BACKGROUND

[0002] Augmented reality technology (AR technology for short) is a computer simulation system that can create and experience a virtual world, which uses a computer to generate a simulated entity environment and is a multi-source information fusion and interactive three-dimensional dynamic view and entity behavior system simulation, so that a user can immerse in the entity environment, for example, can walk around like in the real world, and can simulate grasping things with both hands, etc.

[0003] In the prior art, when multiple users use AR technology to experience a more complex virtual scene, they can only have simple interaction in a specific entity environment, for example, multiple users can have simple interaction in a virtual scene through AR technology in a set experience area. When multiple users have virtual interaction through AR, the multiple users can only experience in the set experience area, and cannot have free interaction of multiple users in different areas or different positions in the same area, thereby reducing the experience of the users. SUMMARY

[0004] Therefore, the present application provides an interaction method and a controller for sharing a virtual scene, a handheld device and a virtual system, which solve the technical problem that multiple users cannot have free interaction when using AR technology to experience a more complex virtual scene in the prior art, thereby reducing the experience of the users.

[0005] According to an aspect of the present application, an interaction method for multiple users to share a virtual scene is provided, which includes: acquiring entity scene information of an entity environment in which a user is located; generating an entity interaction scene model corresponding to the entity environment in which the user is located according to the entity scene information; sending the entity interaction scene model to a related user related to the user; projecting an interactive avatar corresponding to the user into the entity interaction scene model; and acquiring 6DOF data of a handheld device used by the user, and mapping the 6DOF data to the entity interaction scene model.

[0006] In an embodiment of the present application, acquiring 6DOF data of a handheld device used by the user includes: acquiring IMU data of the handheld device used by the user; acquiring spatial positioning picture information of the handheld device; and acquiring 6DOF data of the handheld device according to the spatial positioning picture information and the IMU data of the handheld device.

[0007] In an embodiment of the present application, the 6DOF data of the handheld device is obtained according to the spatial positioning picture information and the IMU data of the handheld device, comprising: when the user is in the same entity environment as the entity interactive scene model sender, directly calculating the spatial positioning picture information and the IMU data of the handheld device to generate the 6DOF data of the handheld device.

[0008] In an embodiment of the present application, the 6DOF data of the handheld device is obtained according to the spatial positioning picture information and the IMU data of the handheld device, comprising: when the user is in the same entity environment as the entity interactive scene model sender, directly calculating the spatial positioning picture information and the IMU data of the handheld device to generate the 6DOF data of the handheld device.

[0009] In an embodiment of the present application, the spatial positioning picture information of the handheld device is obtained, comprising: obtaining multiple picture information of the handheld device; and obtaining the spatial positioning picture information of the handheld device according to the multiple picture information.

[0010] In an embodiment of the present application, before the interactive avatar corresponding to the user is projected into the entity interactive scene model, the multi-user shared virtual scene interaction method further comprises: obtaining a selected interactive avatar selected by the user in a preset interactive avatar library, wherein the selected interactive avatar is determined as the interactive avatar corresponding to the user.

[0011] In an embodiment of the present application, before the interactive avatar corresponding to the user is projected into the entity interactive scene model, the multi-user shared virtual scene interaction method further comprises: obtaining avatar picture information of the user; and generating the interactive avatar corresponding to the user according to the avatar picture information.

[0012] As a second aspect of the present application, the present application further provides a multi-user virtual scene interaction controller, comprising:

[0013] a data receiving module, configured to obtain entity scene information of an entity environment in which the user is located;

[0014] The data determining module is configured to determine 6DOF data of a handheld device used by the user; the entity interaction scene model generating module is configured to render and generate an entity interaction scene model corresponding to an entity environment where the user is located according to the entity scene information; the data sending module is configured to send the entity interaction scene model to a relevant user related to the user; and the mapping module is configured to project an interactive avatar corresponding to the user to the entity interaction scene model, and map the 6DOF data to the entity interaction scene model.

[0015] In an embodiment of the present application, the data determining module comprises a first data receiving module configured to receive IMU data of the handheld device; a second data receiving module configured to receive spatial positioning picture information of the handheld device used by the user; and a determining module configured to determine 6DOF data of the handheld device according to the spatial positioning picture information and the IMU data of the handheld device.

[0016] In an embodiment of the present application, the determining module comprises a judging module configured to judge whether an entity environment where the user is located is the same as an entity environment where the entity interaction scene model sender is located; and a first calculating module configured to, when the entity environment where the user is located is the same as the entity environment where the entity interaction scene model sender is located, directly calculate the spatial positioning picture information and the IMU data of the handheld device to generate the 6DOF data of the handheld device.

[0017] In an embodiment of the present application, the determining module further comprises a mapping relationship determining module configured to, when the entity environment where the user is located is different from the entity environment where the entity interaction scene model sender is located, acquire entity scene information of an actual environment where the user is located and render and generate a model of the entity scene of the user; and determine a scene mapping relationship according to the model of the entity scene of the user and a model of an entity interaction scene of the entity interaction scene model sender; and a second calculating module configured to acquire initial spatial positioning picture information of the handheld device used by the user, and determine spatial positioning picture information of the handheld device of the user in the entity interaction scene model according to the scene mapping relationship; and calculate the spatial positioning picture information of the handheld device of the user and the IMU data of the handheld device of the user to generate 6DOF data of the handheld device of the user.

[0018] In an embodiment of the present application, the multi-user virtual scene interaction controller further comprises an interactive avatar determining module configured to acquire a selected interactive avatar selected by the user in a preset interactive avatar library, and determine the selected interactive avatar as an interactive avatar corresponding to the user; or the interactive avatar determining module is configured to acquire head portrait picture information of the user, and generate an interactive avatar corresponding to the user according to the head portrait picture information.

[0019] As a third aspect of the present application, the present application further provides a handheld device, comprising: a host; and a handle module, which is detachably connected with the host and is in communication connection with the host; wherein the host comprises: the multi-user virtual scene interaction controller as described above; a second camera device, which is used for shooting entity scene information of an entity environment where the user is located; a first camera device, which is used for shooting picture information of the user; and an inertial sensor, which is used for detecting IMU data of the handheld device; wherein the first camera device, the second camera device and the inertial sensor are all in communication connection with the multi-user virtual scene interaction controller.

[0020] In an embodiment of the present application, the number of the first camera devices is four; wherein the four first camera devices are respectively arranged at four different regions of the host.

[0021] As a fourth aspect of the present application, the present application further provides a virtual system, comprising: the handheld device as described above; and a virtual device.

[0022] The multi-user shared virtual scene interaction method provided by the present application can virtualize the entity environment where any one of the multi-users is located into an entity interactive scene model, and the multi-users share the entity interactive scene model, and map 6DOF data of the handheld devices of all the users in the entity environment into the entity interactive scene model, that is, through spatial positioning of the handheld devices, real-time dynamic tracking of the multi-users in the virtual scene is realized, so as to enhance the interaction and AR experience of the participants. Moreover, since the entity environment where any one of the multi-users is located is virtualized into an entity interactive scene model, each user interacts in the entity interactive scene model, so that each user of the multi-users can realize free interaction without being in a specific and well-defined real scene, the spatial protection and freedom of each user are enhanced, and the experience of the users is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures. The accompanying drawings are used to provide a further understanding of the embodiments of the present application, constitute a part of the specification and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally refer to the same parts or steps throughout the figures.

[0024] Figure 1 Fig. 1 shows a flowchart of a multi-user shared virtual scene interaction method according to an embodiment of the present application;

[0025] Figure 2 Fig. 2 shows a flowchart of an interaction method for a multi-user shared virtual scene according to another embodiment of the present application;

[0026] Figure 3 Fig. 2 shows a flowchart of an interaction method for a multi-user shared virtual scene according to another embodiment of the present application;

[0027] Figure 4 Fig. 2 shows a flowchart of an interaction method for a multi-user shared virtual scene according to another embodiment of the present application;

[0028] Figure 5 Fig. 2 shows a flowchart of an interaction method for a multi-user shared virtual scene according to another embodiment of the present application;

[0029] Figure 6 Fig. 2 shows a flowchart of an interaction method for a multi-user shared virtual scene according to another embodiment of the present application;

[0030] Figure 7 Fig. 3 shows a working principle diagram of an interaction controller for a multi-user shared virtual scene according to an embodiment of the present application;

[0031] Figure 8 Fig. 3 shows a working principle diagram of an interaction controller for a multi-user shared virtual scene according to an embodiment of the present application;

[0032] Figure 9 Fig. 3 shows a working principle diagram of an interaction controller for a multi-user shared virtual scene according to an embodiment of the present application;

[0033] Figure 10 Fig. 3 shows a working principle diagram of an interaction controller for a multi-user shared virtual scene according to an embodiment of the present application; Figure 11 Fig. 4 shows a working principle diagram of a handheld device according to an embodiment of the present application;

[0034] Figure 12 Fig. 5 shows a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited. All directional indications (such as upper, lower, left, right, front, back, top, bottom, etc.) in the embodiments of the application are only used to explain the relative position relationship, movement condition, etc. between components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications will also change accordingly. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0036] In addition, the reference to "embodiments" herein means that the specific features, structures or characteristics described in conjunction with the embodiments can be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. The skilled person in the art understands explicitly and implicitly that the embodiments described herein can be combined with other embodiments.

[0037] The technical solutions in the embodiments of the application will be described clearly and completely below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by the person skilled in the art without creative labor are within the protection scope of the application.

[0038] Exemplary method

[0039] Figure 1 Fig. 1 shows a flowchart of an interactive method of a multi-user shared virtual scene according to an embodiment of the application; as Figure 1 As shown, the interactive method of the multi-user shared virtual scene comprises the following steps:

[0040] Step S101: Obtain the entity scene information of the entity environment in which the user is located;

[0041] When multiple users experience a virtual scene using AR technology, the multiple users are divided into a user and related users, so as to illustrate in detail the specific implementation method of each user in implementing the multiple users sharing a virtual scene by a single user. For example, when five users (individually denoted as A user, B user, C user, D user, and E user) experience a virtual scene together, when the user A is divided into a user, then the B user, C user, D user, and E user are other users. When the B user is divided into a user, the A user, C user, D user, and E user are other users. That is, each user in the A user, B user, C user, D user, and E user can be a user or other users. Therefore, the user and other users mentioned in the embodiment of the application are not fixed, and each user in the multiple users can execute the specific steps of the interactive method of the multiple users sharing a virtual scene in the embodiment of the application.

[0042] Specifically, the entity scene information of the entity environment in which the user is located refers to the entity scene in which the user is experiencing AR multi-user interaction, for example, the entity scene information of the entity environment in which the user is in a living room, a classroom, a dormitory, a game room, and the like.

[0043] Specifically, the entity scene information of the entity environment in which the user is located can be acquired by using a camera device, for example, a camera device is installed in a handheld device of the user, the camera device can capture a photo of the entity environment in which the user is located, and the entity scene information can be a photo reflecting the surroundings of the user or a photo reflecting the entity environment behind the user.

[0044] Step S102: generating an entity interaction scene model corresponding to the entity environment in which the user is located according to the entity scene information;

[0045] The entity interaction scene model corresponding to the entity environment in which the user is located is generated according to the entity scene information of the user, that is, the entity scene information of any user in the multiple users is generated into an entity interaction scene model, and the multiple users realize free interaction under the entity interaction scene model.

[0046] Step S102 is to virtually generate an entity interaction scene model from the entity scene information acquired in step S102. The entity interaction scene model can be an equivalent environment of the entity environment in which the user is located, that is, an entity interaction scene model reflecting the entity environment in which the user is located is generated by processing the entity scene information (for example, multiple photos reflecting the entity environment in which the user is located are synthesized).

[0047] Step S103: sending the entity interaction scene model to related users related to the user;

[0048] Specifically, after the user generates the entity interaction scene model, the user shares the entity interaction scene model with relevant users, so that multiple users can experience interaction in the same entity interaction scene model in different or same entity environments.

[0049] Step S104: project the interactive avatar corresponding to the user into the entity interaction scene model;

[0050] The user projects the interactive avatar corresponding to the user into the entity interaction scene model, and is ready for interaction.

[0051] It should be noted that when the user in step S104 is a divided user, the entity interaction scene model in step S104 is an entity interaction scene model generated by the user according to the entity scene information of the entity environment in which the user is located. For example, when five users (denoted as A user, B user, C user, D user, and E user) experience a virtual scene together, and A user is divided into a user, i.e., the user in step S104 is A user, then the entity interaction scene model in step S104 is an entity interaction scene model generated by A user according to the entity scene information of the entity environment in which A user is located.

[0052] When the user in step S104 is a relevant user, the entity interaction scene model in this step is an entity interaction scene model generated by a user divided into a user according to the entity scene information of the entity environment in which the user is located, and then transmitted to the user in step S104. For example, when A user is a user, then the user in step S104 is any one of B user, C user, D user, and E user, then the entity interaction scene model in step S104 is an entity interaction scene model generated by A user according to the entity scene information of the entity environment in which A user is located, and sent to B user.

[0053] Step S105: obtain 6DOF data of the handheld device used by the user.

[0054] 6DOF data refers to six degrees of freedom data, such as rotation data on X, Y, and Z axes and movement data on X, Y, and Z axes, i.e., equivalent to three types of translation motion (translation motion on X and Y planes, translation motion on X and Z planes, and translation motion on Z and Y planes) and three types of rotation motion (rotation motion on X axis, rotation motion on Y axis, and rotation motion on Z axis). Through the 6DOF data of the handheld device, the user's environment and active behavior can be simulated, such as the user's micro-movement (tilting head, shrinking neck, leaning forward, etc.), body movement (can walk, jump, squat, or even lie down), etc.

[0055] Step S106: map the 6DOF data to the entity interaction scene model.

[0056] By acquiring the 6DOF data of the handheld device of the user and mapping the 6DOF data into the entity interaction scene model, the real-time interaction of the avatar of the user in the virtual scene (for example, the entity interaction scene model) can be embodied.

[0057] When the multiple users all complete step S106, the multiple users can start to interact, and each user performs an interaction operation in the respective entity environment, for example, the user holds the respective handheld device so that the handheld device moves in the entity environment. According to the movement of the handheld device, the 6DOF data of the handheld device is acquired, and the 6DOF data is mapped into the entity interaction scene model, that is, the real-time dynamic tracking of the multiple users in the virtual scene is realized by the spatial positioning of the handheld device, so as to enhance the interaction and AR experience of the participants.

[0058] The method for multiple users to share a virtual scene provided by the application virtualizes the entity environment in which any one of the multiple users is located into an entity interaction scene model, the multiple users share the entity interaction scene model, and the 6DOF data of the handheld device of all the users in the entity environment is mapped into the entity interaction scene model, that is, the real-time dynamic tracking of the multiple users in the virtual scene is realized by the spatial positioning of the handheld device, so as to enhance the interaction and AR experience of the participants. Moreover, since the entity environment in which any one of the multiple users is located is virtualized into an entity interaction scene model, each user interacts in the entity interaction scene model, so that each user of the multiple users can freely interact without being in a specific and well-defined real scene, the spatial protection and freedom of each user are enhanced, and the experience of the user is improved.

[0059] Figure 2 As shown in the figure, the method for multiple users to share a virtual scene provided by another embodiment of the application includes the following steps: Figure 2 As shown in the figure, step S105 (acquiring the 6DOF data of the handheld device used by the user) specifically includes the following steps:

[0060] Step S1051: acquiring the IMU data of the handheld device used by the user;

[0061] Specifically, the inertial sensor (Inertial Measurement Unit, IMU for short) is a device for measuring the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three-axis coordinate system of the carrier, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system. Therefore, the inertial sensor can measure the angular velocity and acceleration of the object in three-dimensional space, and calculate the attitude of the object, such as calculating the rotational degrees of freedom of the object. The rotational degrees of freedom refer to the three position-related degrees of freedom of up and down, front and back, and left and right. The IMU data is the result data detected by the inertial sensor, that is, the angular velocity and acceleration data of an object in three-dimensional space detected by the inertial sensor. Therefore, the IMU data of the handheld device can be detected by the inertial sensor arranged on the handheld device, and the attitude of the handheld device can be calculated based on the data, such as the rotational degrees of freedom of the handheld device. The rotational degrees of freedom refer to the three position-related degrees of freedom of up and down, front and back, and left and right.

[0062] Step S1052: Obtain the spatial positioning picture information of the handheld device.

[0063] The spatial positioning picture information refers to the space positioning picture information of the handheld device photographed by the camera arranged on the handheld device when the user holds the handheld device and moves in the real environment during multi-user interaction. According to the spatial positioning picture information, the motion data of the handheld device can be obtained.

[0064] Step S1053: Obtain the 6DOF data of the handheld device according to the spatial positioning picture information and the IMU data of the handheld device.

[0065] The 6DOF data of the handheld device can be calculated based on the spatial positioning picture information of the handheld device and the IMU data of the handheld device. The spatial positioning picture information refers to the spatial positioning information of the handheld device, that is, the movement data of the handheld device on the X, Y and Z axes, which is equivalent to three kinds of translational motion (translational motion on the X and Y planes, translational motion on the X and Z planes, and translational motion on the Z and Y planes).

[0066] Optionally, the spatial positioning picture information in step S1052 (obtaining the spatial positioning picture information of the handheld device) can be a spatial positioning picture information of the handheld device collected by a camera arranged on the handheld device. Alternatively, the spatial positioning picture information of the handheld device can be generated by synthesizing multiple spatial pictures about the space where the handheld device is located, which are photographed by multiple cameras arranged on the handheld device.

[0067] In another embodiment of the present application, Figure 3 Fig. 6 shows a flowchart of an interactive method of a multi-user shared virtual scene according to another embodiment of the present application, and Figure 3 As shown, when the user is a relevant user, the entity interactive scene model of the user is sent by the user, for example, user A is the user, and user B is the relevant user, and then the entity interactive scene model of user B is generated by user A and sent to user B, that is, the entity interactive scene model of user B is sent by user A. In this case, step S1053 (obtaining 6DOF data of the handheld device according to the spatial positioning picture information and the IMU data of the handheld device) specifically comprises:

[0068] Step S10531: judging whether the actual environment where the user is located is the same as the actual environment where the entity interactive scene model is sent by the user,

[0069] When the result of step S10531 is yes, that is, the user is a relevant user, and when the user is in the same entity environment as the sender of the entity interactive scene model, step S10532 is executed.

[0070] Step S10532: directly calculating the spatial positioning picture information and the IMU data of the handheld device to generate 6DOF data of the handheld device.

[0071] That is, when multiple users interact, the actual environments where the multiple users are located are the same, for example, the multiple users are in the same living room to participate in interaction, and at this time, the handheld devices of each user are in the same actual environment, and then the specific motion data of the handheld devices of the users in the actual environment can be obtained according to the spatial positioning picture information of the handheld devices of each user, and the spatial positioning picture information and the IMU data of the handheld device can be directly calculated to generate 6DOF data of the handheld device. When the 6DOF data is mapped to the entity interactive scene model in step S106, since the entity interactive scene model is also virtually formed in the actual environment, the 6DOF data can be directly mapped to the entity interactive scene model, so that real-time dynamic tracking of the multiple users in the virtual scene can be realized through spatial positioning of the handheld device, so as to enhance the interaction and AR experience of the participants.

[0072] When the determination result in step S10531 is no, it indicates that the user and the entity interaction scene model sending user are not in the same actual environment, i.e., the actual environments where the multiple users are located are different. For example, multiple users are respectively in different actual environments such as a bedroom and a living room. Since the entity interaction scene model is virtually generated according to the actual environment where one user is located, when the users interact, the relevant user downloads the entity interaction scene model sent by the user and takes it as his own entity interaction scene model. For example, A user is the user and B user is the relevant user. The entity interaction scene model of B user is the entity interaction scene model generated by A user and sent to B user, i.e., the entity interaction scene model of B user is sent by A user, and B user downloads the entity interaction scene model. In this case, step S1053 (acquiring 6DOF data of the handheld device according to the spatial positioning picture information and the IMU data of the handheld device) specifically includes:

[0073] Step S10533: acquiring entity scene information of the actual environment where the user is located and rendering a model of the entity scene of the user.

[0074] That is, acquiring entity scene information of the actual environment where the user is located, for example, A user is the user and B user is the relevant user. B user as the relevant user needs to acquire scene information of the actual environment where he is located.

[0075] Step S10534: determining a scene mapping relationship according to the model of the entity scene of the user and the model of the entity interaction scene of the entity interaction scene model sender. The scene mapping relationship is the scene mapping relationship between the model of the entity scene of the user and the model of the entity interaction scene of the entity interaction scene model sender. That is, acquiring the entity interaction scene model sent by the entity interaction scene model sender, for example, A user is the user and B user is the relevant user. A user sends the entity interaction scene model virtually generated according to his own entity scene information and actual scene information to B user. B user as the relevant user receives the entity interaction scene model sent by A user and the entity scene information of A user.

[0076] B user as the relevant user needs to determine the model mapping relationship between the two models according to the entity interaction scene model sent by A user and the model of the entity scene of the entity environment where B user is located.

[0077] Since the actual environment where the A user and the B user are located is different, the space where the handheld device of the A user and the handheld device of the B user are located is different, and the reference of the spatial positioning of the A user and the spatial positioning of the handheld device of the B user is different. Therefore, it is necessary to obtain the entity interaction scene model between the A user and the B user and the model of the entity scene to determine the model mapping relationship between the two models, so that the reference of the spatial positioning of the handheld device of the A user and the spatial positioning of the handheld device of the B user is the same.

[0078] Step S10535: obtaining the initial spatial positioning picture information of the handheld device used by the user, and determining the spatial positioning picture information of the handheld device of the user in the entity interaction scene model according to the scene mapping relationship;

[0079] At this time, the user is a related user, for example, since the actual environment where the A user and the B user are located is different, the initial spatial positioning picture information of the handheld device of the B user is obtained, and the initial spatial positioning picture information can be obtained by the camera installed on the handheld device of the B user. According to the scene mapping relationship between the A user and the B user obtained in step S10534, the spatial positioning picture information of the handheld device of the B user is determined, so that multiple users who are not in the same actual environment can interact in the entity interaction scene model rendered in the entity environment where one of the users is located, and the real interaction state of each user is reflected.

[0080] Step S10536: calculating the spatial positioning picture information of the handheld device of the user and the IMU data of the handheld device of the user to generate the 6DOF data of the handheld device of the user.

[0081] That is, when multiple users interact, at least two users in the multiple users are in different actual environments, for example, one of the multiple users is in a bedroom and the remaining users are in a living room; or each user in the multiple users is in a different actual environment. At this time, when the actual environment where the related user and the user (entity interaction scene model generation user) are located is different, the scene mapping relationship of the scene model where the related user and the user are located can be obtained, and then the initial spatial positioning picture information of the handheld device of the related user and the scene mapping relationship are used to determine the spatial positioning picture information of the user, so that the handheld device of the related user and the handheld device of the user can move in the same actual environment. When the 6DOF data is mapped to the entity interaction scene model in step S106, the accuracy of the interaction can be enhanced, the accurate spatial positions of the multiple parties in the interaction can be accurately known, and therefore the real-time dynamic tracking of the multiple users in the virtual scene can be realized to enhance the interaction and AR experience of the participants.

[0082] In an embodiment of the present application, as Figure 4As shown, the step S1052 (acquiring the spatial positioning picture information of the handheld device) specifically comprises the steps of:

[0083] Step S1: acquiring a plurality of spatial picture information of the handheld device;

[0084] The plurality of picture information refers to a plurality of spatial picture information of the movement space of the handheld device respectively photographed by a plurality of camera devices installed on the handheld device when the user holds the handheld device to move in the space, for example, one camera device is installed on each of the four corners of the handheld device, and each camera device photographs 4 spatial picture information of the movement space of the handheld device from different angles, so that the spatial positioning of the handheld device can be more accurately determined.

[0085] Step S2: acquiring the spatial positioning picture information of the handheld device according to the plurality of picture information.

[0086] The step S2 is to process the plurality of picture information photographed from different angles to generate the spatial positioning picture information of the handheld device, so that the spatial positioning information of the handheld device can be more accurately obtained, thereby enhancing the interaction accuracy.

[0087] In another embodiment of the present application, Figure 5 As shown in the flowchart of the interaction method of the multi-user shared virtual scene provided by another embodiment of the present application, Figure 5 As shown, before the step S104 (projecting the interactive avatar corresponding to the user to the physical interactive scene model), the interaction method of the multi-user shared virtual scene further comprises the following steps:

[0088] Step S1040: acquiring the selected interactive avatar selected by the user from the preset interactive avatar library, and determining the selected interactive avatar as the interactive avatar corresponding to the user.

[0089] That is, the interactive avatar of the user can be an interactive avatar selected by the user from the interactive avatar library, and since the interactive avatar library stores a plurality of interactive avatars in different forms, the user can select an interactive avatar according to his / her preference.

[0090] Optionally, Figure 6 As shown in the flowchart of the interaction method of the multi-user shared virtual scene provided by another embodiment of the present application, Figure 6 As shown, before the step S104 (projecting the interactive avatar corresponding to the user to the physical interactive scene model), the interaction method of the multi-user shared virtual scene further comprises the following steps:

[0091] Step S1041: acquiring the head picture information of the user;

[0092] The image picture information of the user can be a head portrait picture of the user taken by a camera installed on the handheld device.

[0093] Step S1042: generating an interactive avatar corresponding to the user according to the head portrait picture information.

[0094] That is, the interactive avatar of the user can be obtained according to the real head portrait picture of the user, so that the authenticity of multi-user interaction is enhanced, and relevant users can easily know the specific real information of the interactive parties according to the interactive avatar, thereby enhancing the authenticity and experience of the users.

[0095] As a second aspect of the present application, the present application also provides a multi-user virtual scene interaction controller 10, Figure 7 The working principle diagram of the multi-user virtual scene interaction controller 10 provided by the present application is shown in the figure. Figure 7 The multi-user virtual scene interaction controller 10 comprises:

[0096] The data receiving module 100 is configured to acquire entity scene information of an entity environment in which the user is located.

[0097] The data determining module 200 is configured to determine 6DOF data of a handheld device used by the user.

[0098] The entity interactive scene model generating module 300 is configured to render and generate an entity interactive scene model corresponding to the entity environment in which the user is located according to the entity scene information.

[0099] The data sending module 400 is configured to send the entity interactive scene model to a relevant user related to the user, and the mapping module 500 is configured to project the interactive avatar corresponding to the user into the entity interactive scene model and map the 6DOF data into the entity interactive scene model. The multi-user shared virtual scene interaction controller provided by the present application virtually forms an entity interactive scene model from an entity environment in which any one of the multi-users is located, and the multi-users share the entity interactive scene model, and maps the 6DOF data of the handheld devices of all the users in the entity environment into the entity interactive scene model, that is, the real-time dynamic tracking of the multi-users in the virtual scene is realized through the spatial positioning of the handheld devices, so as to enhance the interaction and AR experience of the participants.

[0100] In an embodiment of the present application, Figure 8As shown in the working principle diagram of the multi-user virtual scene interaction controller 10 provided by the application, Figure 8 As shown, the data determination module 200 comprises:

[0101] The first data receiving module 201 is configured to receive the IMU data of the handheld device.

[0102] The second data receiving module 202 is configured to receive the spatial positioning picture information of the handheld device used by the user.

[0103] The spatial positioning picture information refers to the spatial positioning picture information of the handheld device photographed by the camera installed on the handheld device when the user moves in the real environment while holding the handheld device in the multi-user interaction. According to the spatial positioning picture information, the motion data of the handheld device can be obtained.

[0104] Optionally, the spatial positioning picture information can be a spatial positioning picture information of the handheld device collected by one camera installed on the handheld device. Alternatively, the spatial positioning picture information of the handheld device can be generated by synthesizing multiple spatial pictures about the space where the handheld device is located, which are photographed by multiple cameras installed on the handheld device.

[0105] The determination module 203 is configured to determine the 6DOF data of the handheld device according to the spatial positioning picture information and the IMU data of the handheld device.

[0106] According to the spatial positioning picture information of the handheld device and the IMU data of the handheld device, the 6DOF data of the handheld device can be calculated. The spatial positioning picture information refers to the spatial positioning information of the handheld device, i.e., the movement data on the X, Y and Z axes of the handheld device, which is equivalent to three kinds of translational motion (translational motion on the X and Y planes, translational motion on the X and Z planes, and translational motion on the Z and Y planes).

[0107] In an embodiment of the application, Figure 9 As shown in the working principle diagram of the multi-user virtual scene interaction controller 10 provided by the application, when the user is a related user, the entity interaction scene model of the user is sent by the user. For example, the user A is a user, and the user B is a related user. Therefore, the entity interaction scene model of the user B is generated by the user A and then sent to the user B. That is, the entity interaction scene model of the user B is sent by the user A. That is, when the user is a related user, as shown in the working principle diagram of the multi-user virtual scene interaction controller 10 provided by the application, Figure 9 As shown, the determination module 203 in the multi-user virtual scene interaction controller 10 used by the user A comprises:

[0108] determining whether the entity environment in which the user and the entity interaction scene model sender are located is the same;

[0109] When the determining module 2031 determines that the entity environment in which the user and the entity interaction scene model sender are located is the same, the first determination information is transmitted to the first calculation module, and the first determination information is that the entity environment in which the user and the entity interaction scene model sender are located is the same. When the determining module 2031 determines that the entity environment in which the user and the entity interaction scene model sender are located is not the same, the second determination information is transmitted to the mapping relationship determination module 2033.

[0110] The first calculation module 2032 is configured to, when the entity environment in which the user and the entity interaction scene model sender are located is the same, directly calculate the spatial positioning picture information and the IMU data of the handheld device to generate 6DOF data of the handheld device.

[0111] When the first calculation module 2032 receives the first determination information sent by the determining module, the spatial positioning picture information and the IMU data of the handheld device are directly calculated to generate 6DOF data of the handheld device.

[0112] That is, when multiple users interact, the actual environments in which the multiple users are located are the same, for example, multiple users participate in interaction in the same living room, at this time, the handheld devices of each user are in the same actual environment, so that the specific motion data of the handheld devices of the users in the actual environment can be obtained according to the spatial positioning picture information of the handheld devices of each user, the spatial positioning picture information and the IMU data of the handheld device can be directly calculated to generate 6DOF data of the handheld device, and when the 6DOF data is mapped to the entity interaction scene model in the mapping module 500, since the entity interaction scene model is also virtually formed in the actual environment, the 6DOF data can be directly mapped to the entity interaction scene model, so that real-time dynamic tracking of multiple users in a virtual scene can be realized through spatial positioning of the handheld device, to enhance the interaction and AR experience of participants.

[0113] The mapping relationship determination module 2033 is configured to, when the entity environment in which the user and the entity interaction scene model sender are located is not the same, acquire entity scene information of the actual environment in which the user is located and render a model of the entity scene of the user; and determine a scene mapping relationship according to the model of the entity scene of the user and the model of the entity interaction scene of the entity interaction scene model sender.

[0114] When the mapping relationship determination module 2033 receives the second judgment information sent by the judgment module, it indicates that the user and the entity interaction scene model sending user are not in the same actual environment. Since the entity interaction scene model is virtual according to the actual environment in which a user is located, when the related user interacts, the user sends the entity interaction scene model downloaded by the related user as the entity interaction scene model of the related user. For example, A user is the user, and B user is the related user. Then, the entity interaction scene model of the B user is the entity interaction scene model generated by the A user and sent to the B user. That is, the entity interaction scene model of the B user is sent by the A user, and the B user downloads the entity interaction scene model. At this time, the entity scene information of the actual environment in which the user is located is obtained, and the model of the entity scene of the user is rendered and generated. For example, when A user is the user and B user is the related user, the B user as the related user needs to obtain the scene information of the actual environment in which the B user is located, and render and generate the model of the entity scene of the B user. And the B user as the related user needs to determine the model mapping relationship between the two models according to the entity interaction scene model sent by the A user and the model of the entity scene of the entity environment in which the B user is located.

[0115] The second calculation module 2034 is configured to obtain the initial spatial positioning picture information of the handheld device used by the user, and determine the spatial positioning picture information of the handheld device of the user in the entity interaction scene model according to the scene mapping relationship. The spatial positioning picture information of the handheld device of the user and the IMU data of the handheld device of the user are calculated to generate the 6DOF data of the handheld device of the user.

[0116] The second calculation module 2034 receives the model mapping relationship sent by the mapping relationship determination module 2033.

[0117] At this time, the user as the related user, for example, since the actual environments in which the A user and the B user are located are different, the initial spatial positioning picture information of the handheld device of the B user is obtained. The initial spatial positioning picture information can be obtained by a camera installed on the handheld device of the B user. And the spatial positioning picture information of the handheld device of the B user is determined according to the model mapping relationship sent by the mapping relationship determination module 2033. This can enable multiple people who are not in the same actual environment to interact in the entity interaction scene model rendered in the entity environment in which one of the users is located, and reflect the real interaction state of each user.

[0118] That is, when multiple users interact, at least two users in the multiple users are in different actual environments, for example, one of the multiple users is in a bedroom and the rest are in a living room; or each user in the multiple users is in a different actual environment, at this time, when the relevant user and the user (entity interaction scene model generation user) are in different actual environments, the scene mapping relationship of the relevant user and the user is determined according to the scene model, and then the initial spatial positioning picture information of the handheld device of the relevant user is determined. The spatial positioning picture information of the user, so that the handheld devices of the relevant user and the user can move in the same actual environment, when the mapping module 500 maps the 6DOF data to the entity interaction scene model, the accuracy of the interaction can be enhanced, and the accurate spatial positions of the multiple parties interacting can be accurately known, so that real-time dynamic tracking of multiple users in a virtual scene can be realized, to enhance the interaction and AR experience of participants.

[0119] In an embodiment of the present application, Figure 10 As shown in the working principle diagram of the multi-user virtual scene interaction controller 10 provided by the present application, the multi-user virtual scene interaction controller 10 comprises a multi-user virtual scene interaction controller 10, a handheld device 20, and a mapping module 500. Figure 10 As shown, the multi-user virtual scene interaction controller 10 further comprises an interactive avatar determination module 700, which is used to obtain a selected interactive avatar selected by a user in a preset interactive avatar library, and determine the selected interactive avatar as an interactive avatar corresponding to the user.

[0120] The interactive avatar determination module 700 is used to obtain the avatar picture information of the user, and generate an interactive avatar corresponding to the user according to the avatar picture information.

[0121] As a third aspect of the present application, the present application further provides a handheld device, Figure 11 As shown in the working principle diagram of the handheld device provided by the present application, the handheld device comprises a host 11 and a handle module 12. Figure 11 As shown, the handheld device comprises a host 11 and a handle module 12. The handle module 12 is detachably connected with the host 11, and the handle module 12 is in communication connection with the host 11. The host 11 comprises the multi-user virtual scene interaction controller 10 described above, a first camera 111 for shooting spatial picture information of the handheld device, an inertial sensor 113 for detecting IMU data of the handheld device, and a second camera 112 for shooting entity scene information of an entity environment where the user is located. The first camera 111, the second camera 112, and the inertial sensor 113 are all in communication connection with the multi-user virtual scene interaction controller 10.

[0122] The first camera can capture spatial picture information of the handheld device in an actual environment, which can determine spatial positioning picture information of the handheld device; the inertial sensor can measure IMU data of the handheld device; and the virtual scene interaction controller can calculate 6DOF data of the handheld device according to the spatial positioning picture information and the IMU data. The second camera can capture entity scene information of an entity environment in which the user is located, and the virtual scene interaction controller can virtually form an entity interactive scene model according to the entity scene information, that is, by virtually forming the entity environment in which any one of the multiple users is located into the entity interactive scene model, the multiple users share the entity interactive scene model, and the 6DOF data of the handheld devices of all the users in the entity environment are mapped into the entity interactive scene model, that is, by realizing real-time dynamic tracking of the multiple users in the virtual scene through spatial positioning of the handheld devices, the interaction and AR experience of the participants are enhanced. Moreover, since the entity environment in which any one of the multiple users is located is virtually formed into the entity interactive scene model, each user interacts in the entity interactive scene model, so that each of the multiple users can freely interact without being in a specific designated real scene, the spatial protection and freedom of each user are enhanced, and the experience of the users is improved.

[0123] Optionally, the number of the first cameras is four. The four first cameras can capture spatial picture information of the handheld device from four different angles, for example, one first camera is installed on each of the four corners of the handheld device, and each first camera captures four spatial pictures of the motion space of the handheld device from different angles, so that the spatial positioning of the handheld device can be more accurate.

[0124] When the number of the first cameras is multiple, the distribution of the first cameras on the host can be determined according to the actual shape of the host, for example, when the host of the handheld device is square, the first cameras can be distributed at the four corner positions of the host; when the host of the handheld device is circular, the first cameras can be uniformly distributed on the circle.

[0125] As a third aspect of the present invention, the present invention also provides a virtual system, including: the handheld device described above; and a virtual device. This virtual system can be an augmented reality (AR) system, which virtualizes the physical environment in which any one of the multiple users is located as a physical interactive scene model. Multiple users share this physical interactive scene model, and the 6DOF data of all users' handheld devices in the physical environment is mapped to this physical interactive scene model. That is, real-time dynamic tracking of multiple users in the virtual scene is achieved through the spatial positioning of the handheld devices, thereby enhancing the interaction of participants and the AR experience. Furthermore, since the physical environment in which any one of the multiple users is located is virtualized as a physical interactive scene model, and each user interacts within this physical interactive scene model, each user can freely interact without being confined to a specific, pre-defined real-world scene, enhancing the spatial protection and freedom of each user and improving the user experience.

[0126] Below, for reference Figure 12 To describe an electronic device according to an embodiment of the present invention. Figure 12 The diagram shown is a structural schematic of an electronic device provided in an embodiment of the present invention.

[0127] like Figure 12 As shown, the electronic device 600 includes one or more processors 601 and memory 602.

[0128] The processor 601 may be a central processing unit (CPU) or other form of processing unit with data processing and / or information execution capabilities, and may control other components in the electronic device 600 to perform desired functions.

[0129] The memory 601 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program information may be stored on the computer-readable storage medium, and the processor 601 may run the program information to implement the multi-user shared virtual scene interaction method or other desired functions described in the various embodiments of the present invention above.

[0130] In one example, the electronic device 600 may also include an input device 603 and an output device 604, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0131] The input device 603 may include, for example, a keyboard, a mouse, etc.

[0132] The output device 604 can output various information to the outside. The output device 604 can include, for example, a display, a communication network and a remote output device connected thereto, and the like.

[0133] Of course, in order to simplify, Figure 9 Only some of the components of the electronic device 600 related to the present application are shown in FIG. 6, and components such as a bus, an input / output interface, and the like are omitted. In addition, the electronic device 600 can include any other appropriate components according to a specific application.

[0134] In addition to the above-mentioned method and device, an embodiment of the present application can be a computer program product including computer program information that, when executed by a processor, causes the processor to perform the steps of the interaction method of the multi-user shared virtual scene according to various embodiments of the present application described in the specification.

[0135] The computer program product can be written in any combination of one or more programming languages including an object-oriented programming language such as Java, C++, and the like, and a conventional procedural programming language such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as a separate software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0136] In addition, an embodiment of the present application can also be a computer readable storage medium having stored thereon computer program information, which, when executed by a processor, causes the processor to perform the steps of the interaction method of the multi-user shared virtual scene according to various embodiments of the present application described in the specification.

[0137] The computer readable storage medium can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above.

[0138] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the above specific details disclosed are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to be necessarily implemented with the above specific details.

[0139] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, meaning "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0140] It should also be noted that in the devices, apparatuses and methods of the present application, each component or step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application.

[0141] The above description of the disclosed aspects is provided so that any person skilled in the art can make or use the present application. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0142] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and the like made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-user shared virtual scene interaction method, characterized in that, include: Obtain entity scene information of the user's physical environment; Based on the entity scene information, an entity interaction scene model corresponding to the entity environment where the user is located is rendered and generated. Send the entity interaction scenario model to relevant users associated with the user; The interactive avatar corresponding to the user is projected onto the physical interactive scene model; and Acquire 6DOF data from the handheld device used by the user, and map the 6DOF data to the entity interaction scene model; Obtaining 6DOF data from the handheld device used by the user, including: Obtain the IMU data of the handheld device used by the user; Obtain spatial positioning image information of the handheld device; and Based on the spatial positioning image information and the IMU data of the handheld device, obtain the 6DOF data of the handheld device; Based on the spatial positioning image information and the IMU data of the handheld device, obtain the 6DOF data of the handheld device, including: When the user's physical environment differs from that of the entity interaction scene model sender, the entity scene information of the user's actual environment is obtained and the model of the user's physical scene is rendered and generated. Based on the user's entity scene model and the entity interaction scene model of the sender, determine the scene mapping relationship; Obtain the initial spatial positioning image information of the user's handheld device, and determine the spatial positioning image information of the user's handheld device in the physical interaction scene model according to the scene mapping relationship; and The spatial positioning image information of the user's handheld device and the IMU data of the user's handheld device are calculated to generate the 6DOF data of the user's handheld device.

2. The interactive method for multi-user shared virtual scenes according to claim 1, characterized in that, Based on the spatial positioning image information and the IMU data of the handheld device, obtain the 6DOF data of the handheld device, including: When the user and the entity interaction scene model sender are in the same entity environment, the spatial positioning image information and the IMU data of the handheld device are directly calculated to generate the 6DOF data of the handheld device.

3. The interactive method for multi-user shared virtual scenes according to claim 1, characterized in that, Obtaining the spatial positioning image information of the handheld device includes: Acquire multiple image information from the handheld device; and Based on the multiple image information, spatial positioning image information of the handheld device is obtained.

4. The multi-user shared virtual scene interaction method according to claim 1, characterized in that, Before projecting the interactive avatar corresponding to the user onto the entity interactive scene model, the multi-user shared virtual scene interaction method further includes: Obtain the selected interactive avatar chosen by the user from the preset interactive avatar library, and determine that the selected interactive avatar is the interactive avatar corresponding to the user.

5. The interactive method for multi-user shared virtual scenes according to claim 1, characterized in that, Before projecting the interactive avatar corresponding to the user onto the physical interactive scene model, the multi-user shared virtual scene interaction method further includes: real Obtain the user's profile picture information; and An interactive avatar corresponding to the user is generated based on the avatar image information.

6. A multi-user virtual scene interaction controller, characterized in that, include: The data receiving module is used to acquire physical scene information of the user's physical environment; The data determination module is used to determine the 6DOF data of the handheld device used by the user; The entity interaction scene model generation module is used to render and generate an entity interaction scene model corresponding to the entity environment where the user is located based on the entity scene information. The data sending module is used to send the entity interaction scene model to relevant users related to the user; as well as The mapping module is used to project the interactive avatar corresponding to the user onto the entity interaction scene model; and to map the 6DOF data onto the entity interaction scene model; The data determination module includes: The first data receiving module is used to receive IMU data from the handheld device; The second data receiving module is used to receive spatial positioning image information from the handheld device used by the user; and The determination module is used to determine the 6DOF data of the handheld device based on the spatial positioning image information and the IMU data of the handheld device; The determining module further includes: The mapping relationship determination module is used to obtain the entity scene information of the actual environment in which the user is located and render and generate the model of the user's entity scene when the entity environment in which the user and the entity interaction scene model sender are in different; and to determine the scene mapping relationship based on the model of the user's entity scene and the model of the entity interaction scene model sender. The second calculation module is used to obtain the initial spatial positioning image information of the handheld device used by the user, and determine the spatial positioning image information of the handheld device in the entity interaction scene model according to the scene mapping relationship; and calculate the spatial positioning image information of the handheld device and the IMU data of the handheld device to generate the 6DOF data of the handheld device.

7. The multi-user virtual scene interaction controller according to claim 6, characterized in that, The determining module includes: The judgment module is used to determine whether the user and the entity interaction scene model sender are in the same entity environment; The first calculation module is used to directly calculate the spatial positioning image information and the IMU data of the handheld device when the user and the entity interaction scene model sender are in the same entity environment, and generate the 6DOF data of the handheld device.

8. The multi-user virtual scene interaction controller according to claim 6, characterized in that, Also includes: The interactive avatar determination module is used to obtain the selected interactive avatar chosen by the user in the preset interactive avatar library, and determine that the selected interactive avatar is the interactive avatar corresponding to the user; or The interactive avatar determination module is used to obtain the user's avatar image information; And generate an interactive avatar corresponding to the user based on the avatar image information.

9. A handheld device, characterized in that, include: Host; as well as A handle module, wherein the handle module is detachably connected to the host and is communicatively connected to the host; The host includes: The multi-user virtual scene interaction controller according to any one of claims 6-8; The second camera device is used to capture physical scene information of the physical environment in which the user is located; A first camera device, used to capture image information of the user; and An inertial sensor, wherein the inertial sensor is used to detect IMU data of the handheld device; The first camera device, the second camera device, and the inertial sensor are all communicatively connected to the multi-user virtual scene interaction controller.

10. The handheld device according to claim 9, characterized in that, The number of the first camera devices is four; The four first camera devices are respectively located in four different areas of the host computer.

11. A virtual system, characterized in that, include: The handheld device as described in claim 9 or 10; as well as Virtual device.

Images