Interaction method, device and equipment for multi-dimensional mixed object in three-dimensional space

Abstract

The invention relates to an interaction method, device and equipment for a multi-dimensional mixed object in a three-dimensional space. Comprising the steps of receiving user operation information sent by input equipment; the operation information is converted into a 3D selector in a three-dimensional space, and the 3D selector is used for a user to interact with content in the three-dimensional space; and when the 3D selector selects the two-dimensional objects in the three-dimensional space, converting the 3D selector into a 2D selector in the two-dimensional objects, calculating interaction information of the 2D selector in the two-dimensional objects according to the change of the operation information, and interacting contents in the two-dimensional objects according to the interaction information. Through the embodiment of the invention, the input mode of the input equipment does not need to be changed, and the selector on the three-dimensional object and the two-dimensional object in the three-dimensional space is automatically switched through the input equipment, so that a user can operate the two-dimensional object by using an operation habit of traditional plane equipment when operating the two-dimensional object in the three-dimensional space, the learning cost is reduced, and the learning efficiency is improved. And the operation precision and the system usability are improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to an interaction method, device and equipment for multi-dimensional hybrid objects in a three-dimensional space. Background Art

[0002] Multidimensional mixed objects in three-dimensional space refer to objects of multiple dimensions in a mixed reality system. For example, a mixed reality system contains both three-dimensional objects and two-dimensional objects, such as three-dimensional objects and two-dimensional web pages.

[0003] The mixed reality system provides a multi-dimensional mixed three-dimensional space for users. From the user's perspective, the three-dimensional space can include both three-dimensional objects and two-dimensional objects. Users interact (operate) with three-dimensional or two-dimensional objects in the three-dimensional space through input devices, and the input devices can be peripherals or tracking of multiple user body parts. The mixed reality system receives input signals from the input devices, and then renders 3D selectors such as rays, 3D collision bodies, cone collision bodies, cubes, cones, etc. corresponding to the input signals in the three-dimensional space. The position and posture of the rendered 3D selectors such as rays are adjusted according to the position change of the user input device or other forms of operation information, so that users can interact with the content in the three-dimensional space through 3D selectors such as rays.

[0004] However, in a multi-dimensional mixed reality system, the operational requirements of two-dimensional objects and three-dimensional objects are very different. Two-dimensional objects generally inherit the traditional 2D interface design, carry more dense information, and require more sophisticated operations. Three-dimensional objects are more complex in spatial relationships in three-dimensional space, and users can interact with three-dimensional content of different sizes, depths, positions, and rotations.

[0005] Although 3D selectors such as rays and 3D collision bodies are convenient for interacting with 3D content in 3D space, and can accurately and quickly select corresponding targets, when used to select 2D objects, 3D selectors such as rays often have large selection errors caused by jitter, which reduces the user's interaction speed and interaction experience. Therefore, if you use the same 3D selector as a 3D object to interact with 2D objects, such as rays, it will increase the difficulty of operations such as selecting 2D content. In addition, it will make users relearn this interaction method, increasing the user's learning cost.

[0006] In summary, the embodiments of this specification are used to address the problems of difficulty in operating two-dimensional objects, high learning cost, low usability and usage efficiency in the cross-dimensional mixed interaction method in the current mixed reality system. Summary of the invention

[0007] In order to solve the problems existing in the prior art, the embodiments of this specification provide an interactive method, device and equipment for multi-dimensional mixed objects in three-dimensional space, which realizes the automatic switching effect of the selector on the three-dimensional and two-dimensional objects of the input device in the three-dimensional space without changing the input mode of the input device. When the user operates the two-dimensional object in the three-dimensional space, the user can use the operating habits of the traditional plane device to operate the two-dimensional object, thereby reducing the user's learning cost and improving the operating accuracy and the ease of use of the system.

[0008] In order to solve any of the above technical problems, the specific technical solutions of this specification are as follows:

[0009] On the one hand, an embodiment of the present specification provides an interaction method of multi-dimensional mixed objects in a three-dimensional space, the method comprising:

[0010] Receiving user operation information sent by an input device;

[0011] Converting the operation information into a 3D selector in a three-dimensional space, where the 3D selector is used for the user to interact with content in the three-dimensional space;

[0012] When the 3D selector selects a two-dimensional object in the three-dimensional space, the 3D selector is converted into a 2D selector in the two-dimensional object, and interaction information of the 2D selector in the two-dimensional object is calculated according to the change of the operation information, and the content in the two-dimensional object is interacted according to the interaction information.

[0013] Furthermore, the operation information includes a spatial position of the input device.

[0014] Furthermore, calculating the interaction information of the 2D selector in the two-dimensional object according to the change of the operation information further includes:

[0015] The plane position of the 2D selector in the two-dimensional object is calculated according to the change of the spatial position of the input device.

[0016] Furthermore, after converting the 3D selector into a 2D selector in the two-dimensional object, the method further includes:

[0017] When the plane position of the 2D selector exceeds the two-dimensional object, the 2D selector is converted into the 3D selector, and the interaction with the content in the two-dimensional object is terminated, so that the user can interact with the content outside the two-dimensional object in the three-dimensional space through the 3D selector.

[0018] Furthermore, when the plane position of the 2D selector is located at the boundary of the two-dimensional object, if the plane position calculated based on the spatial position change of the input device is still moving in the direction of the boundary, the movement of the 2D selector in this direction is stopped in the two-dimensional object, and it is determined whether the distance of the calculated plane position relative to the boundary exceeds a first preset threshold value. If so, it is determined that the plane position of the 2D selector exceeds the two-dimensional object.

[0019] Furthermore, if the plane position calculated according to the change in the spatial position of the input device still moves in the direction of the boundary, after stopping the movement of the 2D selector in the direction in the two-dimensional object, the method further includes:

[0020] Determining whether the distance of the calculated plane position relative to the boundary exceeds a second preset threshold value;

[0021] If not, maintaining the size of the 2D selector in the two-dimensional object;

[0022] If so, before the moving distance exceeds the first preset threshold, as the moving distance gradually increases, the size of the 2D selector in the two-dimensional object is gradually reduced.

[0023] Furthermore, calculating the plane position of the 2D selector in the two-dimensional object according to the spatial position change of the input device further includes:

[0024] Transforming the spatial position change of the input device from a three-dimensional world coordinate system into a two-dimensional local coordinate system to obtain a plane position change of the spatial position change of the input device in the local coordinate system;

[0025] The plane position of the 2D selector in the two-dimensional object is calculated according to the plane position change and a predetermined ratio.

[0026] Furthermore, transforming the spatial position change of the input device from a three-dimensional world coordinate system to a two-dimensional local coordinate system to obtain a planar position change of the spatial position change of the input device in the local coordinate system further includes:

[0027] The spatial position change of the input device is projected onto a plane parallel to the plane where the two-dimensional object is located to obtain the plane position change.

[0028] Furthermore, converting the operation information into a 3D selector in three-dimensional space further comprises:

[0029] The position of the 3D selector in the three-dimensional space is calculated according to the spatial position of the input device.

[0030] Furthermore, the 3D selector selects the two-dimensional object in the three-dimensional space further comprises:

[0031] When the 3D selector intersects with the two-dimensional object, the two-dimensional object is pre-selected, and after receiving a confirmation instruction from the user, the two-dimensional object is selected.

[0032] Furthermore, after preselecting the two-dimensional object, the method further includes:

[0033] A pre-selected two-dimensional object is highlighted in the three-dimensional space.

[0034] Further, the input device includes a physical peripheral or a human tracking device;

[0035] When the input device is a physical peripheral device, receiving the confirmation instruction from the user includes: receiving a confirmation instruction sent by the user through a key on the physical peripheral device;

[0036] When the input device is a human body tracking device, receiving the user's confirmation instruction includes: determining whether the duration of the intersection of the 3D selector whose position and rotation are determined by the human body tracking device and the two-dimensional object exceeds a predetermined duration; if so, receiving the confirmation instruction.

[0037] On the other hand, the embodiment of this specification also provides an interactive device for multi-dimensional mixed objects in a three-dimensional space, including:

[0038] An operation information receiving unit, used for receiving the user's operation information sent by the input device;

[0039] a 3D selector conversion unit, configured to convert the operation information into a 3D selector in a three-dimensional space, wherein the 3D selector is used for the user to interact with the content in the three-dimensional space;

[0040] A 2D selector conversion unit is used to convert the 3D selector into a 2D selector in the two-dimensional object when the 3D selector selects the two-dimensional object in the three-dimensional space, calculate the interaction information of the 2D selector in the two-dimensional object according to the change of the operation information, and interact with the content in the two-dimensional object according to the interaction information.

[0041] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0042] Finally, an embodiment of this specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0043] Through the method of the embodiment of this specification, a 3D selector corresponding to the operation information of the user input device is generated in the three-dimensional space. When the user controls the 3D selector to select a two-dimensional object in the three-dimensional space through the input device, the 3D selector is converted into a 2D selector in the two-dimensional object, and the interaction information of the 2D selector in the two-dimensional object is calculated according to the change of the operation information of the user input device, and the content in the two-dimensional object is interacted according to the interaction information. The embodiment of this specification realizes that in the three-dimensional space of multi-dimensional mixed objects, when the 3D selector selects a two-dimensional object in the three-dimensional space, the 3D selector is converted into a 2D selector, and the user still operates the 2D selector through the input device. Although the input device operates in the three-dimensional space, the 2D selector operates in the plane of the two-dimensional object, so the 2D selector is more convenient for the user to operate the content in the two-dimensional object, and is more in line with the user's operating habits of traditional two-dimensional devices to operate the two-dimensional object, thereby reducing the learning cost and improving the operation accuracy and system usability. The method of the embodiment of this specification converts the 3D selector into a 2D selector, eliminating the jitter caused by rotation. At the same time, users are still supported to use the 3D selector to operate content other than two-dimensional objects in the three-dimensional space. That is, when users operate multi-dimensional mixed objects in the three-dimensional space, the system automatically switches the 2D selector and the 3D selector according to the operation target and related interaction conditions without switching the display of the content in the three-dimensional space, thereby improving the user's interactive experience with multi-dimensional mixed objects in the three-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flow chart of a method for interacting with multi-dimensional mixed objects in a three-dimensional space according to an embodiment of this specification;

[0046] Figure 2 It is a schematic diagram of the flow of visual feedback processing of a 2D selector in an embodiment of this specification;

[0047] Figure 3 It is a schematic diagram of a process of calculating the plane position of the 2D selector in the two-dimensional object according to the spatial position change of the input device in an embodiment of this specification;

[0048] Figure 4 It is a schematic diagram of the structure of a device for interacting with multi-dimensional mixed objects in a three-dimensional space according to the present specification;

[0049] Figure 5 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification;

[0050] Figure 6 , Figure 7 and Figure 8 It is a schematic diagram showing the interaction between a 3D selector generated by a user using any one of input devices such as physical peripherals, image recognition, and eye movement recognition and content in a 3D space in an embodiment of this specification;

[0051] Fig. 9 Shown is a schematic diagram of a scenario example in an embodiment of this specification;

[0052] Fig.10 and Fig.11 It is a schematic diagram showing the user switching from interacting with a 3D model application to interacting with a 2D webpage application in an embodiment of this specification;

[0053] Fig.12 , Fig.13 and Fig.14 The diagram is a schematic diagram showing the user switching from interacting with a 2D web application to interacting with a 2D recording application in an embodiment of the present specification.

[0054]

Description of reference numerals

[0055] 401, operation information receiving unit;

[0056] 402, 3D selector conversion unit;

[0057] 403, 2D selector conversion unit;

[0058] 502. Computer equipment;

[0059] 504, processor;

[0060] 506. Memory;

[0061] 508, driving mechanism;

[0062] 510, input / output module;

[0063] 512. Input devices;

[0064] 514. Output device;

[0065] 516. Presentation equipment;

[0066] 518. Graphical user interface;

[0067] 520, network interface;

[0068] 522, communication link;

[0069] 524. Communication bus. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0071] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0072] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] In order to solve the problems existing in the prior art, the embodiments of this specification provide an interaction method for multi-dimensional mixed objects in a three-dimensional space, which enables the user to continue to operate by combining hand movement and gestures without changing the input method of the input device. The input device automatically switches the selectors on the three-dimensional objects and two-dimensional objects in the three-dimensional space. When the user operates the two-dimensional object in the three-dimensional space, the user can use the operating habits of the traditional planar device to operate the two-dimensional object, thereby reducing the learning cost and improving the operating accuracy and system usability. Figure 1Shown is a flow chart of a method for interacting with a multi-dimensional hybrid object in a three-dimensional space according to an embodiment of the present specification. This figure describes the process of a user interacting with a multi-dimensional hybrid object in a three-dimensional space through an input device, but may include more or fewer operating steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many, and does not represent the only execution order. When the system or device product is executed in practice, it may be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 1 As shown, the method may include:

[0074] Step 101: receiving user operation information sent by an input device;

[0075] Step 102: converting the operation information into a 3D selector in a three-dimensional space, where the 3D selector is used for the user to interact with the content in the three-dimensional space;

[0076] Step 103: When the 3D selector selects a two-dimensional object in the three-dimensional space, the 3D selector is converted into a 2D selector in the two-dimensional object, and the interaction information of the 2D selector in the two-dimensional object is calculated according to the change of the operation information, and the content in the two-dimensional object is interacted according to the interaction information.

[0077] Through the method of the embodiment of this specification, a 3D selector corresponding to the operation information of the user input device is generated in the three-dimensional space. When the user controls the 3D selector to select a two-dimensional object in the three-dimensional space through the input device, the 3D selector is converted into a 2D selector in the two-dimensional object, and the interaction information of the 2D selector in the two-dimensional object is calculated according to the change of the operation information of the user input device, and the content in the two-dimensional object is interacted according to the interaction information. The embodiment of this specification realizes that in the three-dimensional space of multi-dimensional mixed objects, when the 3D selector selects a two-dimensional object in the three-dimensional space, the 3D selector is converted into a 2D selector, and the user still operates the 2D selector through the input device. Although the input device is operated in the space, the 2D selector moves in the plane of the two-dimensional object. Therefore, the 2D selector is more convenient for the user to operate the content in the two-dimensional object, and is more in line with the user's operating habits of traditional two-dimensional devices to operate the two-dimensional object, thereby reducing the learning cost and improving the operation accuracy and system usability. At the same time, users are still supported to use the 3D selector to operate content other than two-dimensional objects in the three-dimensional space. That is, when users operate multi-dimensional mixed objects in the three-dimensional space, the system automatically switches the 2D selector and the 3D selector according to the operation target and related interaction conditions without the need for users to switch manually, thereby improving the user's interaction experience with multi-dimensional mixed objects in the three-dimensional space.

[0078] In the embodiments of this specification, multi-dimensional hybrid objects include at least two-dimensional objects and three-dimensional objects, and the objects include all visual content in the system, such as applications, components, and content in the environment. Two-dimensional objects such as "webpage" applications and "recording" applications have their content rendered in a plane, so there are only local two-dimensional movement, click and other operations, but no depth movement, click and other operations. Three-dimensional objects such as three-dimensional "cups" and "tea tables" can be operated in three dimensions: horizontal, vertical and depth.

[0079] It should be noted that the positions of two-dimensional objects and three-dimensional objects in three-dimensional space are not fixed. Users can adjust the positions and sizes of two-dimensional and three-dimensional objects in three-dimensional space as needed, which will not be elaborated here.

[0080] When a 3D selector is used to interact with content in a 3D space in the current 3D space, the 3D selector is generated based on the operation information sent by the user's input device. The 3D selector can be in the form of rays, 3D colliders, conical colliders, cubes, cones, etc. Taking the most commonly used rays as an example, the system converts the operation information of the user's input device into rays in the 3D space. The rays select the content in the 3D space and realize the interaction in the form of movement and rotation through gestures or input devices. The rays point to the target and pre-select the current interactive target, and then cooperate with key operations, make specified gestures, and other operations to complete the interaction. However, the 3D selector is suitable for operating 3D content, but not for operating 2D content in 3D space. For example, when a user uses a 3D selector to operate a 2D object, the jitter of the 3D selector caused by the jitter of the user's input device will be magnified. At the same time, when the 3D selector operates a 2D object, there are problems such as it is not easy to directly interact with smaller text, buttons, etc. in the 2D object due to the large diameter or cross-section of the selector, and the speed is slow or unintuitive.

[0081] Therefore, in the embodiments of the present specification, when the 3D selector selects two-dimensional content in three-dimensional space, it means that the user starts to interact with the two-dimensional object. At this time, the ray is converted into a 2D selector and the ray disappears in the three-dimensional space. The 2D selector interacts in the plane of the two-dimensional object. The 2D selector can be in the form of a two-dimensional pointer, etc. Compared with rays, two-dimensional pointers are more in line with the user's existing 2D device interaction method.

[0082] It should be noted that in addition to a two-dimensional pointer, the 2D selector can also be a three-dimensional pointer. In some cases, a two-dimensional object may include three-dimensional content, which is possible in an XR environment and has a good visual effect. In this case, the three-dimensional pointer can interact with the three-dimensional content in the two-dimensional object, and this specification does not limit this.

[0083] In addition, after the 3D selector is converted to a 2D selector, a projection of the user's hand is generated in three-dimensional space, that is, the position of the hand is projected on the plane where the two-dimensional object is located. Usually, the position of the wrist joint is used for projection. Compared with other joints, the wrist is more stable, and the conversion from a three-dimensional position to a two-dimensional position reduces one dimension of hand shaking, and the accuracy of selection will also increase to a certain extent. If the 3D selector is a ray, because of the angular shaking of the ray, the longer the distance between the position of the 3D selector and the pointer, the more significant the shaking. There is no angle in the embodiment of this specification, so there is no problem of greater shaking as the distance is longer.

[0084] In the embodiment of this specification, when the 3D selector is converted into a 2D selector in a two-dimensional object, the 2D selector may appear at the center or edge of the 2D object, which is not limited in the embodiment of this specification.

[0085] Then, the interaction information of the two-dimensional pointer in the two-dimensional object is calculated according to the changes in the operation information of the user input device. The interaction information may include the position of the two-dimensional pointer, the user's interaction actions on the content selected by the two-dimensional pointer, etc. Based on the interaction information, the content in the two-dimensional object is selected, moved, and other forms of interaction are performed.

[0086] According to one embodiment of the present specification, the operation information includes a spatial position of the input device.

[0087] Calculating the interaction information of the 2D selector in the two-dimensional object according to the change of the operation information further includes:

[0088] The plane position of the 2D selector in the two-dimensional object is calculated according to the change of the spatial position of the input device.

[0089] It can be understood that the position of the 2D selector in the two-dimensional object is related to the spatial position of the user's input device. For example, when the user moves the input device, the position of the 2D selector will also move accordingly, so that the user can control the 2D selector through the input device to select content in the two-dimensional object, and then issue interactive actions on the selected content through buttons on the input device. The system controls the selected content according to the interactive actions to complete the user interaction.

[0090] According to an embodiment of the present specification, after converting the 3D selector to the 2D selector, the method further includes:

[0091] When the plane position of the 2D selector exceeds the two-dimensional object, the 2D selector is converted into the 3D selector, and the interaction with the content in the two-dimensional object is terminated, so that the user can interact with the content outside the two-dimensional object in the three-dimensional space through the 3D selector.

[0092] It can be understood that when the user wants to interact with three-dimensional content in three-dimensional space, he only needs to move the position of the input device in space. The system calculates the plane position of the 2D selector based on the change in the spatial position of the input device. When the user moves the input device so that the 2D selector leaves the two-dimensional object, it means that the user no longer interacts with the content in the two-dimensional object. Therefore, the system converts the 2D selector back to a 3D selector and ends the interaction with the content in the two-dimensional object. The user can continue to interact with the three-dimensional content in the three-dimensional space through the 3D selector.

[0093] When the user needs to interact with the content in the two-dimensional object or other two-dimensional objects again, the previous operation steps still need to be performed, that is, the two-dimensional object to be interacted with is selected through the 3D selector, and then the system converts the 3D selector into a 2D selector in the selected two-dimensional object, so that the user can interact with the content in the selected two-dimensional object.

[0094] According to one embodiment of the present specification, converting the operation information into a 3D selector in a three-dimensional space further comprises:

[0095] The position of the 3D selector in the three-dimensional space is calculated according to the spatial position of the input device.

[0096] In the embodiments of the present specification, the position of the user input device is also related to the position of the 3D selector in the three-dimensional space. For example, when the user moves the spatial position of the input device, the spatial position of the 3D selector in the three-dimensional space also moves accordingly. Combined with the user's operation of buttons on the input device, the 3D selector can interact with its selected content.

[0097] In an embodiment of the present specification, the judgment method for selecting a two-dimensional object by a 3D selector may be: only judging whether the 3D selector intersects with the two-dimensional object, or after determining that the 3D selector intersects with the two-dimensional object, combining some confirmation conditions. According to an embodiment of the present specification, in order to avoid the situation where a user attempts to operate three-dimensional content in a 3D space through a 3D selector, or the user operates the 3D selector to aim at three-dimensional content in a three-dimensional space, accidentally places the 3D selector in a two-dimensional object, causing the system to determine that the user needs to interact with the two-dimensional content in the two-dimensional object, and directly eliminates the 3D selector, affecting the user's interaction with the three-dimensional content, the embodiment of the present specification preferably uses a combination of pre-selection and confirmation to judge whether the user's intention is to select a two-dimensional object. Specifically, the 3D selector selecting the two-dimensional object in the three-dimensional space further includes:

[0098] When the 3D selector intersects with the two-dimensional object, the two-dimensional object is pre-selected, and after receiving a confirmation instruction from the user, the two-dimensional object is selected.

[0099] It can be understood that, when pre-selected, the three-dimensional content will generally have visual feedback such as highlighting, sinking, size change or color change, and stroke, prompting the user that the three-dimensional content is pre-selected. After the user confirms the instruction, the user can operate correctly and reduce the probability of misselection.

[0100] It should be noted that, because the 3D selector has information about the three-dimensional direction, and the two-dimensional object only has plane information, the intersection of the 3D selector and the two-dimensional object refers to the part of the 3D selector at the depth position of the plane where the two-dimensional object is located intersecting with the two-dimensional object, rather than the entire 3D selector intersecting with the two-dimensional object. For example, when the 3D selector is a ray, when the ray intersects with the two-dimensional object, it means that the two-dimensional object is pre-selected. It should be noted that the ray intersects with the collision body of the two-dimensional object. It is similar to the intersection of the 3D selector and the collision body of the two-dimensional object. In fact, the system will calculate the position and direction of the ray and calculate whether the collision body intersects with the ray through physical calculations, and return the first collision body that intersects with the ray. This event will also return the position of the intersection point, the normal direction and other information. The method of the embodiment of this specification will draw the pointer at this point, and draw a line segment or a semi-transparent gradient ray between the starting point and this point. It is worth noting that the collision body of the two-dimensional object is also three-dimensional. This type of collision body generally also has thickness, but the content displayed by the two-dimensional object is two-dimensional, and the two-dimensional object can also be rendered into a two-dimensional piece, which can be double-sided or single-sided. However, this does not affect the collision body being three-dimensional.

[0101] In the embodiments of this specification, the input device includes a physical peripheral or a human tracking device;

[0102] When the input device is a physical peripheral (such as a handle), receiving the confirmation instruction from the user includes: receiving the confirmation instruction sent by the user through a button on the physical peripheral.

[0103] It can be understood that when the user operates a physical peripheral such as a handle to make the 3D selector intersect with the two-dimensional object, and then operates the button on the physical peripheral to issue a confirmation command, the two-dimensional object can be selected, and then the 3D selector is converted into a 2D selector in the two-dimensional object. The user still controls the plane position of the 2D selector in the two-dimensional object through the physical peripheral, and controls the 2D selector through the physical peripheral to interact with the content in the two-dimensional object.

[0104] When the input device is a human tracking device (such as a head-controlled input device or an eye-movement input device), receiving a confirmation instruction from the user includes: determining whether the duration of continuous intersection of the 3D selector whose position and rotation are determined by the human tracking device and the two-dimensional object exceeds a predetermined duration; if so, receiving the confirmation instruction.

[0105] It can be understood that when the human body tracking device tracks the movement and / or rotation of the user-controlled interaction part so that the 3D selector intersects with the two-dimensional object, a specific operation is performed to select the two-dimensional object. Because it is different from the buttons of physical peripherals, the human body tracking device usually tracks the position and / or rotation of the user's body part used for interaction to identify the user's interaction actions. Therefore, the embodiment of this specification determines whether a specific interaction action is performed after the 3D selector and the two-dimensional object continue to intersect. If so, the two-dimensional object is selected, and then the 3D selector is converted into a 2D selector in the two-dimensional object. The user still controls the planar position of the 2D selector in the two-dimensional object by moving the user's interacting body part, and controls the 2D selector to interact with the content in the two-dimensional object through the interaction action of the user's body part identified by the human body tracking device.

[0106] It should be noted that when the human body tracking device tracks the movement and / or rotation of the user-controlled interaction part so that the 3D selector intersects with the two-dimensional object, as long as the user does not move the 3D selector out of the two-dimensional object, the intersection continues. During the comparison period, it is determined whether a specific interaction action is performed. For example, after the user controls the 3D selector to intersect with the two-dimensional object, even if the human body tracking device recognizes that the user's interaction part has shaken, causing the position of the 3D selector to change, as long as the two-dimensional object is not left, it will not affect the 2D selector's continued interaction with the two-dimensional object.

[0107] In addition, the human body tracking device can also track human body movements, such as blinking, and issue confirmation instructions through human body movements, which will not be repeated here.

[0108] In some other embodiments of the present specification, a two-dimensional object can also be selected by a combination of 3D selectors corresponding to multiple different input devices. For example, there are 3D selectors of a handheld input device or an input device based on image recognition and a 3D selector of an eye-movement input device in a three-dimensional space. The user selects three-dimensional content through the 3D selector of the eye-movement input device and interacts with the selected three-dimensional content through the 3D selector of the handheld input device or the input device based on image recognition. If the user needs to interact with the two-dimensional object, the eye-movement input device can be operated to make the corresponding 3D selector point to the two-dimensional object, and then the two-dimensional object can be selected through the buttons on the handheld input device or the input device based on image recognition. At this time, the 3D selector corresponding to the handheld input device or the input device based on image recognition or the 3D selector corresponding to the eye-movement input device (which can be set by the system) is converted into a 2D selector in the two-dimensional object, and the user begins to interact with the two-dimensional object.

[0109] In some other embodiments of the present specification, in order to improve the user's interactive experience, after pre-selecting the two-dimensional object, the method further includes:

[0110] A pre-selected two-dimensional object is highlighted in the three-dimensional space.

[0111] Exemplarily, when a ray in three-dimensional space points to a two-dimensional object, the system determines that the user has pre-selected the two-dimensional object and highlights the two-dimensional object. The highlighting method may include highlighting, displaying prompt information, etc., which is not limited in the embodiments of this specification.

[0112] In the embodiments of the present specification, the user uses an input device to interact with the content in the three-dimensional space. When the user operates the input device to control the 3D selector to select a two-dimensional object, the 3D selector is converted to a 2D selector. However, the position of the user's input device has not changed. Only the virtual selector presented in the three-dimensional space has changed from a 3D selector to a 2D selector. At this time, the position of the user's hand may not be a comfortable position. For example, the two-dimensional object is at a higher position in the three-dimensional space. The user needs to move the physical peripheral such as a handle upward to make the 3D selector select the two-dimensional object. After that, the 3D selector is converted to a 2D selector, but the user still holds the input device high. At this time, the operation is very uncomfortable, especially the hand-raising operation is easy to cause fatigue, because the position of the 2D selector is related to the position of the physical peripheral (handle). However, if the user moves the physical peripheral (handle) to a comfortable position, the plane position of the 2D selector may also exceed the two-dimensional object, causing the 2D selector to be converted to a 3D selector again.

[0113] In response to this situation, according to an embodiment of the present specification, when the plane position of the 2D selector is located at the boundary of the two-dimensional object, if the plane position calculated based on the spatial position change of the input device is still moving in the direction of the boundary, the movement of the 2D selector in the direction is stopped in the two-dimensional object, and a determination is made as to whether the distance of the calculated plane position relative to the boundary exceeds a first preset threshold value; if so, it is determined that the plane position of the 2D selector exceeds the two-dimensional object.

[0114] In an embodiment of the present specification, when a user moves the position of an input device so that the plane position of a 2D selector is located at a boundary of a two-dimensional object, if the user still moves the input device in the direction of the boundary, the calculated plane position of the 2D selector is also moved in the direction of the boundary. In the two-dimensional local coordinate system of the two-dimensional object, the 2D selector is stopped from moving in the direction of the boundary (but movement in other directions of boundaries is not restricted), and a determination is made as to whether the distance of the calculated plane position relative to the boundary exceeds a first preset threshold value. If so, it is determined that the plane position of the 2D selector exceeds the two-dimensional object, the 2D selector is converted into a 3D selector, and the interaction with the content in the two-dimensional object is ended.

[0115] For example, if a two-dimensional object is located at a higher position in the three-dimensional space, the user moves the input device upward so that the 3D selector selects the two-dimensional object, and then the 3D selector is converted to a 2D selector because the user is holding up the input device at this time. Then the user can move the input device downward. Although the 2D selector moves downward with the movement of the input device, when the 2D selector moves to the boundary of the two-dimensional object, the user still moves the input device downward. At this time, although the calculated plane position of the 2D selector still moves in the direction of the boundary (that is, the calculated plane position of the 2D selector has exceeded the boundary of the two-dimensional object), ), but the embodiment of the present specification still keeps the 2D selector at the position of the boundary. If the user moves the input device to a comfortable position, the calculated distance between the plane position of the 2D selector and the boundary is less than the first preset threshold value, then the system does not determine that the 2D selector exceeds the boundary, and the 2D selector is still at the boundary position. At this time, the user has moved the input device to a comfortable position. The user only needs to move the input device up a little bit to move the 2D selector into the boundary of the two-dimensional object. Therefore, the user can operate the input device at a comfortable position to realize the interaction between the 2D selector and the content in the two-dimensional object.

[0116] If the user moves the input device in one direction all the time, it can be understood as "swinging" the input device, and the 2D selector can be "swung" out of the two-dimensional object, ending the interaction with the two-dimensional object. After the 2D selector is "swung" out of the two-dimensional object, it is immediately converted into a 3D selector, and the user interacts with other content in the three-dimensional space through the 3D selector.

[0117] Through the above method, the user can achieve the purpose of ending the interaction with the two-dimensional object through simple and continuous operations. This process is faster than activating the interface through gestures or buttons.

[0118] In the embodiment of this specification, the first preset threshold is a value in the system, but the user does not know how much distance to move to reach the first preset threshold, which may cause the user to mistakenly move the 2D selector beyond the first preset threshold. In view of this situation, according to an embodiment of this specification, Figure 2 As shown, if the plane position calculated according to the change in the spatial position of the input device is still moving in the direction of the boundary, then after stopping the movement of the 2D selector in the direction in the two-dimensional object, the method further includes:

[0119] Step 201: determining whether the distance of the calculated plane position relative to the boundary exceeds a second preset threshold value;

[0120] Step 202: If not, maintaining the size of the 2D selector in the two-dimensional object;

[0121] Step 203: If yes, before the moving distance exceeds the first preset threshold, the size of the 2D selector in the two-dimensional object is gradually reduced as the moving distance gradually increases.

[0122] In an embodiment of the present specification, when the 2D selector stays on the boundary of a two-dimensional object, if the user still moves the 2D selector toward the boundary, the size of the 2D selector in the two-dimensional object remains unchanged when the movement starts. As the moving distance increases, when the second preset threshold value is exceeded, the 2D selector begins to gradually decrease, prompting the user that the 2D selector is about to become a 3D selector. When the first preset threshold value is reached, the 2D selector can be reduced to the set minimum value. When the first preset threshold value is exceeded, the 2D selector is immediately converted to a 3D selector.

[0123] It can be understood that through Figure 2 The method shown can give the user timely and accurate feedback after the user moves the 2D selector to the boundary of the two-dimensional object, thereby improving the user's operating efficiency. For example, when a user double-clicks a target in a personal computer, the user needs to see the target show a selected state, and then load and open the target. If the target does not have any feedback, the user may repeatedly double-click the target, increasing the computing burden of the personal computer. Similarly, in the embodiment of this specification, when the 2D selector moves to the boundary of the two-dimensional object and continues to move in the direction of the boundary, if the moving distance exceeds the second preset threshold value but does not exceed the first preset threshold value, the system still maintains the interactive state of the two-dimensional object, but only restricts the 2D selector from moving out of the two-dimensional object, and gradually reduces the size of the 2D selector as the moving distance increases, prompting the user to move to what state to end the interaction with the two-dimensional object, and at the moment when the moving distance exceeds the first preset threshold value, the 2D selector is immediately converted to a 3D selector, ending the interaction with the two-dimensional object and starting to interact with other content in the three-dimensional space. In this way, more accurate visual feedback and prompts can be given to the user, prompting the user's operation level and improving the user's operation accuracy. In addition, gradually reducing the size of the 2D selector can also avoid the jump in the conversion of the 2D selector to the 3D selector, thereby improving the user's interactive experience.

[0124] In the embodiments of the present specification, when the 3D selector selects a two-dimensional object, the 3D selector is converted into a 2D selector within the two-dimensional object. However, both the 2D selector and the 3D selector are operated through the user's input device (such as a physical peripheral such as a handle), and the input device is operated in space, that is, the position change of the input device belongs to the spatial position change, and the change of the position of the 3D selector in the three-dimensional space also belongs to the spatial position change. However, the position change of the 2D selector in the two-dimensional object inputs a plane position change. Therefore, when the 3D selector is converted into a 2D selector, the spatial position change of the input device needs to be specially processed to match the plane position change of the 2D selector.

[0125] Specifically, according to one embodiment of this specification, Figure 3 As shown, calculating the plane position of the 2D selector in the two-dimensional object according to the spatial position change of the input device further includes:

[0126] Step 301: transforming the spatial position change of the input device from a three-dimensional world coordinate system into a two-dimensional local coordinate system of a 2D object, and obtaining a plane position change of the spatial position change of the input device in the two-dimensional local coordinate system;

[0127] Step 302: Calculate the plane position of the 2D selector in the two-dimensional object according to the plane position change and a predetermined ratio.

[0128] In the embodiments of the present specification, the spatial position change of the input device is reduced in dimensionality and converted into a plane position change in a two-dimensional local coordinate system. Then, the plane position of the 2D selector in the two-dimensional space is calculated based on the plane position change and a predetermined ratio. The predetermined ratio is designed by the designer. Then, this ratio is multiplied by the longest side of the two-dimensional object to obtain the distance moved by the 2D selector. The purpose is to map the limited hand-operated input device movement to the position of the entire 2D window, so that the user can reach (control) the content within the entire 2D window.

[0129] In some other scenarios, in order to allow users to operate input devices more easily, more labor-saving, and with smaller amplitudes to operate 2D objects of different sizes. Set a threshold t, which is the maximum distance that the projection of the hand on the plane where the 2D object is located moves in one direction, such as a threshold t of 0.2 meters. This design is similar to the operation of a mouse on a computer desktop in reality. If the user's operation target is small, such as a 0.18-meter-wide 2D object, this movement method allows the user's hand to move within a range of 0.2 meters to cover the entire operation range of the 2D object, which is similar to the experience of operating a tablet device in reality. In practice, the width of many 2D objects far exceeds the limit range of the user's hand relative to the body when operating the input device, such as the size of a TV, the size of a cinema screen, and even some longer 2D content, such as some launcher interfaces and other objects with large horizontal and vertical ratios. For these objects, this percentage-based movement method can achieve full-range operation of the 2D object regardless of the actual size of the 2D object, that is, operation of all the contents of the 2D object, by moving the hand within a range of 0.2 meters. If the percentage method is not used, but a one-to-one projection method is used, that is, the projection position of the hand on the plane where the 2D object is located moves a certain distance in a certain direction, and the 2D selector also moves the same distance along the same position, for larger content, although the user has moved for a long time, the moving distance of the 2D selector only occupies a very small part of the two-dimensional object, which will make the user feel that the operation is very difficult.

[0130] Furthermore, transforming the spatial position change of the input device from a three-dimensional world coordinate system to a two-dimensional local coordinate system to obtain a planar position change of the spatial position change of the input device in the local coordinate system further includes:

[0131] The spatial position change of the input device is projected onto a plane parallel to the plane where the two-dimensional object is located to obtain the plane position change.

[0132] It can be understood that the projection process is a calculation process of the system, and this projection may be invisible to the user. The system projects the spatial position change of the input device on a plane parallel to the plane where the two-dimensional object is located. For example, if the plane where the two-dimensional object is located is a vertical plane, then the projection only retains the change information of the spatial position change of the input device on the vertical plane, thereby converting the spatial position change of the input device into a plane position change.

[0133] For example, Figure 6 , Figure 7 and Figure 8As shown, by default, the user uses a 3D selector generated by any of the input devices such as physical peripherals, image recognition, and eye movement recognition to interact with the content in the 3D space. The target pointed by the 3D selector determines the current interaction target, and then cooperates with the operation of confirming the action such as buttons and gestures to complete the interaction action. When the user wants to interact with 2D content, it automatically switches to the mode of interacting with 2D content. The judgment condition can be any one or a combination of the following:

[0134] The user's gaze is directed to the plane where the 2D content is located (such as a 2D window or a 2D UI panel);

[0135] The 3D selector that the user interacts with (head-controlled 3D selector or 3D selector controlled by physical peripherals or image recognition devices) points to the plane where the 2D content is located;

[0136] The user interacts with the plane where the 2D content is located through the 3D selector, such as clicking.

[0137] After the user starts to interact with the 2D content, the currently interacting 3D selector disappears, and the Pointer appears at the center of the plane where the 2D content is located or at the edge where the 3D selector intersects with the plane where the 2D content is located (not specifically limited). At this time, the movement of the 3D selector controlled by the physical peripheral or image recognition device will be projected on the plane where the window is located, and the movement of the 3D selector controlled by the physical peripheral or image recognition device in 3D space will be projected onto a plane parallel to the plane where the 2D content is located. When the projection of the corresponding 3D selector controlled by the physical peripheral or image recognition device moves a certain distance in a certain direction up, down, left, or right on a plane parallel to the plane where the 2D content is located, the Pointer on the window will also move a corresponding distance in the same direction up, down, left, or right in proportion. This is similar to the logic of a mouse operating on the screen. Here, the movement of the 3D selector controlled by the physical peripheral or image recognition device refers to: If the physical peripheral is the position of the peripheral, if the image recognition device determines the device position by identifying the hand, it refers to the movement of the wrist (if the hand is relatively stable, you can use the index finger tip + anti-shake) after the position is projected onto the 2D window plane (here is the core of comfortable operation). In order to facilitate user operation, especially when the user operates the input device, raising the hand to operate is prone to cause fatigue, and to reduce the amplitude and time of the user's hand raising operation when operating the input device, it is set that the user's physical peripheral or image recognition device only needs a certain distance to operate the Pointer in the same direction on the 2D window (physical peripheral or image The distance of the projection movement of the 3D selector controlled by the physical peripheral or image recognition device on the plane parallel to the 2D content is a percentage of the preset distance multiplied by the length of the longest side of the 2D content, which is the distance that the Pointer moves in the 2D window. This method can be used to achieve that the 3D selector controlled by the physical peripheral or image recognition device moves a very short distance, and the Pointer can move a longer distance in the 2D content according to the percentage. The preset distance can be adjusted to meet the needs of different users, allowing users to operate the Pointer on the 2D content more sensitively or accurately.

[0138] This method has no restrictions on the scale and size of the 2D window. When the Pointer moves to the boundary, the coordinate direction of the Pointer corresponding to the boundary will be restricted, which visually appears as the Pointer sliding on the boundary. For example, when the Pointer moves to the upper boundary of the 2D window, the Pointer will slide left and right but will not move upward. However, the Pointer's downward movement is not restricted.

[0139] The judgment conditions for ending the interaction with 2D content and switching to the interaction with 3D content are as follows: when the Pointer stays at the upper boundary, the "hand" continues to move upward for a distance, and the size of the Pointer remains unchanged. When the second threshold value is exceeded and the 3D selector controlled by the physical peripheral or image recognition device continues to move upward, the Pointer will become smaller until it continues to move for a distance. When the first threshold value is exceeded, the Pointer will become smaller until it disappears. When it disappears, the Pointer is closed and the 3D selector is activated. The advantage of this is that the user can adjust the relative position of the 3D selector controlled by the physical peripheral or image recognition device and the Pointer by letting the Pointer stay at the boundary, making the operation in a certain direction easier. At the same time, when you do not want to continue to operate the 2D window, you can stop the interaction with the 2D window by swinging the 3D selector controlled by the physical peripheral or image recognition device more significantly. And in the following operations, the 2D window will not be operated by mistake.

[0140] Examples of specific usage scenarios:

[0141] The application described in this usage scenario can be in the form of 3D application, 2D application, 3D and 2D hybrid application, single-person application, multi-person collaborative application, etc. The input device takes an image recognition device operating a 3D selector as an example.

[0142] For example, a prosecutor user opens a 3D model application to view an evidence model to find evidence, searches for relevant records of the evidence model in a 2D web application, and records relevant findings through voice in a 2D recording application. The scenario diagram is as follows Fig. 9 The user's usage process is as follows:

[0143] 1. The user uses a combination of hands and rays to remotely operate the 3D model, move, zoom in and rotate the model, and carefully view the details of the model. At close range, the user can also directly touch the target with his hands for direct interaction.

[0144] 2. If Figure 10-11 As shown, the user looks at the 2D web application, the eye movement selection ray hits the 2D web application, the 2D web application is pre-selected, the user pinches the thumb and index finger together to select the 2D web application through the Pinch behavior, and starts to interact with the 2D web application.

[0145] 3. Such as Fig.11 As shown, at this time, the ray on the user's hand disappears, and a Pointer appears in the center of the 2D web application. The user moves his hand up, down, left, and right to operate the Pointer in the 2D web application in a similar way to a mouse, and achieves the effect of clicking by pinching the thumb and index finger to query the relevant records and data of the evidence in the database.

[0146] 4. During the query process, if the user unconsciously looks at other applications outside the 2D web application while thinking, it will not affect the operation of the Pointer in the 2D web application. Because the Pointer appears in the center of the web page at the beginning, the user's hand may not be in a comfortable position at this time. For example, the user's left hand is in a very right position. The user can move the Pointer to the left edge of the 2D web application. At this time, the Pointer will be stuck at the left edge of the 2D web application. Although the Pointer can move up and down and to the right, it will not continue to move to the left. If the user continues to move his hand to the left, the Pointer will shrink. As long as the Pointer does not disappear, the user can continue to interact with the 2D web application. When the user feels that his hand has moved to a comfortable position, he can move his hand to the right and start to operate the 2D web application comfortably.

[0147] 5. If the user finds a new clue at this time, he wants to record it. Figure 12-13 As shown, the user looks at the 2D recording application, the 2D recording application is pre-selected, and then the user's hand moves in the direction of the 2D recording application. The Pointer stops moving after moving to the boundary. The user continues to move his hand in the direction of the 2D recording application. The Pointer first becomes smaller (triggering the second threshold value) and then disappears (triggering the first threshold value). At this time, a ray appears, and the user stops interacting with the 2D web application.

[0148] 6. Such as Fig.14 As shown, the 2D recording application has been pre-selected. The user pinches the thumb and index finger together to select the 2D recording application and starts to interact with the 2D recording application. The interaction process is the same as that of the 2D web application. After the Pointer appears in the center of the 2D recording application, the user moves the Pointer in the 2D recording application by moving the hand up, down, left, and right. The user moves the Pointer to the recording button and pinches the thumb and index finger together to trigger the click event, triggering the recording function and recording the clues just found.

[0149] 7. If the user needs to continue to view the 3D model application to find new clues, the user's hand moves toward the 3D model, and the Pointer moves toward the boundary of the 2D recording application in the direction of the 3D model and stays there after reaching the boundary. The user's hand continues to move toward the 3D model, and the Pointer becomes smaller (triggering the second threshold value) and disappears (triggering the first threshold value), ending the interaction with the 2D recording application, and the ray automatically appears.

[0150] 8. The user continues to use rays to move, scale and rotate the 3D model application to find new clues.

[0151] Based on the same inventive concept, the embodiment of this specification also provides an interactive device for multi-dimensional mixed objects in a three-dimensional space, such as Figure 4 As shown, including:

[0152] An operation information receiving unit 401 is used to receive user operation information sent by an input device;

[0153] A 3D selector conversion unit 402, configured to convert the operation information into a 3D selector in a three-dimensional space, wherein the 3D selector is used for the user to interact with the content in the three-dimensional space;

[0154] The 2D selector conversion unit 403 is used to convert the 3D selector into a 2D selector in the two-dimensional object when the 3D selector selects the two-dimensional object in the three-dimensional space, calculate the interaction information of the 2D selector in the two-dimensional object according to the change of the operation information, and interact with the content in the two-dimensional object according to the interaction information.

[0155] The beneficial effects obtained by the above-mentioned device are consistent with the beneficial effects obtained by the above-mentioned method, and will not be described in detail in the embodiments of this specification.

[0156] like Figure 5 As shown, a computer device provided in an embodiment of this invention is provided. The apparatus in this invention can be a computer device in this embodiment. The method of this invention is executed. The computer device 502 may include one or more processors 504, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 502 may also include any memory 506, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 506 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 502. In one case, when the processor 504 executes an associated instruction stored in any memory or a combination of memories, the computer device 502 may perform any operation of the associated instruction. The computer device 502 also includes one or more drive mechanisms 508 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0157] The computer device 502 may also include an input / output module 510 (I / O) for receiving various inputs (via input devices 512) and for providing various outputs (via output devices 514). A specific output mechanism may include a presentation device 516 and an associated graphical user interface (GUI) 518. In other embodiments, the input / output module 510 (I / O), the input device 512, and the output device 514 may not be included, and the computer device 502 may be used as a computer device in a network. The computer device 502 may also include one or more network interfaces 520 for exchanging data with other devices via one or more communication links 522. One or more communication buses 524 couple the components described above together.

[0158] The communication link 522 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 522 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0159] The embodiments of the present specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0160] The embodiments of the present specification also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the above method.

[0161] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0162] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0163] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0165] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0167] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0169] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A method for interacting multi-dimensional mixed objects in a three-dimensional space, characterized in that: The method comprises: Receiving user operation information sent by an input device; Converting the operation information into a 3D selector in a three-dimensional space, where the 3D selector is used for the user to interact with content in the three-dimensional space; When the 3D selector selects a two-dimensional object in the three-dimensional space, the 3D selector is converted into a 2D selector in the two-dimensional object, and interaction information of the 2D selector in the two-dimensional object is calculated according to the change of the operation information, and the content in the two-dimensional object is interacted according to the interaction information.

2. The method according to claim 1, characterized in that The operation information includes the spatial position of the input device.

3. The method according to claim 2, characterized in that Calculating the interaction information of the 2D selector in the two-dimensional object according to the change of the operation information further includes: The plane position of the 2D selector in the two-dimensional object is calculated according to the change of the spatial position of the input device.

4. The method according to claim 3, characterized in that After converting the 3D selector into a 2D selector in the two-dimensional object, the method further includes: When the plane position of the 2D selector exceeds the two-dimensional object, the 2D selector is converted into the 3D selector, and the interaction with the content in the two-dimensional object is terminated, so that the user can interact with the content outside the two-dimensional object in the three-dimensional space through the 3D selector.

5. The method according to claim 4, characterized in that The method also includes: when the plane position of the 2D selector is located at the boundary of the two-dimensional object, if the plane position calculated according to the spatial position change of the input device is still moving in the direction of the boundary, then stopping the movement of the 2D selector in the direction in the two-dimensional object, and judging whether the distance of the calculated plane position relative to the boundary exceeds a first preset threshold value; if so, determining that the plane position of the 2D selector exceeds the two-dimensional object.

6. The method according to claim 5, characterized in that If the plane position calculated according to the change in the spatial position of the input device is still moving in the direction of the boundary, then after stopping the movement of the 2D selector in the direction in the two-dimensional object, the method further includes: Determining whether the distance of the calculated plane position relative to the boundary exceeds a second preset threshold value; If not, maintaining the size of the 2D selector in the two-dimensional object; If so, before the moving distance exceeds the first preset threshold, as the moving distance gradually increases, the size of the 2D selector in the two-dimensional object is gradually reduced.

7. The method according to claim 3, characterized in that Calculating the plane position of the 2D selector in the two-dimensional object according to the change in the spatial position of the input device further comprises: Transforming the spatial position change of the input device from a three-dimensional world coordinate system to a local coordinate system of a two-dimensional object to obtain a plane position change of the spatial position change of the input device in the local coordinate system; The plane position of the 2D selector in the two-dimensional object is calculated according to the plane position change and a predetermined ratio.

8. The method according to claim 7, characterized in that Transforming the spatial position change of the input device from a three-dimensional world coordinate system to a two-dimensional local coordinate system to obtain a planar position change of the spatial position change of the input device in the local coordinate system further comprises: The spatial position change of the input device is projected onto a plane parallel to the plane where the two-dimensional object is located to obtain the plane position change.

9. The method according to claim 2, characterized in that: Converting the operation information into a 3D selector in three-dimensional space further comprises: The position of the 3D selector in the three-dimensional space is calculated according to the spatial position of the input device.

10. The method according to claim 9, characterized in that The 3D selector selects the two-dimensional object in the three-dimensional space further comprising: When the 3D selector intersects with the two-dimensional object, the two-dimensional object is pre-selected, and after receiving a confirmation instruction from the user, the two-dimensional object is selected.

11. The method according to claim 10, characterized in that After preselecting the two-dimensional object, the method further includes: A pre-selected two-dimensional object is highlighted in the three-dimensional space.

12. The method according to claim 10, characterized in that The input device includes a physical peripheral or a human tracking device; When the input device is a physical peripheral device, receiving the confirmation instruction from the user includes: receiving a confirmation instruction sent by the user through a key on the physical peripheral device; When the input device is a human body tracking device, receiving the user's confirmation instruction includes: determining whether the duration of continuous intersection between the 3D selector whose position and rotation are determined by the human body tracking device and the two-dimensional object exceeds a predetermined duration; if so, receiving the confirmation instruction.

13. An interactive device for multi-dimensional mixed objects in three-dimensional space, characterized in that: The device comprises: An operation information receiving unit, used for receiving the user's operation information sent by the input device; a 3D selector conversion unit, configured to convert the operation information into a 3D selector in a three-dimensional space, wherein the 3D selector is used for the user to interact with the content in the three-dimensional space; A 2D selector conversion unit is used to convert the 3D selector into a 2D selector in the two-dimensional object when the 3D selector selects the two-dimensional object in the three-dimensional space, calculate the interaction information of the 2D selector in the two-dimensional object according to the change of the operation information, and interact with the content in the two-dimensional object according to the interaction information.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images