A fine selection method, device and equipment in three-dimensional space

Abstract

This article relates to a fine selection method, device and equipment in three-dimensional space. It includes determining the current interactive object of the user in the three-dimensional space; generating an interactive target plane according to the position of the current interactive object of the user in the three-dimensional space; mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane; selecting the interactive object according to the mapping information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane. Because the difficulty of selecting objects on the interactive target plane is much lower than the difficulty of selecting objects in the three-dimensional space, the difficulty of selecting interactive objects is reduced, and there is no need to add additional equipment, which improves the convenience of use and ultimately improves the user experience.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a method, device and equipment for fine selection in three-dimensional space. Background Art

[0002] Existing XR systems usually select content in space or define the selection range based on the rays generated by the 6Dof information of the interactive source. However, the accuracy of the input device directly affects the accuracy experience of selecting in space in this way, and the denser the arrangement of multiple candidate objects, the smaller the distance between objects, the smaller the objects, the farther the objects are from the user, and the greater the degree of mutual occlusion between objects, the more difficult it is to select accurately. In addition, the lack of continuous tactile feedback during the operation will increase the accuracy of the operation. Secondly, if the user uses his hands or a handheld handle to complete the selection operation, his hands or even forearms will become tired, affecting the interactive experience. Summary of the invention

[0003] To solve the problems existing in the prior art, the embodiments of this specification provide a method, device and equipment for fine selection in three-dimensional space, which maps the 6dof movement of the input source into a 2D movement on a target plane or a plane in the target space, and performs anti-shake processing on the movement according to scene- or context-related rules, and determines the range of the selected object based on the processed results.

[0004] In order to solve any of the above technical problems, the specific technical solutions of this specification are as follows:

[0005] On the one hand, an embodiment of this specification provides a fine selection method in a three-dimensional space, the method comprising:

[0006] Determine the user's current interactive object in three-dimensional space;

[0007] Generate an interactive target plane according to the position of the user's current interactive object in the three-dimensional space;

[0008] Mapping movement information of the user's input device in the three-dimensional space to the interactive target plane;

[0009] The interactive object is selected according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane.

[0010] Furthermore, the interactive object includes a two-dimensional interactive object and a three-dimensional interactive object.

[0011] Furthermore, if it is a two-dimensional interactive object, generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space further includes:

[0012] generating the interactive target plane at the position of the two-dimensional interactive object in the three-dimensional space;

[0013] Mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane further includes:

[0014] Mapping the two-dimensional movement information of the input device in the three-dimensional space corresponding to the interactive target plane to the interactive target plane;

[0015] Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane further includes:

[0016] The two-dimensional interactive object is selected according to the correspondence between the position corresponding to the mapping position information of the input device on the interactive target plane and the position information of the two-dimensional interactive object on the interactive target plane.

[0017] Furthermore, if it is a three-dimensional interactive object, generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space further includes:

[0018] generating the interactive target plane parallel to the near clipping plane at the position of the three-dimensional interactive object in the three-dimensional space;

[0019] The method further comprises: projecting all three-dimensional interactive objects in the field of view onto the interactive target plane to obtain position information of all three-dimensional interactive objects in the field of view on the interactive target plane;

[0020] Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane further includes:

[0021] The three-dimensional interactive object is selected according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane.

[0022] Furthermore, a current interaction mode of the input device is determined, where the interaction mode includes a suspended operation mode and a plane operation mode.

[0023] Furthermore, if the interaction mode is a plane operation mode, mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane further includes:

[0024] Mapping the two-dimensional movement information of the input device in the three-dimensional space to the interactive target plane;

[0025] Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane further includes:

[0026] The three-dimensional interactive object is selected according to the correspondence between the position corresponding to the two-dimensional mapping position information of the input device on the interactive target plane and the position information of the three-dimensional interactive object projected in the interactive target plane.

[0027] Furthermore, if the interaction mode is a suspended operation mode, mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane further includes:

[0028] Mapping the three-dimensional movement information of the input device in the three-dimensional space to the interactive target plane;

[0029] Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane further includes:

[0030] A three-dimensional interactive object that is originally blocked on the interactive target plane is selected according to movement information of a depth dimension of the interactive target plane and the input device, wherein the depth dimension is a dimension corresponding to a user's line of sight.

[0031] Furthermore, mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane further includes:

[0032] The moving distances of the respective dimensions on the interactive target plane are calculated according to the moving distances of the respective dimensions of the input device in the three-dimensional space and the mapping proportional coefficients corresponding to the respective dimensions.

[0033] Furthermore, if the interactive object is a two-dimensional interactive object, the method further includes:

[0034] The moving path visual prompt information of the input device is generated on the interactive target plane according to the moving distances of the respective dimensions on the interactive target plane.

[0035] Furthermore, generating the moving path visual prompt information of the input device on the interactive target plane according to the moving distances of the respective dimensions on the interactive target plane further includes:

[0036] Determining the dominant direction of movement according to the moving distances of each dimension on the interactive target plane;

[0037] The moving path visual prompt is generated according to the moving dominant direction.

[0038] Furthermore, determining the dominant direction of movement according to the moving distances of each dimension on the interactive target plane further includes:

[0039] Calculate the coordinates of the current moving target point in each dimension on the interactive target plane and the coordinates of the historical moving target points in each dimension to calculate the moving amplitude of each dimension, and take the direction corresponding to the dimension with the largest moving amplitude as the moving advantage direction;

[0040] Generating the moving path visual prompt according to the moving dominant direction further comprises:

[0041] The coordinate value of the current moving target point in the dimension corresponding to the moving dominant direction and the coordinate value of the historical moving target point in another dimension are used as the visual prompt position;

[0042] The movement path visual cue is generated at the visual cue location.

[0043] Furthermore, if the environment of the two-dimensional interactive object is regularly arranged or has contextual content, generating the moving path visual prompt at the visual prompt position further includes:

[0044] The movement path visual cue is generated at a specified offset value from the visual cue location.

[0045] Furthermore, the method further comprises:

[0046] taking at least one two-dimensional interactive object in the interactive target plane whose distance from the coordinates of the moving path visual prompt is less than a threshold as a candidate object;

[0047] The best selected object is determined from the candidate objects according to the context or history information of the two-dimensional interactive object.

[0048] Furthermore, when the interactive object is a two-dimensional interactive object or a three-dimensional interactive object in the plane operation mode, or when the interactive object is a two-dimensional interactive object in the suspended operation mode, the method further includes:

[0049] An auxiliary input surface is generated on any plane content in the three-dimensional space, so that the user can operate the input device to input on the actual physical plane corresponding to the auxiliary input surface.

[0050] On the other hand, the embodiments of this specification also provide a fine selection device in three-dimensional space, the device comprising:

[0051] An interactive object determination unit, used to determine the user's current interactive object in the three-dimensional space;

[0052] An interactive target plane generating unit, configured to generate an interactive target plane according to a position of a current interactive object of the user in the three-dimensional space;

[0053] An input device information mapping calculation unit, used for mapping movement information of the user's input device in the three-dimensional space to the interactive target plane;

[0054] A selection processing unit is used to select the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane.

[0055] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0056] Finally, an embodiment of this specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0057] In order to solve the problem that it is difficult to select objects in three-dimensional space through rays, etc., and selection through additional devices such as touchpads will increase the convenience of use, the embodiment of the present specification generates an interactive target plane at the position of the interactive object, and then maps the movement information of the user's input device in the three-dimensional space to the interactive target plane, and then selects the interactive object according to the mapping position information of the input device in the interactive target plane and the position information of the interactive object on the interactive target plane. Because the difficulty of selecting objects on the interactive target plane is much lower than the difficulty of selecting objects in three-dimensional space, the difficulty of selecting interactive objects is reduced, and there is no need to add additional equipment, which improves the convenience of use and ultimately improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 It is a flowchart of a fine selection method in three-dimensional space according to an embodiment of this specification;

[0060] Figure 2The figure shows a flow chart of generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space if the interactive object is a two-dimensional object in the embodiment of the specification;

[0061] Figure 3 It is a schematic diagram of a process of generating visual prompt information of a moving path of the input device on the interactive target plane according to the moving distance of each dimension of the interactive device on the interactive target plane in an embodiment of the present specification;

[0062] Figure 4 It is a schematic diagram of a visual prompt of a moving path of an input device described in an embodiment of this specification;

[0063] Figure 5 The figure is a schematic diagram of a process of determining the best selected object according to the visual prompt determined by the moving path and the interaction history information in the embodiment of this specification;

[0064] Figure 6 Shown is a schematic diagram of determining the position of a visual cue in an embodiment of this specification;

[0065] Figure 7 It is a schematic diagram of an embodiment of the plane operation mode in the embodiments of this specification;

[0066] Figure 8 The figure shows a flow chart of generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space if it is a three-dimensional interactive object in the embodiment of this specification;

[0067] Fig. 9 Shown is a schematic diagram of a near clipping plane and a far clipping plane in an embodiment of this specification;

[0068] Fig.10 It is a schematic diagram showing an example of projecting three-dimensional content onto an auxiliary target plane in an embodiment of this specification;

[0069] Fig.11 It is a schematic diagram of a process of mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane if the interaction mode is a plane operation mode in an embodiment of this specification;

[0070] Fig.12 It is a schematic diagram of a process of mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane if the interaction mode is the suspended operation mode in the embodiment of this specification;

[0071] Fig.13 It is a schematic diagram of the structure of a fine selection device in three-dimensional space in an embodiment of this specification;

[0072] Fig.14 The figure is a schematic diagram of the structure of a computer device in an embodiment of the present specification.

[0073]

Description of the accompanying drawings

[0074] 1301, interactive object determination unit;

[0075] 1302, interactive target plane generating unit;

[0076] 1303, input device information mapping calculation unit;

[0077] 1304. Select a processing unit;

[0078] 1402. Computer equipment;

[0079] 1404. Processing equipment;

[0080] 1406. Storage resources;

[0081] 1408, driving mechanism;

[0082] 1410, input / output module;

[0083] 1412. Input device;

[0084] 1414. Output device;

[0085] 1416. Presentation equipment;

[0086] 1418. Graphical user interface;

[0087] 1420, network interface;

[0088] 1422, communication link;

[0089] 1424. Communication bus. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0091] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0092] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0093] In order to solve the problems existing in the prior art, the embodiments of this specification provide a fine selection method in three-dimensional space, which maps the 6dof movement of the input source into a 2D movement on a target plane or a plane in a target space, and performs anti-shake processing on the movement according to scene- or context-related rules, and determines the selected object or the range of the selected object based on the processed results. Figure 1 The flowchart of a method for fine selection in three-dimensional space according to an embodiment of the present specification is shown. This figure describes the process of fine selection in three-dimensional space, but it may include more or fewer operation steps based on conventional or non-creative work. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or device product is executed in practice, it can be executed in the order or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 1 As shown, the method may include:

[0094] Step 101: Determine the user's current interaction object in the three-dimensional space;

[0095] Step 102: Generate an interactive target plane according to the position of the user's current interactive object in the three-dimensional space;

[0096] Step 103: Mapping movement information of the user's input device in the three-dimensional space to the interactive target plane;

[0097] Step 104: Select the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane.

[0098] In order to solve the problem that it is difficult to select objects in three-dimensional space through rays, etc., and selection through additional devices such as touchpads will increase the convenience of use, the embodiment of the present specification generates an interactive target plane at the position of the interactive object, and then maps the movement information of the user's input device in the three-dimensional space to the interactive target plane, and then selects the interactive object according to the mapping position information of the input device in the interactive target plane and the position information of the interactive object on the interactive target plane. Because the difficulty of selecting objects on the interactive target plane is much lower than the difficulty of selecting objects in three-dimensional space, the difficulty of selecting interactive objects is reduced, and there is no need to add additional equipment, which improves the convenience of use and ultimately improves the user experience.

[0099] In the embodiments of this specification, the user's input device may be the user's hand or a handle, and this embodiment of this specification does not limit this.

[0100] In the embodiments of the present specification, the user's operation can also be identified, and specific interaction instructions, such as a selection start instruction or a selection end instruction, can be generated based on the user's operation or interaction and a preset threshold. Therefore, in actual implementation, it is continuously monitored whether the user issues a selection start instruction or a selection end instruction. For example, if the user issues a selection start instruction, the selection step of the interactive object is executed; if the user issues a selection end instruction, the selection step of the interactive object is ended.

[0101] In the embodiments of this specification, the currently active interactive object can be determined based on the preset rules of the system and the user's interactive information. Interactive objects include two-dimensional interactive objects and three-dimensional interactive objects. It can be determined whether the user's current interactive object is two-dimensional or three-dimensional based on the input information of the XR device and the properties of the interactive objects in the scene. The embodiments of this specification do not limit the method for determining the active interactive object. One possible implementation method is, for example, when the user moves the handle, and the handle points to a two-dimensional UI and presses the Trigger key of the handle, the two-dimensional UI currently being pointed to is regarded as the "active" interactive object.

[0102] The embodiments of this specification also define the interaction modes into two categories, Mode 1 is the In Air mode, and Mode 2 is the Surface mode. Determine the user's input / interaction mode: monitor the 6Dof or 3Dof information of the input device (hand or handle), and determine whether the user is currently in In Air mode 1 or Surface mode 2 based on a preset threshold or AI or environmental recognition (the embodiments of this specification do not limit this determination method). An example of a determination method is: when the user's palm remains in the same plane for more than a threshold (such as 1 second), it is considered that the user's hand is placed on a physical plane. At this time, relying on this plane for interaction can not only obtain continuous tactile feedback, but also enable the arm to obtain support from this plane to reduce the user's hand fatigue. That is, this is Surface mode 2.

[0103] Record the position P0 of the input device (hand or handle) in space when the user starts the selection operation. Record the current interactive object T0.

[0104] If the interaction mode is the suspended operation mode, the input device has operation information in three dimensions, x, y and z. In order to ensure stable and accurate interaction with the candidate interaction target, the present invention will discard the operation information of one dimension of the input device according to the rules, and only retain the operation information of two dimensions. For example, when a user interacts with a vertically placed two-dimensional web page object, if the user uses a hand or a handle to turn the web page, the present invention will discard the operation information of the dimension (z) corresponding to the user's line of sight, that is, discard the operation information of the hand or handle in the z dimension corresponding to the depth of field direction, and only retain the operation information of the x and y dimensions.

[0105] If the interaction mode is a plane mode, at this time, only the operation information of the input device in two dimensions on the operation plane is obtained. How to obtain the information in two dimensions needs to be determined according to the direction of the operation plane of the user's hand or the handle. For example, if the user's hand operates on a horizontal plane, only the operation information of the hand in the x and z dimensions is obtained. If the user's hand operates on a vertical plane, only the operation information of the hand in the x and y dimensions is obtained. If the user's hand operates on a plane inclined 30° from the horizontal plane, the 6Dof or 3Dof information of the hand in the world space is obtained and converted to obtain the x and y dimensional information of the hand on the plane. It should be noted that the XR device can obtain the actual 6Dof or 3Dof information of the user input in real time, and the system also has the function of obtaining the corresponding two-dimensional operation information according to the direction of the user's operation plane.

[0106] In the embodiments of the present specification, the process of selecting a two-dimensional interactive object is similar regardless of whether it is in a suspended operation mode or a plane operation mode, because in the plane interaction mode, the hand or handle operates in two dimensions (for example, the hand or handle is placed on the desktop for operation), and when selecting a two-dimensional interactive object in the suspended operation mode, although there is no restriction on the operation of the hand or handle in the third dimension, the operation information of this dimension is discarded.

[0107] Therefore, if Figure 2 As shown, if it is a two-dimensional interactive object, generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space further includes:

[0108] Step 201: generating the interactive target plane at the position of the two-dimensional interactive object in the three-dimensional space;

[0109] In this step, if the user is interacting with a two-dimensional interactive object, the interactive target plane is the plane where the two-dimensional interactive object is located. The interactive target plane may be invisible to the user and is only used for system calculations, because a two-dimensional interactive object, such as a button, only has a position relative to its parent set, that is, the panel that hosts it, and does not have a global three-dimensional position. Therefore, when you want to calculate which specific two-dimensional object the input device has selected, you still need to first generate a virtual plane with the same position and rotation as the two-dimensional interactive object (such as a button) that hosts it.

[0110] Mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane further includes:

[0111] Step 202: Mapping the two-dimensional movement information of the input device in the three-dimensional space corresponding to the interactive target plane to the interactive target plane;

[0112] In this step, the changes in the three dimensions x_input, y_input, and z_input of the 3Dof information of the input device (the specific mapping method is to convert the global movement operation of the input device into local coordinates determined according to the operation plane) are converted into x_target and y_target on the interactive target plane.

[0113] Specifically, a coordinate conversion can be performed between the three-dimensional coordinate system of the input device in the three-dimensional space and the two-dimensional coordinate system of the interactive target plane, for example, the coordinates of the input device in the three-dimensional coordinate system in the three-dimensional space are converted to the coordinates of the two-dimensional coordinate system of the interactive target plane. However, because the input device is mobile, it is also necessary to convert the distance moved by the input device in the three-dimensional space to the interactive target plane. Therefore, mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane further includes: calculating the movement distance of each dimension on the interactive target plane according to the movement distance of each dimension of the input device in the three-dimensional space and the mapping proportional coefficient corresponding to each dimension.

[0114] In the embodiments of the present specification, the mapping scale coefficient may be set by a staff member, and the corresponding mapping scale coefficients of respective dimensions may be different, thereby facilitating the user to interact with an interactive object with a larger aspect ratio.

[0115] In addition, after mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane, the mapping position information of the input device on the interactive target plane may be de-jittered.

[0116] Specifically, an anti-shake algorithm is used, such as a Kalman filter or a Euro filter, or a combination of several anti-shake algorithms, which is not limited in the embodiments of this specification. The point Surface Position (x_target, y_target) on the two-dimensional plane is anti-shake processed to obtain the Static Target Position recorded as: STP (STP_x, STP_y).

[0117] Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane further includes:

[0118] Step 203: Select the two-dimensional interactive object according to the correspondence between the position corresponding to the mapping position information of the input device on the interactive target plane and the position information of the two-dimensional interactive object on the interactive target plane.

[0119] In this step, for example, if the position corresponding to the mapping position information of the input device on the interactive target plane is within the range of the position of the two-dimensional interactive object on the interactive target plane, the two-dimensional interactive object is selected. The embodiment of this specification does not limit the detailed processing method of the two-dimensional object selection process.

[0120] According to an embodiment of the present specification, in order to improve user experience, if the interactive object is a two-dimensional interactive object, the method further includes:

[0121] The moving path visual prompt information of the input device is generated on the interactive target plane according to the moving distances of the respective dimensions on the interactive target plane.

[0122] Preferably, if Figure 3 As shown, generating the moving path visual prompt information of the input device on the interactive target plane according to the moving distance of each dimension on the interactive target plane further includes:

[0123] Step 301: determining a dominant moving direction according to moving distances in each dimension on the interactive target plane;

[0124] In this step, the dominant direction of movement refers to the main moving direction of the moving track of the user's input device on the interactive target plane, for example, mainly moving in the direction of the x-axis, or mainly moving in the direction of the y-axis. Specifically, the coordinates of the current moving target point in each dimension on the interactive target plane and the coordinates of the historical moving target points in each dimension can be calculated to calculate the movement amplitude of each dimension, and the direction corresponding to the dimension with the largest movement amplitude is taken as the dominant direction of movement.

[0125] For example, if in the interactive target plane, the x-axis coordinate of the current moving target point is x_target_now, and the y-axis coordinate is y_target_now (or the x-axis coordinate of the current moving target point after debounce is STP_x_now, and the y-axis coordinate is STP_y_now), the x-axis coordinate of the previous n-th historical moving target point is x_target_previous, and the y-axis coordinate is y_target_previous (or the x-axis coordinate of the previous n-th historical moving target point after debounce is STP_x_previous, and the y-axis coordinate is STP_y_previous), where n is a positive integer >= 1. Calculate the movement amplitude of the x-axis and y-axis:

[0126] ΔVertical = |x_target_now – x_target_previous|, or ΔVertical_STP = |STP_x_now –STP_x_previous|;

[0127] ΔHorizontal = |y_target_now – y_target_previous|, or ΔHorizontal_STP = |STP_y_now – STP_y_previous|.

[0128] Among them, ΔVertical is the movement amplitude of the x-axis, ΔVertical_STP is the movement amplitude of the x-axis calculated after debouncing, ΔHorizontal is the movement amplitude of the y-axis, and ΔHorizontal_STP is the movement amplitude of the y-axis calculated after debouncing.

[0129] Then calculate the moving advantage direction:

[0130] ;or,

[0131]

[0132] It should be noted that the step of calculating the dominant direction of movement using the coordinate value after de-jittering is optional, and the embodiments of this specification do not limit whether to perform de-jittering processing.

[0133] Step 302: Generate the moving path visual prompt according to the moving dominant direction.

[0134] In this step, the position represented by the coordinate value of the current moving target point in the dimension corresponding to the moving dominant direction and the coordinate value of the historical moving target point in another dimension is used as a visual prompt position;

[0135] For example, if the dominant direction AD = Vertical, the visual cue moves linearly in the x-axis direction, and the visual cue position is:

[0136] Position_cue = (x_target_now, y_target_previous)

[0137] If the dominant direction AD = Horizontal, the visual cue moves linearly in the y-axis direction, and the visual cue position is:

[0138] Position_cue = (x_target_previous, y_target_now)

[0139] Finally, the moving path visual prompt is generated at the visual prompt position.

[0140] For example, Figure 4 The following is a schematic diagram of the visual prompt of the moving path. The visual prompt of the moving path will not appear as Figure 4 The diagonal movement path is shown by the dashed icon in the figure.

[0141] Optionally, if the interactive object environment is a regularly arranged or contextual content (the context here includes the context of the text, the context of the environment information, the context of the target object attributes, and all other related content), the visual cue may appear at a position with a specified offset value offset of the object to be selected, rather than exactly at the position of the Position_cue obtained above. This can avoid the visual cue from blocking the object to be selected, such as appearing in the gap between text or the space between folder icons. For example, the visual cue appears at the end of the text closest to the Position_cue.

[0142] Optionally, to facilitate user selection, such as Figure 5 As shown, the method also includes:

[0143] Step 501: taking at least one two-dimensional interactive object in the interactive target plane whose distance from the coordinates of the moving path visual prompt is less than a threshold as a candidate object;

[0144] Step 502: Determine the best selected object from the candidate objects according to the context or history information of the two-dimensional interactive object.

[0145] Specifically, the best selection object is determined based on the interactive context, environment, etc. and a predetermined threshold or AI method. The embodiments of this specification do not limit the specific threshold or AI method.

[0146] A feasible solution is: Figure 6 , when the position mapped on the target plane appears within the range shown by the circle in the figure. Objects on the plane whose distance to the position is less than the threshold d (e.g., 2cm): letters (Text) "o", "m", "y" and symbol "(" are all candidates. However, according to the interaction context "the user started the selection operation at the letter E" and the user's reading habits from left to right and from top to bottom, among the four options above, it is determined that the user is most likely to select the letter "y".

[0147] According to the position of the "best selection object" in the previous step, context information, etc., the position of the visual prompt cue_pos (x_cue, y_cue) of the user's selection position is determined. This is the position of the final user prompt UI. A specific solution that can be implemented is shown in Figure 6 The currently selected object is the letter "y" on the two-dimensional target plane. In order not to block the user's line of sight and the next interaction, the embodiment of this specification determines to shift the visual prompt to the right to the illustrated position according to the environment around the candidate target. This achieves the effect of not blocking the user's line of sight.

[0148] It should be noted that the method of the embodiments of this specification can not only select interactive objects, but also define the selection range.

[0149] Exemplarily, if the system detects that the user has issued a selection start instruction, the position P0 (x_input_0, y_input_0, z_input_0) of the input device in the three-dimensional space in the frame of the "selection start instruction", the current interactive object T0: the UI of the text editing box; and the position P_start_target (0,1,1) of the current interactive object T0 are recorded. If it is determined that the current interactive object T0 is two-dimensional (for example, T0 is a text editing box UI). Map the input device on the interactive target plane where T0 is located to obtain P0_local (x_p0_local, y_p0_local). In each subsequent frame, until the selection end instruction is received, it is necessary to determine whether the current mode is In Air mode or Surface mode. If it is Surface mode, the selection method in Surface mode is executed; if it is In Air mode, the selection method in In Air mode is executed. At the same time, the mapping position information of the input device on the interactive target plane from the selection start instruction to the selection end instruction is recorded. The optional targets (such as text content in a web page interactive object) located on the path of the mapped location information constitute the selected target set (i.e., the defined selection range). The targets in this set are the targets selected by the user. The system should provide some visual prompts for these targets, such as changing the color or the background color of the selected area, etc. (this specification does not impose any specific restrictions). (For example, Figure 7 as shown).

[0150] It should be noted that the processing flow of defining the selection range can refer to the process of selecting an interactive object, which will not be repeated here.

[0151] Optionally, the currently selected interactive object is determined according to the user's operation (input), or the start or end of the selection area is determined. This embodiment of the specification does not limit this operation, such as pressing a button of the handle controller to start, lifting the button to end, etc., thereby determining the selected interactive object.

[0152] According to one embodiment of this specification, Figure 8 As shown, if it is a three-dimensional interactive object (such as a three-dimensional interactive object in the in air mode or a three-dimensional interactive object in the surface mode), generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space further includes:

[0153] Step 801: generating the interactive target plane parallel to the near clipping plane at the position of the three-dimensional interactive object in the three-dimensional space;

[0154] like Fig. 9As shown, there are two restrictions on the content rendered by the camera, namely the near clipping plane and the far clipping plane. The near clipping plane and the far clipping plane are parallel to the XY plane of the camera, and the two are separated by a certain distance along the center line. The content between the near clipping plane and the far clipping plane is visible to the camera, and any object closer to the camera than the near clipping plane and any object farther from the camera than the far clipping plane will not be rendered. In addition, any object outside the diverging line at the edge of the image is invisible to the camera.

[0155] The embodiment of the present specification generates an interactive target plane parallel to the near clipping plane or the far clipping plane at the position of the three-dimensional interactive object in the three-dimensional space. The interactive target plane is located between the near clipping plane and the far clipping plane, so the content on the interactive target plane is visible to the user.

[0156] The method further comprises:

[0157] Step 802: Projecting all three-dimensional interactive objects in the field of view onto the interactive target plane to obtain position information of all three-dimensional interactive objects in the field of view on the interactive target plane;

[0158] like Fig.10 As shown, the left side shows all the three-dimensional interactive objects in the field of view, and the right side shows all the three-dimensional interactive objects in the field of view projected onto the interactive target plane.

[0159] Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane further includes:

[0160] Step 803: Select the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane.

[0161] Specifically, Fig.11 As shown, if the interaction mode is a plane operation mode, mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane further includes:

[0162] Step 1101: Mapping movement information of the input device in the three-dimensional space to the interactive target plane;

[0163] In the embodiment of the present specification, in the plane operation mode, the movement information of the input device in the three-dimensional space is three-dimensional, so the three-dimensional movement information of the input device in the three-dimensional space is mapped to the interaction target plane.

[0164] Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane further includes:

[0165] Step 1102: Select the three-dimensional interactive object according to the correspondence between the position corresponding to the two-dimensional mapping position information of the input device on the interactive target plane and the position information of the three-dimensional interactive object projected in the interactive target plane.

[0166] In the embodiment of this specification, for example, if the position corresponding to the mapping position information of the input device on the interactive target plane is within the range of the position of the projection of the three-dimensional interactive object on the interactive target plane, the three-dimensional interactive object is selected. Fig.10 When the projection of the 3D interactive object shown on the right side of is partially blocked on the interactive target plane, and the position corresponding to the mapping position information of the input device on the interactive target plane is within the range of the position of the projection of the blocked 3D interactive object on the interactive target plane, the blocked 3D interactive object is selected. If the projection of a 3D interactive object on the interactive target plane is completely blocked, in the plane interaction mode, you need to use auxiliary tools to select the 3D interactive object whose projection is completely blocked.

[0167] like Fig.12 As shown, if the interaction mode is the suspended operation mode, mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane further includes:

[0168] Step 1201: Mapping the three-dimensional movement information of the input device in the three-dimensional space to the interactive target plane;

[0169] In the embodiment of the present specification, in the suspended operation mode, the movement information of the input device in the three-dimensional space is three-dimensional, so the three-dimensional movement information of the input device in the three-dimensional space is mapped to the interactive target plane.

[0170] Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane further includes:

[0171] Step 1202: Select the three-dimensional interactive object that is originally blocked on the interactive target plane according to the movement information of the depth dimension of the input device on the interactive target plane, where the depth dimension is the dimension corresponding to the user's line of sight.

[0172] In this step, the method for selecting the three-dimensional content whose projection is not blocked can refer to step 1102. For the three-dimensional content whose projection is blocked, because the movement information of the input device in the suspended operation mode is three-dimensional, that is, it has a dimension corresponding to the user's line of sight direction, the embodiment of this specification cleverly uses this dimension to select the three-dimensional interactive object with blockage projected on the interactive target plane according to the movement information of the depth dimension of the input device on the interactive target plane. For example, the user operates the input device to place the position corresponding to the mapping position information of the input device on the interactive target plane within the range of the position of the projection of the three-dimensional interactive object with blockage on the interactive target plane, and then moves the input device to the dimension corresponding to the line of sight direction, so that the blocked three-dimensional interactive object can be selected.

[0173] It should be noted that the method of mapping the input device movement information to the interactive target plane in the two-dimensional interactive object selection process can be applied to the three-dimensional interactive object selection process, that is, the moving distance of each dimension on the interactive target plane is calculated according to the moving distance of each dimension of the input device in the three-dimensional space and the mapping proportional coefficient corresponding to each dimension, and the mapped moving distance is de-jittered, etc., which will not be repeated here.

[0174] When selecting a 3D interactive object, you can avoid generating Figure 4 Instead of the visual prompt shown, selection feedback of the object determined to be selected can be set, such as highlighting, sound feedback, etc., which is not limited in the embodiments of this specification.

[0175] According to an embodiment of the present specification, in order to reduce the fatigue of the user during input, in the plane operation mode, the method further includes:

[0176] An auxiliary input surface is generated on any plane content in the three-dimensional space, so that the user can operate the input device to input on the actual physical plane corresponding to the auxiliary input surface.

[0177] In the embodiment of this specification, in the plane operation mode, the input device moves in the plane, and the system obtains and records the two-dimensional movement information of the input device mapped to the target input plane. Therefore, in the above case, the embodiment of this specification generates an auxiliary input surface on any plane content in the three-dimensional space, and the plane for generating the auxiliary input surface is also a physical plane in the real environment, so the user can place the input device on the real physical plane corresponding to the auxiliary input surface for input, thereby reducing the user's input fatigue.

[0178] Based on the same inventive concept, the embodiment of this specification also provides a fine selection device in three-dimensional space, such as Fig.13 As shown, the device comprises:

[0179] The interactive object determining unit 1301 is used to determine the current interactive object of the user in the three-dimensional space;

[0180] An interactive target plane generating unit 1302, configured to generate an interactive target plane according to a position of a current interactive object of the user in the three-dimensional space;

[0181] An input device information mapping calculation unit 1303, configured to map movement information of the user's input device in the three-dimensional space to the interaction target plane;

[0182] The selection processing unit 1304 is used to select the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane.

[0183] The beneficial effects obtained by the above-mentioned device are consistent with the beneficial effects obtained by the above-mentioned method, and will not be described in detail in the embodiments of this specification.

[0184] like Fig.14 The diagram shows the structure of a computer device in an embodiment of the present specification. The apparatus in the present specification may be the computer device in the embodiment, executing the method in the present specification.

[0185] Computer device 1402 may include one or more processing devices 1404, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads.

[0186] The computer device 1402 may also include any storage resources 1406 for storing any kind of information such as code, settings, data, etc. For example, without limitation, the storage resources 1406 may include any one or a combination of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc.

[0187] More generally, any storage resource may use any technology to store information.

[0188] Further, any storage resource may provide volatile or non-volatile retention of information.

[0189] Further, any storage resources may represent fixed or removable components of computer device 1402 .

[0190] In one case, when the processing device 1404 executes the associated instructions stored in any storage resource or combination of storage resources, the computer device 1402 may perform any of the operations of the associated instructions.

[0191] The computer device 1402 also includes one or more drive mechanisms 1408 for interacting with any storage resources, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0192] The computer device 1402 may also include an input / output module 1410 (I / O) for receiving various inputs (via input devices 1412 ) and for providing various outputs (via output devices 1414 ).

[0193] One particular output mechanism may include a presentation device 1416 and an associated graphical user interface (GUI) 1418 .

[0194] In other embodiments, the input / output module 1410 (I / O), the input device 1412 and the output device 1414 may not be included, and the computer only serves as a computer device in the network.

[0195] The computer device 1402 may also include one or more network interfaces 1420 for exchanging data with other devices via one or more communication links 1422. One or more communication buses 1424 couple the components described above together.

[0196] The communication link 1422 may be implemented in any manner, such as through a local area network, a wide area network (eg, the Internet), a point-to-point connection, etc., or any combination thereof.

[0197] The communications link 1422 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0198] The embodiments of the present specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0199] The embodiments of the present specification also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the above method.

[0200] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0201] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0202] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0204] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0206] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0207] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0208] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A fine selection method in three-dimensional space, characterized in that: The method comprises: Determining a current interactive object of a user in a three-dimensional space, wherein the interactive object includes a three-dimensional interactive object; Generate an interactive target plane according to the position of the user's current interactive object in the three-dimensional space; Mapping movement information of the user's input device in the three-dimensional space to the interactive target plane; Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane; If it is a three-dimensional interactive object, generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space further includes: generating the interactive target plane parallel to the near clipping plane at the position of the three-dimensional interactive object in the three-dimensional space; The method further comprises: projecting all three-dimensional interactive objects in the field of view onto the interactive target plane to obtain position information of all three-dimensional interactive objects in the field of view on the interactive target plane; Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane further includes: Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane; The method further includes: determining a current interaction mode of the input device, the interaction mode including a suspended operation mode; If the interaction mode is a suspended operation mode, mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane further includes: Mapping the three-dimensional movement information of the input device in the three-dimensional space to the interactive target plane; Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane further includes: A three-dimensional interactive object that is originally blocked on the interactive target plane is selected according to movement information of a depth dimension of the interactive target plane and the input device, wherein the depth dimension is a dimension corresponding to a user's line of sight.

2. The method according to claim 1, characterized in that The interactive object includes a two-dimensional interactive object.

3. The method according to claim 2, characterized in that If it is a two-dimensional interactive object, generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space further includes: generating the interactive target plane at the position of the two-dimensional interactive object in the three-dimensional space; Mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane further includes: Mapping the two-dimensional movement information of the input device in the three-dimensional space corresponding to the interactive target plane to the interactive target plane; Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane further includes: The two-dimensional interactive object is selected according to the correspondence between the position corresponding to the mapping position information of the input device on the interactive target plane and the position information of the two-dimensional interactive object on the interactive target plane.

4. The method according to claim 1, characterized in that: The interactive mode includes a planar operation mode.

5. The method according to claim 4, characterized in that If the interaction mode is a plane operation mode, mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane further includes: Mapping the two-dimensional movement information of the input device in the three-dimensional space to the interactive target plane; Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane further includes: The three-dimensional interactive object is selected according to the correspondence between the position corresponding to the two-dimensional mapping position information of the input device on the interactive target plane and the position information of the three-dimensional interactive object projected in the interactive target plane.

6. The method according to claim 2, characterized in that Mapping the movement information of the user's input device in the three-dimensional space to the interactive target plane further includes: The moving distances of the respective dimensions on the interactive target plane are calculated according to the moving distances of the respective dimensions of the input device in the three-dimensional space and the mapping proportional coefficients corresponding to the respective dimensions.

7. The method according to claim 6, characterized in that If the interactive object is a two-dimensional interactive object, the method further includes: The moving path visual prompt information of the input device is generated on the interactive target plane according to the moving distances of the respective dimensions on the interactive target plane.

8. The method according to claim 7, characterized in that Generating the moving path visual prompt information of the input device on the interactive target plane according to the moving distances of the respective dimensions on the interactive target plane further comprises: Determining the dominant direction of movement according to the moving distances of each dimension on the interactive target plane; The moving path visual prompt is generated according to the moving dominant direction.

9. The method according to claim 8, characterized in that Determining the dominant direction of movement according to the moving distances of each dimension on the interactive target plane further includes: Calculate the coordinates of the current moving target point in each dimension on the interactive target plane and the coordinates of the historical moving target points in each dimension to calculate the moving amplitude of each dimension, and take the direction corresponding to the dimension with the largest moving amplitude as the moving advantage direction; Generating the moving path visual prompt according to the moving dominant direction further comprises: The coordinate value of the current moving target point in the dimension corresponding to the moving dominant direction and the coordinate value of the historical moving target point in another dimension are used as the visual prompt position; The movement path visual cue is generated at the visual cue location.

10. The method according to claim 9, characterized in that If the environment of the two-dimensional interactive object is regularly arranged or has contextual content, generating the moving path visual prompt at the visual prompt position further includes: The movement path visual cue is generated at a specified offset value from the visual cue location.

11. The method according to claim 10, characterized in that The method further comprises: taking at least one two-dimensional interactive object in the interactive target plane whose distance from the coordinates of the moving path visual prompt is less than a threshold as a candidate object; The best selected object is determined from the candidate objects according to the context or history information of the two-dimensional interactive object.

12. The method according to claim 4, characterized in that When the interactive object is a two-dimensional interactive object or a three-dimensional interactive object in the plane operation mode, or when the interactive object is a two-dimensional interactive object in the suspended operation mode, the method further includes: An auxiliary input surface is generated on any plane content in the three-dimensional space, so that the user can operate the input device to input on the actual physical plane corresponding to the auxiliary input surface.

13. A fine selection device in three-dimensional space, characterized in that: The device comprises: An interactive object determination unit, used to determine a user's current interactive object in a three-dimensional space, wherein the interactive object includes a three-dimensional interactive object; An interactive target plane generating unit, configured to generate an interactive target plane according to a position of a current interactive object of the user in the three-dimensional space; An input device information mapping calculation unit, used for mapping movement information of the user's input device in the three-dimensional space to the interactive target plane; A selection processing unit, configured to select the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane; If it is a three-dimensional interactive object, generating an interactive target plane according to the position of the user's current interactive object in the three-dimensional space further includes: generating the interactive target plane parallel to the near clipping plane at the position of the three-dimensional interactive object in the three-dimensional space; The device is further used to: project all three-dimensional interactive objects in the field of view onto the interactive target plane to obtain position information of all three-dimensional interactive objects in the field of view on the interactive target plane; Selecting the interactive object according to the mapping position information of the input device on the interactive target plane and the position information of the interactive object on the interactive target plane further includes: Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane; The apparatus is further used to: determine a current interaction mode of the input device, the interaction mode comprising a suspended operation mode; If the interaction mode is a suspended operation mode, mapping the movement information of the user's input device in the three-dimensional space to the interaction target plane further includes: Mapping the three-dimensional movement information of the input device in the three-dimensional space to the interactive target plane; Selecting the three-dimensional interactive object according to the mapping position information and the position information of the three-dimensional interactive object projected on the interactive target plane further includes: A three-dimensional interactive object that is originally blocked on the interactive target plane is selected according to movement information of a depth dimension of the interactive target plane and the input device, wherein the depth dimension is a dimension corresponding to a user's line of sight.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images