Multi-selection method and device in three-dimensional space

Abstract

The invention discloses a multi-selection method and device in a three-dimensional space, and relates to the technical field of expanded reality, and the method comprises the steps: displaying a prompt range to a user when the first hand of the user holds a first object, the second hand of the user holds a second object and the distance between the first hand and the second hand is smaller than a first threshold value in an expanded reality scene; the prompt range represents a triggering range for triggering a multi-selection operation in an expanded reality scene; the first object and the second object represent elements in the expanded reality scene; and when any hand in the prompt range releases the held object, controlling the other hand to hold the first object and the second object. According to the method, the steps of multi-choice operation in the expanded reality scene can be reduced, the multi-choice operation can be quickly and conveniently realized, and the smoothness of the expanded reality operation is improved.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a multiple selection method and device in three-dimensional space. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] Multi-select operation is an important function in the extended reality operating system. However, in the extended reality environment, objects are often not limited to the 2D UI interface, and in most cases, especially when operating 3D objects, due to the lack of keyboard assistance, the commonly used shortcut key triggering or key multi-select state method is not applicable to the extended reality operating environment. In extended reality, to implement multi-select operations, it is usually necessary to use the UI interface and complex gestures. For example, it is necessary to open the interface to complete a series of selections, terminate other ongoing or attempted operations, and resume other normal operations after completing the multi-select operation. This process is cumbersome and the operation fluency is poor. In the traditional extended reality environment, this multi-step multi-selection method greatly increases the complexity of the operation and reduces the fluency of the operation. Summary of the invention

[0004] The embodiment of the present invention provides a multi-selection method in a three-dimensional space, which is used to reduce the steps of multi-selection operation in augmented reality, quickly and conveniently implement multi-selection operation, and improve the fluency of augmented reality operation. The method includes:

[0005] When a user's first hand holds a first object and a second hand holds a second object in an augmented reality scene, and a distance between the first hand and the second hand is less than a first threshold, a prompt range is displayed to the user; the prompt range indicates a trigger range for triggering a multi-select operation in the augmented reality scene;

[0006] When any hand within the prompt range releases the object held, the other hand is controlled to hold the first object and the second object; any one hand is the first hand or the second hand, when any one hand is the first hand, the other hand is the second hand, when any one hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in an augmented reality scene.

[0007] In the embodiment of the present invention, each selection of an object can be used as a condition for starting a multi-selection operation. With the simple cooperation of both hands, the basic operations of selecting, holding, moving, and releasing objects can achieve multi-selection without the need for additional precise operations. This greatly simplifies the steps of multi-selection operations in the augmented reality scene, improves the smoothness of augmented reality operations, and also improves the ease of use and versatility of multi-selection operations in augmented reality. During the multi-selection process, too much UI will not be required to occupy too much space in the field of vision, and will not affect the current user interface. This simple design and operation can allow users to put down their burdens and operate easily, optimizing the user's operating experience.

[0008] In one embodiment, the number of objects held by any one hand may be one or more, and the number of objects held by the other hand may be 0, one or more.

[0009] In one embodiment, any one hand holds the object in a first designated posture; the first designated posture includes a two-finger pinch.

[0010] In one embodiment, when any hand within the prompt range releases the object it is holding, the other hand is controlled to hold the first object and the second object, including: when within the prompt range and the distance between the first hand and the second hand is less than a second threshold, when any hand releases the object it is holding, the other hand is controlled to hold the first object and the second object; the second threshold is less than the first threshold.

[0011] In one embodiment, the multiple objects held by any one hand have different data types and different interactive expressions;

[0012] When it is detected that any hand enters a multiple selection state and is in a second designated posture, each object held by any hand is controlled to be folded and displayed in a uniform size in an augmented reality scene; the second designated posture includes the first designated posture and the palm faces the first direction; the folded display includes front-to-back arrangement display; the multiple selection state indicates holding multiple objects;

[0013] When it is detected that any one hand enters a multiple selection state and exhibits a third designated posture, each object held by the hand is controlled to have a uniform size and to be displayed flatly in the augmented reality scene; the third designated posture includes the first designated posture and the palm faces the second direction; the flat display includes one of a rectangular range, an arc range, and an annular range.

[0014] In one embodiment, when it is detected that any one hand holds multiple objects, a variety of multi-select object operations are provided; the multi-select object operations are: operating all the objects held by any one hand as a whole.

[0015] In one embodiment, the multi-select object operation includes one or any combination of the following:

[0016] Group, Ungroup, Copy, Paste, Delete, Close, Minimize, Send To, Rename, Auto Arrange, Auto Arrange, Create Shortcut, Edit, Open with App.

[0017] In one embodiment, when any hand is detected to be performing the second designated posture or the third designated posture: in the augmented reality scene, the designated object selected from the multiple objects displayed by the other hand is rayed: hovering and highlighting, and / or, a dragging operation is performed after hovering and highlighting.

[0018] In one embodiment, when any one hand is in a multi-selection state, the first designated gesture is ended, and the multiple objects held by the any one hand are released.

[0019] In one embodiment, when the operation of selecting multiple objects is completed, the multiple objects held by any one hand are released.

[0020] In one embodiment, the hand posture and operation are monitored by a gesture sensor or a handle; the gesture sensor includes one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyographic signal gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

[0021] In one embodiment, any operation of any hand can be replaced by monitoring data of a hardware device; the monitoring data of the hardware device includes one or any combination of button operation, position parameters, and rotation parameters.

[0022] The embodiment of the present invention further provides a multi-selection device in a three-dimensional space, which is used to reduce the steps of multi-selection operation in augmented reality, quickly and conveniently implement multi-selection operation, and improve the fluency of augmented reality operation. The device includes:

[0023] A prompt range display module, used to display a prompt range to the user when the user's first hand holds a first object and the second hand holds a second object in the augmented reality scene, and the distance between the first hand and the second hand is less than a first threshold; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scene;

[0024] A multi-select holding module is used to control the other hand to hold a first object and a second object after any hand within a prompt range releases the object held; the any hand is the first hand or the second hand, when the any hand is the first hand, the other hand is the second hand, when the any hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in an extended reality scene.

[0025] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multiple selection method in three-dimensional space when executing the computer program.

[0026] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multiple selection method in the three-dimensional space is implemented.

[0027] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the multiple selection method in the three-dimensional space is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0029] Figure 1 Schematic diagram of a flow chart of a multiple selection method in a three-dimensional space according to an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 1 ;

[0031] Figure 3 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 2 ;

[0032] Figure 4 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 3 ;

[0033] Figure 5 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 1 ;

[0034] Figure 6 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 2 ;

[0035] Figure 7 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 3 ;

[0036] Figure 8 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 4 ;

[0037] Fig. 9 is a schematic diagram of a multiple selection device in a three-dimensional space according to an embodiment of the present invention;

[0038] Fig.10 It is a specific example diagram of a multiple selection device in a three-dimensional space according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0040] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order.

[0041] First, the technical terms involved in the present invention are explained.

[0042] Pinch: in the embodiment of the present invention, refers to the state of selecting a target by hand or a handle, which is generally triggered by gestures: pinching the index finger and thumb together, closing the five fingers together, or pressing the trigger key (trigger key) on the handle, or pressing the grip key (the trigger key is optional), or other gestures or button triggering methods for selecting the target may also be considered.

[0043] Hand: Hands here refer to controllers such as gestures and handles, because these controllers often have similar functions as input sources in augmented reality. In some special cases, there is only one input source, such as single-handed gesture recognition, single handle, mobile phone, separate controller, head display, etc., which are also considered as one hand.

[0044] Multi-selection has always been an important function in the extended reality operating system. However, in the extended reality environment, objects are often not limited to the 2D UI interface, and in most cases, especially when operating 3D objects, coupled with the lack of keyboard assistance, the commonly used shortcut keys to trigger the multi-select state are not suitable for the extended reality operating environment. In extended reality, to achieve multi-selection, it is usually achieved through the UI interface and complex gestures. It is necessary to open the interface to complete a series of selections, terminate other operations that were previously in progress or attempted, and complete the multi-selection operation before resuming normal operation. This process is cumbersome and the operation fluency is poor. In the traditional extended reality environment, this multi-step multi-selection method greatly increases the complexity of the operation and reduces the fluency of the operation.

[0045] Two multiple selection operation solutions in the prior art are introduced below.

[0046] Existing solution one: providing a graphical user interface through a head display device, responding to a player's gaze operation and first gesture change information, determining multiple objects to be selected; determining the depth area where each object to be selected is located and the mapped depth distance corresponding to each object to be selected, the mapped depth distance being used to indicate the distance between the depth area where the object to be selected is located and the adjacent lower depth area; responding to the player's second gesture change information, determining a target selected object from the multiple objects to be selected based on the depth area where each object to be selected is located and the mapped depth distance corresponding to each object to be selected, and displaying the target selected object in a preset display mode.

[0047] In order to facilitate the selection of targets at different depths, the existing solution 1 combines hand-eye interaction to achieve selection by further selecting objects from the selected multiple candidates. The shortcomings of the existing solution 1 are obvious: First, the solution 1 has more steps, because it is necessary to pre-select multiple candidates, and then further select in depth through gestures. In this way, it is actually to select once among many candidates, and then select again. Even when there are multiple candidates at the same depth, it is necessary to move horizontally and select again. The object can be selected through at least two or even three selections. Due to the large number of steps, different gestures are introduced, which not only increases the operation time, but also increases the learning cost and operation cost. Second, the existing solution 1 requires UI assistance to complete the multi-selection operation, and the UI occupies the main position of the field of vision. During the operation of solution 1, the user needs to focus on the multi-selection operation, which is not convenient for other operations, which is also a break for the overall user experience. Third, due to the above two reasons, the user needs extra time to make multiple selections, and needs to keep his hands in the air continuously, and cannot perform other operations in parallel. The overall time to complete the task involving multi-selection operations will be extended, which will not only increase the user's fatigue, but also increase the power consumption of the device and reduce the device usage time.

[0048] Existing solution two: generating a selection area in three-dimensional space and receiving user input of a selection path to define a selection path, defining a selection volume based on the selection area and the selection path, and selecting entities in the 3D space based on the selection volume, receiving user input of a changed selection area while defining the selection path, and the size or shape of the selection area can change in response thereto.

[0049] The existing solution 2 combines interactive elements such as path selection, volume selection, and interface, and is good at selecting a large number of 3D objects. However, it has the following disadvantages: First, when facing some objects that are superimposed or closely attached, the existing solution 2 can be selected through trajectories and later range size adjustment, but in order to accurately select objects, it takes more time to draw accurate trajectories and carefully adjust the size of the selection area so that the objects to be selected are within the range and the objects that do not need to be selected are outside the range. This will take a lot of time and the user may not be comfortable. The existing solution 2 has large limitations in usage scenarios. Second, the operation of the existing solution 2 is relatively complex. Whether it is setting the selection area at the beginning, drawing the trajectory, or adjusting the range multiple times through the UI interface, these greatly increase the learning cost of the operation for the user, and because there are many steps and it takes a lot of time, it is not conducive to user experience. Third, the rendering performance requirements are high. Real-time display of the selection volume and path, as well as tracking the controller or gesture input by the user, all require high computing resources. Especially when rendering complex scenes, this requirement may burden the system performance and affect the response speed and fluency of the device. Fourth, the visual occlusion problem of multiple object selection. In complex 3D scenes, if multiple objects overlap or are close to each other, the selection volume may block the user's line of sight, making it difficult for them to clearly see all candidate objects. In this case, the user may need to repeatedly adjust the selection volume, affecting the efficiency of the operation. Fifth, the dependency on gestures and controllers. The system relies on handheld controllers or similar devices for path definition and volume generation, which is not friendly to users who do not have such devices or users who only use gesture recognition. When further expanded to gesture recognition, high-precision sensors are required to achieve similar operations.

[0050] In general, the existing multiple-selection solutions have many steps, high operational complexity, and the UI occupies the field of vision. Users need to focus on the multiple-selection operation, which breaks the operational experience. In some solutions, it is also difficult to make accurate selections. Problems such as visual occlusion and device dependence also affect the user experience.

[0051] The embodiments of the present invention aim to reduce learning costs and operation time and improve the fluency and comfort of augmented reality operations through a natural and smooth multi-select interaction method.

[0052] Figure 1 FIG. 1 is a flow chart of a multiple selection method in a three-dimensional space according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0053] Step 101: When a user's first hand holds a first object and a second hand holds a second object in an augmented reality scene, and a distance between the first hand and the second hand is less than a first threshold, a prompt range is displayed to the user; the prompt range indicates a trigger range for triggering a multi-select operation in the augmented reality scene;

[0054] Step 102: when any hand within the prompt range releases the object held, control the other hand to hold the first object and the second object; the any hand is the first hand or the second hand, when the any hand is the first hand, the other hand is the second hand, when the any hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in an augmented reality scene.

[0055] The multiple selection method in three-dimensional space according to the embodiment of the present invention is explained in detail below.

[0056] During implementation, the user is placed in an augmented reality scene, and the hand posture and operation are detected by a gesture sensor or a handle; the gesture sensor includes but is not limited to one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyography gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

[0057] For example, the user wears a head-mounted display and the head-mounted display camera monitors the posture of the hands, or the user is in a motion capture scene and the camera monitors the posture of the hands. The user's hands can be bare, or they can wear hand controllers such as handles, or sensors such as gloves and wristbands to achieve multiple selections.

[0058] In one embodiment, the hand (the first hand or the second hand) holds the object in a first specified posture.

[0059] In the embodiment of the present invention, the user has two hands, wherein the first hand is not limited to the left hand or the right hand, and the second hand is not limited to the left hand or the right hand. When the first hand is the left hand, the second hand is the right hand, and when the first hand is the right hand, the second hand is the left hand.

[0060] In an embodiment, the object is a first object or a second object, and the object represents an element in an augmented reality scene, for example, any holdable element in the augmented reality scene, such as a ball, a file, and the like.

[0061] In a preferred embodiment, the first designated gesture comprises a two-finger pinch.

[0062] When the user wants to select an element in the extended reality scene, he or she can do so by pinching two fingers together.

[0063] In order to improve the accuracy and usability of the multi-selection operation, a second threshold can be further set according to different products and user operation habits. When any hand in the prompt range releases the held object, the other hand is controlled to hold the first object and the second object, including:

[0064] When within the prompt range, that is, the distance between the first hand and the second hand is less than a second threshold, after either hand releases the object held, the other hand is controlled to hold the first object and the second object; the second threshold is less than the first threshold.

[0065] That is, within the prompt range, multiple selections can be achieved only when the two hands are brought closer together to a distance smaller than the second threshold.

[0066] In an embodiment, the first hand holds one or more objects, and the second hand also holds one or more objects. As long as the two hands are close to less than a second threshold, if either hand releases the one or more objects it holds, the other hand will simultaneously hold all the objects previously held by the two hands.

[0067] That is, in step 102, within the prompt range: after the first hand releases the first object held, the second hand is controlled to hold the first object and the second object, or after the second hand releases the second object held, the first hand is controlled to hold the first object and the second object. The number of objects held by each hand includes one or more.

[0068] For example, the left hand holds multiple pictures and the right hand also holds multiple video files. When the two hands are close to less than a second threshold, if the left hand releases the multiple pictures, the right hand will hold multiple pictures and multiple video files at the same time; when the two hands are close to less than the second threshold, if the right hand releases the multiple video files, the left hand will hold multiple pictures and multiple video files at the same time.

[0069] In step 102, the number of objects held by any one hand may be one or more, and the number of objects held by the other hand may be one or more.

[0070] The embodiment of the present invention provides multiple trigger components, including: a hand trigger point, a prompt range trigger, and a multi-select event trigger.

[0071] (1) Hand trigger point: represents the position when the hand is pinched. In order to have a better trigger experience, the hand trigger point is set near the point where the thumb and index finger are pinched on the inner side of the hand. Figure 2 , Figure 2 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 1 , hold the object with your thumb and index finger respectively. Figure 2 The objects held are only indicated and not specifically depicted.

[0072] (2) When one hand selects an object and the trigger point of the hand enters the prompt range of the other hand, the prompt of the multiple selection range will be triggered, so that the user can better see when the multiple selection state will be triggered. Figure 3 , Figure 3Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 2 ,When certain conditions are met, for example, when both hands hold one or more objects, when the hand trigger point of one hand enters the prompt range trigger range of the other hand, the prompt of the multi-selection range will be triggered.

[0073] (3) Furthermore, when one hand selects an object and the trigger point of the hand enters the multi-select event trigger range of the other hand, if the hand selecting the object releases, the multi-selection will be triggered. Figure 4 , Figure 4 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 3 ,When the hand trigger point of the hand holding the object enters the range of the multi-select event trigger of the other hand, the hand holding the object releases the holding object, and the multi-selection will be triggered and the other hand will hold two objects at the same time.

[0074] (4) The prompt range trigger and the multi-select event trigger can also be replaced by the first threshold S and the second threshold s, corresponding to Figure 2 Medium prompt range trigger circle range mark, multi-select event trigger circle range mark. Figure 4 Although both the prompt range trigger and the multi-select event trigger use spherical colliders, other more complex colliders can also be used, such as multiple spherical colliders or colliders of other shapes, or more complex threshold judgments. In this case, simple threshold judgments cannot replace the colliders and need to be handled in combination with specific circumstances.

[0075] When the user operates objects in the augmented reality environment, if he needs to control multiple objects to realize the multi-selection function, when one hand holds object A and the other hand holds object B, the multi-selection can be triggered by bringing the hand holding object A close to the hand holding object B, and the distance between the hand holding object A and the hand holding object B is less than the second threshold s, and then releasing. If object A is released at this time, the hand holding object B enters the multi-selection mode and holds object A and object B. The hand holding object A returns to the initial state and can perform other operations or select more objects to repeat the previous process for multi-selection.

[0076] If one hand holds object A and the other hand does not hold an object, the hand holding object A can also be brought close to the hand not holding the object. If the hand not holding the object is in the Pinch state, the hand holding object A is close to the hand not holding the object and reaches the second threshold s, and object A is released. Then object A will be held by the hand that did not hold the object before, completing the hand-changing action. At this time, if the action of the previous step is repeated, the multi-selection state can also be activated. In the embodiment of the present invention, the Pinch state is realized by the first specified posture.

[0077] That is, in the extended reality scenario, when the user's first hand does not hold any object, the second hand holds the second object, and the distance between the first hand and the second hand is less than a first threshold, the prompt range is displayed to the user; when the second hand releases the object held within the prompt range, the first hand is controlled to hold the second object; in the extended reality scenario, when the user's first hand holds the first object, the second hand does not hold an object, and the distance between the first hand and the second hand is less than a first threshold, the prompt range is displayed to the user; when the first hand releases the object held within the prompt range, the second hand is controlled to hold the first object.

[0078] The embodiment of the present invention provides a novel multi-selection operation mode, which combines the operation of the left and right hands. In the user's operation perspective, when both hands hold objects, the user only needs to put the object held by one hand into the other hand to automatically trigger the multi-selection state. In this way, the hand that triggers the multi-selection state will hold the multi-selected target, while the other hand can continue other operations or select more objects to add to the multi-selection set. Compared with the existing solution, the operation is simple and smooth, and the user experience is better.

[0079] The following describes the contents of the multiple-select display.

[0080] In one embodiment, the multiple objects held by any one hand have different data types and different interactive forms; the data types include but are not limited to text, images, videos, audios, files, structured data, binary data, three-dimensional models, application windows, three-dimensional subspaces, etc., and the interactive forms include but are not limited to 2D forms, 3D forms, a combination of 2D and 3D forms, etc. Each interactive form can be appropriately selected and presented according to the interactive needs in the virtual space of the augmented reality. Those skilled in the art should know that the interactive objects in the virtual space of the augmented reality can be a combination of multiple data types or forms of expression, so they are not listed here one by one.

[0081] When it is detected that any one hand enters a multiple selection state and is in a second specified posture, each object held by any one hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second specified posture includes the first specified posture and the palm faces the first direction; the folded display includes front and back arrangement display; any one hand is the first hand or the second hand; for example, in front of the user's perspective in the augmented reality scene, the objects are displayed in front and back arrangement and staggered to the left and right by a specified distance (reference Figure 8 );

[0082] When it is detected that any one hand enters a multiple selection state and exhibits a third designated posture, each object held by the hand is controlled to have a uniform size and to be displayed flatly in the augmented reality scene; the third designated posture includes the first designated posture and the palm faces the second direction; the flat display includes one of a rectangular range, an arc range, and an annular range.

[0083] In an embodiment, when a hand holding multiple objects shows different postures, the multiple objects held by the hand can be folded and displayed or laid out and displayed accordingly.

[0084] It should be noted that after one hand activates the multi-select state, in order to maintain the multi-select state, the Pinch state must be maintained.

[0085] After entering the multi-select mode, a hand may contain multiple held objects of different types, including 2D windows, 3D windows, models, text, pictures, folders, clipboards and other UI elements. In the embodiment of the present invention, when these elements are abstracted into cubic bounding boxes, it can be found that these elements have different sizes, heights and thicknesses. In order to facilitate display together, the bounding length, width and height of all elements are unified. Figure 5 , Figure 5 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 1 In the unification process, some elements that are too long or too wide will be reduced or cropped to achieve the purpose of unifying the bounding box size, that is, objects that are too long or too wide will be cropped by the window. After the bounding box size is unified, in order to facilitate the movement of multiple selected objects, the embodiment of the present invention treats these multiple selected objects as a whole.

[0086] On top of each object, there will be markings such as object name, icon (a small graphic symbol with a specified meaning and function), notes, attributes, etc. You can refer to the tabs on a web page for this.

[0087] In the multi-select state, when the palm of the hand is facing down, it is considered the default state.

[0088] Palm down state definition: Figure 6 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 2 ,refer to Figure 6 The palm-down state is defined as the angle between the ray emitted from the palm and the ray with the direction of (0, -1, 0) in the world coordinate system (that is, the ray pointing directly downward in the world coordinate system) is less than or equal to a third threshold (for example, 90 degrees);

[0089] Palm-up state definition: Figure 7 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 3 ,refer to Figure 7, palm-up is defined as the angle between the ray emitted from the palm and the ray with the direction of (0, 1, 0) in the world coordinate system (that is, the ray pointing directly upward in the world coordinate system) is less than a fourth threshold (for example, 90 degrees).

[0090] In one embodiment, when it is detected that any one hand holds multiple objects, a variety of multi-select object operations are provided; the multi-select object operations are: operating all the objects held by any one hand as a whole.

[0091] For example, when a hand is detected holding multiple objects, the hand can hold the multiple objects in the second designated posture or the third designated posture, providing a variety of multi-select object operations. That is, in order to conveniently move or operate the multiple selected objects, they are regarded as a whole and moved.

[0092] The multi-select object operation includes but is not limited to one or any combination of the following common operations:

[0093] Group, ungroup, copy, cut, paste, delete, close, minimize, send to, rename, auto-arrange, auto-layout, create shortcut, edit, open with application. The embodiments of the present invention only list common operations of multiple selection objects, and those skilled in the art should know that the multiple selection object operation may also include other operations.

[0094] In order to further improve the user experience and the smoothness of the multiple selection operation, in a preferred embodiment, when any hand is detected to be performing the second designated posture or the third designated posture: in the augmented reality scene, the designated object selected by the other hand from the multiple objects displayed is highlighted by hovering, and / or, a drag operation is performed after hovering highlighting.

[0095] Figure 8 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 4 ,like Figure 8 As shown, the left hand holds multiple objects in the first designated posture. When the palm of the hand in the multi-select state is in the downward state, in order to conveniently move the selected multiple multi-select objects, these objects are regarded as a whole, and this whole will move with the relevant hand. Figure 8As shown, the multiple selected objects are arranged at a certain distance d in front and back. At the same time, in order to facilitate the other hand to select by ray or contact, these objects are also arranged at a certain distance d' on the left and right (the values ​​of d and d' are set according to the actual situation). This arrangement is similar to the stage of drawing cards in poker, which can ensure the overall convenience of movement and not occupy too much visual field area. When you need to quickly select an object from the multiple selected objects, you can better see some details of the object due to the misalignment of the left and right. In addition, this method is also very friendly to ray and collision body detection, which can better fit the wrist movement arc of the other hand as the selector, so that the hand as the selector does not need to move forward and backward, but only needs to turn the wrist to switch the selected objects more closely at a uniform speed. In order to facilitate users to observe which object will be selected, the Hover object is more prominent. When hovering, you can add animation to the object to emphasize it, such as the Hover object jumping up, the non-selected object tilting to the front and back, the Hover object enlarging, the Hover object changing color, and the Hover object adding a border or mark. If the animation involves angle and position changes, please note that the collision body will also be processed accordingly to ensure the continuity of the operation.

[0096] When the palm is facing up in the multi-select state, in order to better display the details of the multi-selected objects and facilitate users to observe each multi-selected object. The multi-selected objects will be flattened in front of the user in a rectangular range or arc range of length m times width n. The excess part will be displayed by mask cropping. Users can slide the display area by sliding the blank part or dragging the bar to browse the originally blocked part. In order to facilitate observation and timely discover mistakes or objects that are not selected, the latest multi-selected objects will be arranged at the end, and if new multi-selected objects are added, the internal area of ​​the mask will also move accordingly to ensure that the latest multi-selected objects appear at the end and can be seen by users. This state can display the entire or partial screenshots of the multi-selected objects in the mask, allowing users to better browse the details of the multi-selected objects.

[0097] In the embodiment of the present invention, during the multi-selection process, the display mode of the multi-selected objects can also be changed by simple gesture changes, which is convenient for different user needs. This operation conforms to the operating habits of people in the real environment, has a very low learning cost, and can directly start and end the multi-selection without opening any interface. Moreover, when one hand enters the multi-selection state, the other hand can perform completely independent operations without interrupting other operations. This solves the problem of multiple and discontinuous steps in entering the multi-selection mode.

[0098] The following describes how to add and remove multiple selection objects.

[0099] Add multi-select objects: When one hand enters the multi-select state, if the multi-select action mentioned above is triggered, the objects selected by the current hand will be added to the multi-select object set of the other hand that has entered the multi-select state. The newly added multi-select objects will be arranged at the front of the multi-select objects in the default mode. In the hand-up mode, they will appear at the end of the multi-select objects, and if the new multi-select objects appear outside the mask range at this time, there will be an automatic displacement, and the multi-select objects in the mask will be moved as a whole, so that the new multi-select objects appear within the mask range.

[0100] Removing multi-selected objects: In the most common state, when one hand enters the multi-select state and the other hand does not select any object, if the hand that does not select any object selects any object in the multi-selected objects, the object is removed from the set of multi-selected objects. The subdivision of removing multi-selected objects is as follows:

[0101] If an object is selected with a far selector, the object appears at a default distance from the far selector start point in the direction of the far selector, and moves according to the movement rules of the far selector.

[0102] If you select an object with the near selector, the object moves according to the near selector's movement rules. The far selector and the near selector are displayed in the virtual scene.

[0103] In some cases, for better user visual and operational experience, after the object is selected, it can temporarily not comply with the movement rules of the far selector and the near selector or the default operation position, distance, and direction, but continue to move in a small range within the mask range and comply with the responsive design rules, such as being visible outside the mask range, until the user operates the selected object to move further away from the mask range by a certain threshold distance, or there are other attempts to operate the selected object to leave the mask range, such as quickly dragging, moving above or below the mask, etc. When we determine that the user intends to operate the selected object to leave the mask, we will make the selected object comply with the near or far selection rules. This can give users more room for error, enhance user confidence in operation, and improve user operation experience.

[0104] Other operations can also remove multi-select objects, such as: sent, packaged, deleted, closed, and automatically arranged to a certain position. These operations often mean that the object position or state changes, resulting in no longer meeting the requirements of the current multi-select state, in which case the multi-select objects will also be removed.

[0105] The following describes the contents of the end of the multi-select mode.

[0106] In one embodiment, when any one hand is in a multi-selection state, the first designated gesture is ended, and the multiple objects held by the any one hand are released.

[0107] For example, the thumb and index finger of the multi-selecting hand are no longer pinched, and the multiple objects held by the hand are immediately released, ending the multi-selection operation.

[0108] In one embodiment, when the operation of selecting multiple objects is completed, the multiple objects held by any one hand are released.

[0109] For example, in order to send multiple objects to a specified location, when one hand holds multiple multi-select objects, if the first specified gesture is ended, the multi-select objects will be released, ending the multi-select operation, and in some cases the released multi-select objects will be automatically sent to the specified location.

[0110] Specifically, the end of the multiple selection mode includes the following:

[0111] (1) The multi-selection mode ends when there are no more multi-selection objects in the hand in the multi-selection mode.

[0112] (2) The multi-selection mode ends when the hand pinching state in the multi-selection mode ends.

[0113] (3) In the multi-selection mode, when all the multi-selected objects in the hand are grouped and this group of objects is selected by the other hand, or as in the previous paragraph, when it is determined that the other hand has the intention to move this group of objects and leave the multi-selection mode, the multi-selection mode ends.

[0114] (4) When the multiple selection objects in the multi-selection mode are operated individually or in groups to meet the conditions for exiting the multi-selection mode, such as being sent, packaged, deleted, closed, removed, automatically arranged to a certain position, etc., and there are no other multi-selection objects in the multi-selection mode, the multi-selection mode ends.

[0115] (5) Automated operations, such as automatic arrangement, can also be triggered when the multi-select mode ends.

[0116] The above solution is based on the situation that the gesture sensor and the handle are intact and can detect both hands. In actual operation, if there is only a single controller, such as only one hand can be detected, there is only one handle or controller, or there is only head control option, this solution can also be implemented.

[0117] In one embodiment, any operation of a hand (including the first hand and the second hand) is replaced by monitoring data of a hardware device; the monitoring data of the hardware device includes one or any combination of button operation, position parameters, and rotation parameters.

[0118] Only appropriate modifications need to be made: the prompt range trigger and the multi-select event trigger will be suspended separately in front of the body, and the multi-select event trigger itself can also add interactive functions. Through a single controller, the interaction of the multi-select event trigger, such as clicking and long pressing, can simulate the pinch function and gesture change function of multiple players, thereby realizing the switching of display status (simulating the palm up and down status), ending multiple selection and other operations.

[0119] In summary, the embodiments of the present invention have the following technical advantages:

[0120] Compared with the existing solution one: First, the steps of the embodiment of the present invention are simple and there is no need to pre-select multiple selection objects. Each selected object can be used as a condition for starting the multiple selection operation or adding or deleting objects in the multiple selection operation, and the steps are as simple as possible. Second, the embodiment of the present invention does not require too much UI for the basic function of multiple selection, which effectively reduces the user's cognitive load and reduces the cost of learning and operation. Third, the left and right hands are separated in the embodiment of the present invention, and there is no need to keep the hands raised. If the hand in the multiple selection state is a handle, it can be put down at will. If it is gesture recognition, just keep the hand position within the recognition range, and there is no need to keep raising the hand to select. The hand responsible for selection can also make selections or rest at will, and there is no situation where the user is forced to keep raising his hand to complete the operation.

[0121] Compared with the existing solution 2: First, the embodiment of the present invention only needs the most basic rays or gestures to select the target, and there is no need to deliberately adjust the range. The process is much simpler than the existing solution 2. It can also accurately select objects that are blocked, overlapped, and densely arranged. As long as the objects are supported by the most basic operations such as rays and gestures, they are within the applicable scope. This almost covers all current mainstream extended reality operation methods and has a wide range of applications. Second, the embodiment of the present invention is very concise. In the process of implementing multiple selections, simple basic operations such as selection, merging, selection, and merging will be repeated. Objects can be selected without so many operation steps in the existing solution 2. Third, unlike the existing solution 2, the embodiment of the present invention does not require more computing power to render and process the physical calculations of 3D collision bodies. Relying on the most basic selection interaction, it is more cost-effective than the existing solution 2 in terms of performance. Fourth, the embodiment of the present invention is particularly suitable for processing occluded objects, because based on rays or gestures, the unoccluded part of the occluded candidate object can be selected more accurately, and the completely occluded candidate object can also be moved normally to select the candidate object, or the previous occluded object can be crossed at a different angle for accurate selection, which is much simpler than the existing solution 2 in terms of process. Fifth, the embodiment of the present invention is suitable for handles, gesture tracking and even head control, and has low requirements for equipment. As long as basic ray and gesture selection can be performed, this solution can be operated, and its versatility is wider than the existing solution 2.

[0122] The embodiment of the present invention has a wide range of applications. This solution can be used for both 2D and 3D objects, and is applicable to dense or sparse arrangements, occlusion or overlapping arrangements, and is consistent with user cognition. The embodiment of the present invention does not require too many UI controls, and basic operations such as starting, adding, removing objects, and ending multiple selections are performed directly through simple operations. Since the embodiment of the present invention uses two-handed cooperation, selection and multiple selection are separated. The selecting hand can perform multiple selection operations, and can also perform other operations such as selection, movement, clicking, long pressing, etc., without affecting its original operation. Even in the case of a single input source such as one-hand or head control, the auxiliary multi-selection component enters the multi-selection mode, and a single input source such as one-hand or head control can also perform other operations such as selection, movement, clicking, long pressing, etc., and the overall smoothness of the operation is not affected by the multiple selection operation.

[0123] In addition, in the multi-selection state, the embodiment of the present invention also has the advantages of convenient display and operation of multi-selection objects, such as the convenience of moving the multi-selection objects, the default state that does not occupy too much field of view, and the display state that can browse more details can be switched by gestures. Both states are designed for convenient selection and improved usability. And the multi-selection objects can move with the hand, which is also convenient for users to operate the multi-selection objects in a large space or across space, or operate the multi-selection objects as a whole. Compared with the existing solutions, these advantages of the embodiment of the present invention in the multi-selection process can facilitate users to better perform other operations based on multi-selection, and better browse and select multi-selection objects. If faced with the situation where multiple selection objects need to be added or removed, the embodiment of the present invention should also be more concise and fast, without the need to expand too many UI interfaces, or adjust a certain selection area. The embodiment of the present invention uses the most basic selection and movement operations to achieve the above purpose, so it has higher universality in expanding reality, can save users' operation time, and can also reduce learning costs and increase the usability of the solution, thereby optimizing the user's operation experience.

[0124] The present invention also provides a three-dimensional space multiple selection device, as described in the following embodiments. Since the principle of solving the problem by the device is similar to the multiple selection method in three-dimensional space, the implementation of the device can refer to the implementation of the multiple selection method in three-dimensional space, and the repeated parts will not be repeated.

[0125] Fig. 9 is a schematic diagram of a multiple selection device in a three-dimensional space in an embodiment of the present invention, such as Fig. 9 As shown, the device 900 includes:

[0126] The prompt range display module 901 is used to display a prompt range to the user when the user's first hand holds the first object and the second hand holds the second object in the augmented reality scene, and the distance between the first hand and the second hand is less than a first threshold; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scene;

[0127] The multi-select holding module 902 is used to control the other hand to hold the first object and the second object after any hand within the prompt range releases the object held; the any hand is the first hand or the second hand, when the any hand is the first hand, the other hand is the second hand, when the any hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in the augmented reality scene.

[0128] In one embodiment, the number of objects held by any one hand may be one or more, and the number of objects held by the other hand may be one or more.

[0129] In one embodiment, either hand holds the object in a first specified posture.

[0130] In one embodiment, the multi-select holding module 902 is specifically used for: when within the prompt range and the distance between the two hands is less than a second threshold, after either hand releases the held object, controlling the other hand to hold the first object and the second object; the second threshold is less than the first threshold.

[0131] In one embodiment, the multiple objects held by any one hand have different data types and different interaction expressions.

[0132] Fig.10 FIG. 1 is a specific example diagram of a multiple selection device in a three-dimensional space in an embodiment of the present invention. Fig.10 As shown, the device 900 includes a display processing module 1003, and the display processing module 1003 is specifically used for:

[0133] When it is detected that any hand is in a second designated posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second designated posture includes the first designated posture and the palm faces the first direction; the folded display includes front-to-back arrangement display;

[0134] When it is detected that any hand is performing a third designated posture, each object held by the hand is controlled to have a uniform size and to be displayed flatly in the augmented reality scene; the third designated posture includes the first designated posture and the palm faces the second direction; the flat display includes one of a rectangular range, an arc range, and an annular range.

[0135] In one embodiment, the device 900 includes a multiple selection operation module, which is specifically used to:

[0136] When it is detected that any hand holds multiple objects, a variety of multi-select object operations are provided; the multi-select object operation is: all objects held by any hand are operated as a whole.

[0137] In one embodiment, the multi-select object operation includes one or any combination of the following:

[0138] Group, Ungroup, Copy, Paste, Delete, Close, Minimize, Send To, Rename, Auto Arrange, Auto Arrange, Create Shortcut, Edit, Open with App.

[0139] In one embodiment, the display processing module 1003 is specifically used for: when any hand is monitored to perform the second designated posture or the third designated posture: in the augmented reality scene, the designated object selected by the other hand ray from the multiple objects displayed is highlighted by hovering, and / or, a dragging operation is performed after hovering highlighting.

[0140] In one embodiment, when any hand ends the first designated gesture, the multiple objects held by the any hand are released.

[0141] In one embodiment, when the operation of selecting multiple objects is completed, the multiple objects held by any one hand are released.

[0142] In one embodiment, the hand posture and operation are monitored by a gesture sensor or a handle; the gesture sensor includes one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyographic signal gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

[0143] In one embodiment, any operation of the first hand and the second hand is implemented by monitoring data of a hardware device; the monitoring data of the hardware device includes one or any combination of button operations, position parameters, and rotation parameters.

[0144] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multiple selection method in three-dimensional space when executing the computer program.

[0145] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multiple selection method in the three-dimensional space is implemented.

[0146] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the multiple selection method in the three-dimensional space is implemented.

[0147] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0151] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multiple selection method in three-dimensional space, characterized in that: include: When a user's first hand holds a first object and a second hand holds a second object in an augmented reality scene, and a distance between the first hand and the second hand is less than a first threshold, a prompt range is displayed to the user; the prompt range indicates a trigger range for triggering a multi-select operation in the augmented reality scene; When any hand within the prompt range releases the object held, the other hand is controlled to hold the first object and the second object; any one hand is the first hand or the second hand, when any one hand is the first hand, the other hand is the second hand, when any one hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in an augmented reality scene.

2. The method according to claim 1, characterized in that The number of objects held by the first hand or the second hand is one or more.

3. The method according to claim 2, characterized in that Either hand holds the object in a first specified posture.

4. The method according to claim 3, characterized in that Multiple objects held in either hand have different data types and different forms of interaction; When it is detected that any hand is in a second designated posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second designated posture includes the first designated posture and the palm faces the first direction; the folded display includes front-to-back arrangement display; When it is detected that any hand is performing a third designated posture, each object held by the hand is controlled to have a uniform size and to be displayed flatly in the augmented reality scene; the third designated posture includes the first designated posture and the palm faces the second direction; the flat display includes one of a rectangular range, an arc range, and an annular range.

5. The method according to claim 4, characterized in that When it is detected that any hand holds multiple objects, a variety of multi-select object operations are provided; the multi-select object operation is: all objects held by any hand are operated as a whole.

6. The method according to claim 4, characterized in that When any hand is detected to be performing the second designated posture or the third designated posture: in the augmented reality scene, the other hand is directed to a designated object selected from multiple displayed objects: hovering highlighting, and / or, performing a dragging operation after hovering highlighting.

7. The method according to claim 3, characterized in that When any one hand ends the first designated gesture, the multiple objects held by the any one hand are released.

8. The method according to claim 5, characterized in that When the operation of selecting multiple objects is completed, the multiple objects held by any one hand are released.

9. The method according to any one of claims 1 to 8, characterized in that: The hand posture and operation are monitored through a gesture sensor or a handle; the gesture sensor includes one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyographic signal gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

10. The method according to any one of claims 1 to 8, characterized in that: Any operation of any hand is realized by monitoring data of the hardware device; the monitoring data of the hardware device includes one or any combination of button operation, position parameter, and rotation parameter.

11. A multiple selection device in three-dimensional space, characterized in that: include: A prompt range display module, used to display a prompt range to the user when the user's first hand holds a first object and the second hand holds a second object in the augmented reality scene, and the distance between the first hand and the second hand is less than a first threshold; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scene; A multi-select holding module is used to control the other hand to hold a first object and a second object after any hand within a prompt range releases the object held; the any hand is the first hand or the second hand, when the any hand is the first hand, the other hand is the second hand, when the any hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in an extended reality scene.

12. The device according to claim 11, characterized in that The number of objects held by the first hand or the second hand is one or more.

13. The device according to claim 12, characterized in that Either hand holds the object in a first specified posture.

14. The device according to claim 13, characterized in that The multiple objects held by any one hand have different data types and different interactive expressions; The device includes a display processing module, which is specifically used for: When it is detected that any hand is in a second designated posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second designated posture includes the first designated posture and the palm faces the first direction; the folded display includes front-to-back arrangement display; When it is detected that any hand is performing a third designated posture, each object held by the hand is controlled to have a uniform size and to be displayed flatly in the augmented reality scene; the third designated posture includes the first designated posture and the palm faces the second direction; the flat display includes one of a rectangular range, an arc range, and an annular range.

15. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

17. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images