Multi-selection method and apparatus in three-dimensional space

Abstract

The application discloses a multi-selection method and device in a three-dimensional space, and relates to the technical field of extended reality, and the method comprises the following steps: when a first hand of a user holds a first object, a second hand of the user holds a second object, and the distance between the first hand and the second hand is less than a first threshold value in an extended reality scene, a prompt range is displayed to the user; the prompt range represents a trigger range for triggering a multi-selection operation in the extended reality scene; the first object and the second object represent elements in the extended reality scene; after any hand in the prompt range releases the held object, the other hand holds the first object and the second object. The application can reduce the steps of the multi-selection operation in the extended reality scene, quickly and conveniently realize the multi-selection operation, and improve the smoothness of the extended reality operation.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a multiple selection method and device in three-dimensional space. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Multi-selection is an important feature in augmented reality operating systems. However, in an augmented reality environment, objects are often not limited to 2D UI interfaces, and in most cases, especially when manipulating 3D objects, the commonly used shortcut key triggering or key-press multi-selection state methods are not applicable to the augmented reality operating environment due to the lack of keyboard assistance. In augmented reality, multi-selection operations usually require the use of UI interfaces and complex gestures. For example, it is necessary to open an interface to complete a series of selections, terminate other ongoing or attempted operations, and resume other normal operations only after completing the multi-select operation. This process is cumbersome and the operation fluency is poor. In traditional augmented reality environments, this multi-step multi-selection method greatly increases the complexity of operations and reduces the smoothness of operations. Summary of the Invention

[0004] An embodiment of the present invention provides a method for multiple selection in three-dimensional space, which is used to reduce the steps of multiple selection operations in augmented reality, quickly and conveniently implement multiple selection operations, and improve the fluency of augmented reality operations. The method includes:

[0005] When, in an augmented reality scenario, a user's first hand holds a first object and a second hand holds a second object, and a distance between the first hand and the second hand is less than a first threshold, a prompt range is displayed to the user; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scenario;

[0006] When any hand within the prompt range releases the object held, the other hand is controlled to hold the first object and the second object; the either hand is the first hand or the second hand, when the either hand is the first hand, the other hand is the second hand, when the either hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in the augmented reality scene.

[0007] In an embodiment of the present invention, each object selection can be used as a condition for enabling multiple selection operations. With simple coordination of both hands, basic operations such as selecting, holding, moving, and releasing objects can be performed, without requiring additional precise operations. This greatly simplifies the steps of multiple selection operations in augmented reality scenarios, improves the smoothness of augmented reality operations, and enhances the usability and versatility of multiple selection operations in augmented reality. During the multiple selection process, there is no need for excessive UI to occupy too much space in the field of view, and it does not affect the current user interface. This simple design and operation allows users to unburden themselves and operate with ease, optimizing the user's operating experience.

[0008] In one embodiment, the number of objects held by any one hand may be one or more, and the number of objects held by the other hand may be 0, one or more.

[0009] In one embodiment, the object is held by any hand in a first designated posture; the first designated posture comprises a two-finger pinch.

[0010] In one embodiment, when either hand releases the object held within the prompt range, the other hand is controlled to hold the first object and the second object, including: when within the prompt range and the distance between the first hand and the second hand is less than a second threshold, when either hand releases the object held, the other hand is controlled to hold the first object and the second object; the second threshold is less than the first threshold.

[0011] In one embodiment, the multiple objects held by any one hand have different data types and different interaction expressions;

[0012] When any hand is detected to enter a multiple selection state and to be in a second designated posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second designated posture includes the first designated posture and the palm is facing the first direction; the folded display includes a front-to-back arrangement; the multiple selection state indicates holding multiple objects;

[0013] When it is detected that any hand enters a multi-selection state and performs a third specified posture, each object held by the hand is controlled to be of uniform size and displayed flatly in the augmented reality scene; the third specified posture includes the first specified posture and the palm faces the second direction; the flat display includes one of a rectangular range, an arc range, and a circular range.

[0014] In one embodiment, when it is detected that any hand holds multiple objects, various multi-object selection operations are provided; the multi-object selection operations are: operating all objects held by any hand as a whole.

[0015] In one embodiment, the multi-object selection operation includes one or any combination of the following:

[0016] Group, ungroup, copy, paste, delete, close, minimize, send to, rename, auto arrange, auto layout, create shortcut, edit, open with app.

[0017] In one embodiment, when any hand is detected to be performing the second designated posture or the third designated posture: in the augmented reality scene, the other hand is directed to a designated object selected from multiple displayed objects: hovering and highlighting, and / or, performing a dragging operation after hovering and highlighting.

[0018] In one embodiment, when any one hand is in a multi-selection state, the first designation gesture is ended, and the multiple objects held by the any one hand are released.

[0019] In one embodiment, when the multi-object selection operation is completed, the multiple objects held by any one hand are released.

[0020] In one embodiment, the hand posture and operation are monitored by a gesture sensor or a handle; the gesture sensor includes one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyographic gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

[0021] In one embodiment, any operation of any hand can be replaced by monitoring data of a hardware device; the monitoring data of the hardware device includes one or any combination of button operation, position parameters, and rotation parameters.

[0022] An embodiment of the present invention further provides a multi-selection device in a three-dimensional space, which is used to reduce the steps of multi-selection operations in augmented reality, quickly and conveniently implement multi-selection operations, and improve the fluency of augmented reality operations. The device includes:

[0023] A prompt range display module, configured to display a prompt range to the user when, in an augmented reality scenario, the user's first hand holds a first object, the second hand holds a second object, and the distance between the first hand and the second hand is less than a first threshold; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scenario;

[0024] A multi-select holding module is used to control the other hand to hold the first object and the second object when any hand within the prompt range releases the object held; the any hand is the first hand or the second hand, when the any hand is the first hand, the other hand is the second hand, when the any hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in the augmented reality scene.

[0025] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multiple selection method in three-dimensional space when executing the computer program.

[0026] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned multiple selection method in three-dimensional space.

[0027] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned multiple selection method in three-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0029] Figure 1 Schematic diagram of a flow chart of a multiple selection method in a three-dimensional space according to an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 1 ;

[0031] Figure 3 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 2 ;

[0032] Figure 4 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 3 ;

[0033] Figure 5 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 1 ;

[0034] Figure 6 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 2 ;

[0035] Figure 7 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 3 ;

[0036] Figure 8 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 4 ;

[0037] Figure 9 is a schematic diagram of a multiple selection device in a three-dimensional space according to an embodiment of the present invention;

[0038] Figure 10 FIG. 4 is a diagram showing a specific example of a multiple selection device in a three-dimensional space according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0040] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0041] First, the technical terms involved in the present invention are explained.

[0042] Pinch: In the present invention, it refers to the state of selecting a target by hand or using a controller. It is generally triggered by a gesture: pinching the index finger and thumb together, closing the five fingers together, or pressing the trigger key (trigger key) or the grip key (the trigger key is optional) on the controller. Other gestures or button triggering methods for selecting a target may also be considered.

[0043] Hand: Hands refer to controllers such as gestures and gamepads, as these often serve similar purposes as input sources in augmented reality. In some special cases, a single input source, such as single-handed gesture recognition, a single gamepad, a mobile phone, a separate controller, or a head-mounted display, is also considered a hand.

[0044] Multi-selection has always been a crucial feature in augmented reality (AR) operating systems. However, in AR environments, objects are often not limited to 2D UI interfaces. Furthermore, in most cases, especially when manipulating 3D objects, and without keyboard assistance, the commonly used shortcut keys to trigger multi-selection are not suitable for AR environments. In AR, multi-selection is typically achieved through UI interfaces and complex gestures. This requires opening the interface, completing a series of selections, terminating other ongoing or attempted operations, and then completing the multi-selection operation before resuming normal operation. This process is cumbersome and has poor operational fluency. In traditional AR environments, this multi-step multi-selection method greatly increases operational complexity and reduces operational fluency.

[0045] Two multiple selection operation solutions in the prior art are introduced below.

[0046] Existing solution one: providing a graphical user interface through a head-mounted display device, responding to the player's gaze operation and first gesture change information, determining multiple objects to be selected; determining the depth area where each object to be selected is located and the mapped depth distance corresponding to each object to be selected, the mapped depth distance being used to indicate the distance between the depth area where the object to be selected is located and the adjacent lower depth area; responding to the player's second gesture change information, determining a target selected object from the multiple objects to be selected based on the depth area where each object to be selected is located and the mapped depth distance corresponding to each object to be selected, and displaying the target selected object in a preset display mode.

[0047] To facilitate the selection of objects at different depths, existing solution 1 incorporates hand-eye interaction to achieve selection by further selecting objects from among multiple selected candidates. The drawbacks of existing solution 1 are obvious: First, it involves multiple steps, as it requires pre-selecting multiple candidates and then using gestures to further select them at different depths. This effectively involves selecting from a large number of candidates once, then again. Even when there are multiple candidates at the same depth, horizontal movement is required to select again, requiring at least two or even three selections before the object is selected. The high number of steps and the introduction of different gestures not only increase operation time, but also the learning and operating costs. Second, existing solution 1 requires UI assistance to complete the multi-selection operation, and this UI occupies a large portion of the user's field of view. During operation, solution 1 requires users to focus on the multi-selection operation, hindering other operations and disrupting the overall user experience. Third, due to these two reasons, users need to keep their hands raised in the air to perform multi-selection, and they cannot perform other operations simultaneously. This increases the overall time required to complete tasks involving multi-selection, which not only increases user fatigue but also increases device battery usage and reduces device usability.

[0048] Existing solution two: generating a selection area in three-dimensional space and receiving user input of a selection path to define a selection path, defining a selection volume based on the selection area and the selection path, and selecting entities in the 3D space based on the selection volume, receiving user input of a changed selection area while defining the selection path, and the size or shape of the selection area can be changed in response thereto.

[0049] Existing Solution 2 combines interactive elements such as selection paths, selection volumes, and an interface, making it suitable for selecting large numbers of 3D objects. However, it has the following drawbacks: First, while it can select overlapping or closely spaced objects using trajectory and post-processing range resizing, accurate object selection requires more time to draw precise trajectories and carefully adjust the selection area to keep the desired objects within the selected area and the unwanted objects outside. This is time-consuming and not always user-friendly, limiting its use cases. Second, Solution 2's operational complexity significantly increases the learning curve for users, from initially setting the selection area to drawing the trajectory and repeatedly adjusting the range through the UI. The numerous steps and time consumption also negatively impact the user experience. Third, it requires high rendering performance. Real-time display of the selection volume and path, as well as tracking user input such as controllers or gestures, requires significant computing resources. This demand can place a significant burden on system performance, impacting device responsiveness and smoothness, especially when rendering complex scenes. Fourth, there's the issue of visual occlusion when selecting multiple objects. In complex 3D scenes, if multiple objects overlap or are close together, the selection volume may obstruct the user's view, making it difficult for them to clearly see all candidate objects. In this case, the user may need to repeatedly adjust the selection volume, affecting operational efficiency. Fifth, there's the dependency on gestures and controllers. This system relies on handheld controllers or similar devices for path definition and volume generation, making it less user-friendly for users who don't have such devices or who rely solely on gesture recognition. Further expansion to gesture recognition requires high-precision sensors to achieve similar operations.

[0050] In general, existing multi-selection solutions have many steps, high operational complexity, and the UI occupies the field of vision, forcing users to focus on the multi-selection operation, which fragments the operational experience. Some solutions also have difficulty in making accurate selections, and problems such as visual occlusion and device dependence also affect the user experience.

[0051] The embodiments of the present invention aim to reduce learning costs and operation time and improve the fluency and comfort of augmented reality operations through a natural and smooth multi-select interaction method.

[0052] Figure 1 FIG. 1 is a flow chart of a multiple selection method in a three-dimensional space according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0053] Step 101: When a user holds a first object in a first hand and a second object in a second hand in an augmented reality scene, and the distance between the first hand and the second hand is less than a first threshold, a prompt range is displayed to the user; the prompt range indicates a trigger range for triggering a multi-select operation in the augmented reality scene;

[0054] Step 102: When any hand within the prompt range releases the object held, control the other hand to hold the first object and the second object; the either hand is the first hand or the second hand, when the either hand is the first hand, the other hand is the second hand, when the either hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in the augmented reality scene.

[0055] The following is a detailed explanation of the multiple selection method in three-dimensional space according to an embodiment of the present invention.

[0056] During implementation, the user is placed in an augmented reality scene, and the hand posture and operation are detected by a gesture sensor or handle; the gesture sensor includes but is not limited to one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyographic signal gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

[0057] For example, a user can wear a headset and have their hand gestures monitored by the headset's camera, or they can be in a motion capture environment and have their hand gestures monitored by the camera. The user's hands can be bare, or they can be wearing hand controllers such as handles, or sensors like gloves or wristbands to enable multiple selections.

[0058] In one embodiment, the hand (the first hand or the second hand) holds the object in a first specified posture.

[0059] In the embodiment of the present invention, the user has two hands, wherein the first hand is not limited to the left hand or the right hand, and the second hand is not limited to the left hand or the right hand. When the first hand is the left hand, the second hand is the right hand, and when the first hand is the right hand, the second hand is the left hand.

[0060] In an embodiment, the object is a first object or a second object, and the object represents an element in the augmented reality scene, for example, any holdable element in the augmented reality scene, such as a ball, a file, etc.

[0061] In a preferred embodiment, the first designated gesture comprises a two-finger pinch.

[0062] When the user wants to select an element in the extended reality scene, they can do so by pinching two fingers together.

[0063] To improve the accuracy and usability of the multi-select operation, a second threshold can be further set according to different products and user operating habits. When any hand within the prompt range releases the object held, the other hand is controlled to hold the first and second objects, including:

[0064] When within the prompt range, that is, the distance between the first hand and the second hand is less than a second threshold, after either hand releases the object held, the other hand is controlled to hold the first object and the second object; the second threshold is less than the first threshold.

[0065] That is, within the prompt range, multiple selections can only be achieved when the two hands are closer together to a distance smaller than the second threshold.

[0066] In an embodiment, the first hand holds one or more objects, and the second hand also holds one or more objects. As long as the two hands are close to each other to a distance less than a second threshold, if either hand releases the one or more objects it holds, the other hand will simultaneously hold all the objects previously held by the two hands.

[0067] That is, in step 102, when within the prompt range: after the first hand releases the first object held, the second hand is controlled to hold the first object and the second object, or after the second hand releases the second object held, the first hand is controlled to hold the first object and the second object. The number of objects held by each hand may include one or more.

[0068] For example, the left hand holds multiple pictures and the right hand also holds multiple video files. When the two hands are close to less than a second threshold: if the left hand releases the multiple pictures, the right hand will hold multiple pictures and multiple video files at the same time; when the two hands are close to less than the second threshold: if the right hand releases the multiple video files, the left hand will hold multiple pictures and multiple video files at the same time.

[0069] In step 102 , the number of objects held by any one hand may be one or more, and the number of objects held by the other hand may be one or more.

[0070] The embodiment of the present invention provides multiple trigger components, including: a hand trigger point, a prompt range trigger, and a multi-select event trigger.

[0071] (1) Hand trigger point: represents the position when the hand is pinched. In order to have a better triggering experience, the hand trigger point is set near the point where the thumb and index finger are pinched on the inner side of the hand. Figure 2 , Figure 2 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 1 , hold the object with your left and right hands by pinching the thumb and index finger respectively, Figure 2 The objects held in the picture are only indicated and not specifically depicted.

[0072] (2) When one hand selects an object and the trigger point of the hand enters the prompt range trigger range of the other hand, the multi-select range prompt will be triggered, which makes it easier for users to see when the multi-select state will be triggered. Figure 3 , Figure 3Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 2 ,When certain conditions are met, for example, when both hands hold one or more objects, and the hand trigger point of one hand enters the prompt range trigger range of the other hand, the ,multi-selection range prompt will be triggered.

[0073] (3) Furthermore, when one hand selects an object and the trigger point of the hand enters the multi-select event trigger range of the other hand, if the hand selecting the object releases the hand, the multi-selection will be triggered. Figure 4 , Figure 4 Schematic diagram of triggering the multi-selection mode in an embodiment of the present invention Figure 3 ,When the hand trigger point of the hand holding the object enters the multi-select event trigger range of the other hand, ,the hand holding the object releases its grip, releasing the held object ,which will trigger multi-selection, and the other hand will hold two objects at the same time.

[0074] (4) The prompt range trigger and the multi-select event trigger can also be replaced by the first threshold S and the second threshold s, corresponding to Figure 2 Medium prompt range trigger circle range mark, multi-select event trigger circle range mark. Figure 4 In the prompt range trigger and the multi-select event trigger, although both use spherical colliders, other more complex colliders can also be used, such as multiple spherical colliders or colliders of other shapes, or more complex threshold judgments. In this case, simple threshold judgments cannot replace the colliders and need to be handled in combination with specific circumstances.

[0075] When a user manipulates objects in an augmented reality environment and needs to control multiple objects for multi-selection, while holding object A in one hand and object B in the other, they can trigger multi-selection by bringing the hand holding object A close to the hand holding object B, with the distance between the two hands less than a second threshold, s, and then releasing them. If object A is released at this time, the hand holding object B enters multi-selection mode and holds both objects A and B. The hand originally holding object A returns to its initial state and can perform other operations or select more objects and repeat the previous process for multi-selection.

[0076] If one hand holds object A and the other hand does not, the user can first bring the hand holding object A closer to the hand not holding the object. If the hand not holding the object is in the Pinch state, the hand holding object A moves closer to the hand not holding the object and reaches a second threshold s. Object A is then released, and object A is then held by the hand not holding the object, completing the hand-switch. Repeating the previous step can also activate the multi-select state. In this embodiment of the present invention, the Pinch state is achieved through the first specified gesture.

[0077] That is, in the augmented reality scenario, when the user's first hand does not hold any object, the second hand holds the second object, and the distance between the first hand and the second hand is less than a first threshold, the prompt range is displayed to the user; when the second hand releases the object held within the prompt range, the first hand is controlled to hold the second object; in the augmented reality scenario, when the user's first hand holds the first object, the second hand does not hold an object, and the distance between the first hand and the second hand is less than a first threshold, the prompt range is displayed to the user; when the first hand releases the object held within the prompt range, the second hand is controlled to hold the first object.

[0078] The present invention provides a novel multi-selection operation method that combines left-hand and right-hand operation. From the user's perspective, when both hands are holding objects, the user simply places the object held in one hand onto the other hand to automatically trigger the multi-selection state. The hand that triggers the multi-selection state then holds the selected objects, while the other hand can continue other operations or select more objects to add to the multi-selection set. Compared to existing solutions, this method offers simpler and smoother operation, and a better user experience.

[0079] The following describes the content of the multiple-select display.

[0080] In one embodiment, multiple objects held in either hand have different data types and different interactive presentation forms. The data types include, but are not limited to, text, images, video, audio, files, structured data, binary data, three-dimensional models, application windows, and three-dimensional subspaces, and the interactive presentation forms include, but are not limited to, 2D, 3D, or a combination of 2D and 3D. Each interactive presentation form can be appropriately selected and presented based on the interactive needs within the augmented reality virtual space. Those skilled in the art will appreciate that interactive objects within the augmented reality virtual space can be a combination of multiple data types or presentation forms, and therefore are not listed here.

[0081] When any hand is detected to enter a multi-select state and to be in a second designated posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second designated posture includes the first designated posture and the palm is facing the first direction; the folded display includes a front-to-back arrangement display; the hand is the first hand or the second hand; for example, in the augmented reality scene, in front of the user's perspective, the objects are displayed in a front-to-back arrangement and staggered to the left and right by a designated distance (reference Figure 8 );

[0082] When it is detected that any hand enters a multi-selection state and performs a third specified posture, each object held by the hand is controlled to be of uniform size and displayed flatly in the augmented reality scene; the third specified posture includes the first specified posture and the palm faces the second direction; the flat display includes one of a rectangular range, an arc range, and a circular range.

[0083] In an embodiment, when a hand holding multiple objects shows different postures, the multiple objects held by the hand can be folded and displayed or flattened and displayed accordingly.

[0084] It should be noted that if you want to maintain the multi-select state after activating the multi-select state with one hand, you need to keep the Pinch state.

[0085] After entering the multi-select mode, a hand may contain multiple objects of different types, including 2D windows, 3D windows, models, text, pictures, folders, clipboards and other UI elements. In the embodiment of the present invention, when these elements are abstracted into cubic bounding boxes, it can be found that these elements have different sizes, heights, and thicknesses. In order to facilitate the display together, the bounding length, width and height of all elements are unified. Figure 5 , Figure 5 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 1 During the unification process, some elements that are too long or too wide will be reduced or cropped to achieve the purpose of unifying the bounding box size. In other words, objects that are too long or too wide will be cropped by the window. After the bounding box size is unified, in order to facilitate the movement of multiple selected objects, this embodiment of the present invention treats these multiple selected objects as a whole.

[0086] Each object will also have a label on it, indicating the object name, icon (a small graphic symbol with a specified meaning and function), notes, properties, etc. This can be referred to as the tab page of a web page.

[0087] In the multi-select state, when the palm of the hand is facing down, it is considered the default state.

[0088] Palm down state definition: Figure 6 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 2 ,refer to Figure 6 The palm-down state is defined as the angle between the ray emitted from the palm and the ray with the direction of (0, -1, 0) in the world coordinate system (that is, the ray pointing directly downward in the world coordinate system) is less than or equal to a third threshold (for example, 90 degrees);

[0089] Palm-up state definition: Figure 7 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 3 ,refer to Figure 7Palm-up is defined as the angle between the ray emitted from the palm and the ray with the direction of (0, 1, 0) in the world coordinate system (that is, the ray pointing directly upward in the world coordinate system) is less than a fourth threshold (for example, 90 degrees).

[0090] In one embodiment, when it is detected that any hand holds multiple objects, various multi-object selection operations are provided; the multi-object selection operations are: operating all objects held by any hand as a whole.

[0091] For example, when a hand is detected holding multiple objects, the hand can hold multiple objects in the second or third designated posture, providing a variety of multi-select object operations. In other words, in order to facilitate the movement or operation of multiple selected objects, they are treated as a whole and moved.

[0092] The multi-object selection operation includes but is not limited to one or any combination of the following common operations:

[0093] Group, Ungroup, Copy, Cut, Paste, Delete, Close, Minimize, Send to, Rename, Auto Arrange, Auto Arrange, Create Shortcut, Edit, Open with Application. The embodiments of the present invention only list common operations for selecting multiple objects. Those skilled in the art will appreciate that multiple object selection operations may also include other operations.

[0094] In order to further improve the user experience and the smoothness of the multiple selection operation, in a preferred embodiment, when any hand is detected to be performing the second designated posture or the third designated posture: in the augmented reality scene, the designated object selected from the multiple objects displayed by the other hand is rayed: hovering highlighting, and / or, dragging operation is performed after hovering highlighting.

[0095] Figure 8 Schematic diagram of the multi-select mode in the embodiment of the present invention Figure 4 ,like Figure 8 As shown, the left hand holds multiple objects in the first designated posture. When the palm of the hand in the multi-select state is facing downward, in order to conveniently move the selected multiple objects, these objects are regarded as a whole, and this whole will move with the relevant hand. Figure 8As shown, multiple selected objects are arranged with a certain spacing d in front and back. To facilitate selection using raycast or contact methods with the other hand, these objects are also arranged with a certain spacing d' in the left and right directions (the values ​​of d and d' are set according to actual conditions). This arrangement is similar to the drawing phase of a deck of cards: it ensures easy movement without occupying too much visual area. Furthermore, when quickly selecting a single object from the multi-selected list, the left-right offset allows for better visibility of the object's details. This approach is also very user-friendly for raycast and collision detection, as it better tracks the wrist motion of the other hand, allowing the selecting hand to switch objects at a more consistent speed without moving back and forth, simply by rotating the wrist. To make it easier for users to see which object will be selected, hovering objects are highlighted. Hovering objects can be animated to emphasize this, such as making the hovered object leap upward, tilting non-selected objects forward and backward, enlarging the hovered object, changing its color, or adding a border or marker to the hovered object. If animations involve angle or position changes, be aware that collisions are also handled accordingly to ensure operational continuity.

[0096] When the palm is facing up in the multi-select state, the multi-selected objects are displayed in a rectangular or curved area (m by n) in front of the user, with the excess area masked and cropped. The user can slide the display area by sliding the blank area or dragging the bar to view the previously obscured area. To facilitate observation and promptly identify incorrect or unwanted selections, the most recently added multi-selected objects are arranged at the end. If new objects are added, the mask area will also move accordingly to ensure that the most recently added multi-selected objects appear at the end and are visible to the user. This state can display a full or partial screenshot of the multi-selected objects in the mask, allowing users to better view the details of the multi-selected objects.

[0097] In an embodiment of the present invention, during the multi-selection process, the display of multiple selected objects can be changed through simple gestures, accommodating different user needs. This operation conforms to people's operating habits in real-world environments, has a very low learning cost, and can directly start and end multi-selection without opening any interface. Moreover, while one hand enters the multi-selection state, the other hand can perform completely independent operations without interrupting other operations. This solves the problem of multiple and discontinuous steps in entering the multi-selection mode mentioned above.

[0098] The following describes how to add and remove multiple selection objects.

[0099] Adding Multi-Select Objects: When one hand is in the Multi-Select state, triggering the aforementioned Multi-Select action will add the objects currently selected by the hand to the Multi-Select set of the other hand. In default mode, the newly added multi-selected objects will appear at the front of the multi-selected set. In hand-up mode, however, they will appear at the end of the multi-selected set. If the newly added multi-selected objects appear outside the mask range, they will automatically shift, moving the entire multi-selected set within the mask so that the new objects appear within it.

[0100] Removing multi-selected objects: In the most common state, when one hand enters the multi-select state and the other hand does not select any objects, if the hand that does not select any objects selects any object in the multi-selected objects, the object will be removed from the multi-selected objects. The details of removing multi-selected objects are as follows:

[0101] If an object is selected with the Far Selector, the object appears at a default distance from the Far Selector's starting point in the direction of the Far Selector, and moves according to the Far Selector's movement rules.

[0102] If you select an object with the near selector, the object will move according to the near selector's movement rules. The far selector and the near selector are displayed in the virtual scene.

[0103] In some cases, to provide a better user experience and visual experience, after an object is selected, the movement rules of the far and near selectors, or the default operation position, distance, and direction, can be temporarily disregarded. Instead, the object continues to move within the mask in a small range and adheres to responsive design rules, such as being visible outside the mask. This occurs until the user moves the selected object farther from the mask than a certain threshold, or attempts to move the selected object out of the mask by quickly dragging it, moving it up or down the mask, and so on. Once we determine that the user intends to move the selected object out of the mask, we then adapt the selected object to the near or far selection rules. This gives users more room for error, increases their confidence, and improves their operational experience.

[0104] Other operations can also remove multi-selected objects, such as sending, packaging, deleting, closing, and automatically arranging them. These operations often indicate that the object's position or state has changed, causing it to no longer meet the requirements of the current multi-select state. In these cases, the multi-selected objects will also be removed.

[0105] The following describes the content of the end of the multi-select mode.

[0106] In one embodiment, when any one hand is in a multi-selection state, the first designation gesture is ended, and the multiple objects held by the any one hand are released.

[0107] For example, the thumb and index finger of the hand that is multi-selecting are no longer pinched, and the multiple objects held by the hand are immediately released, ending the multi-selection operation.

[0108] In one embodiment, when the multi-object selection operation is completed, the multiple objects held by any one hand are released.

[0109] For example, in order to send multiple objects to a specified location, when one hand holds multiple multi-select objects, if the first specified gesture is ended, the multi-select objects will be released, ending the multi-select operation, and in some cases the released multi-select objects will be automatically sent to the specified location.

[0110] Specifically, the end of the multi-select mode includes the following:

[0111] (1) The multi-selection mode ends when there are no more objects in the hand in the multi-selection mode.

[0112] (2) The multi-selection mode ends when the hand in the multi-selection mode ends the pinching state.

[0113] (3) When all the multi-select objects in the hand are grouped and this group of objects is selected by the other hand, or as in the case of the previous paragraph, when it is determined that the other hand has the intention to move this group of objects and exit the multi-select mode, the multi-select mode ends.

[0114] (4) When the multiple selection objects in the multi-selection mode are operated individually or in groups to meet the conditions for exiting the multi-selection mode, such as being sent, packaged, deleted, closed, removed, automatically arranged to a certain position, etc., and there are no other multi-selection objects in the multi-selection mode, the multi-selection mode ends.

[0115] (5) Automatic operations, such as automatic arrangement, can also be triggered when the multi-select mode ends.

[0116] The above solution is based on the situation where the gesture sensor and the handle are intact and can detect both hands. In actual operation, if there is only a single controller, such as only one hand can be detected, only one handle or controller, or only head control option, this solution can also be implemented.

[0117] In one embodiment, any operation of a hand (including the first hand and the second hand) is replaced by monitoring data of a hardware device; the monitoring data of the hardware device includes one or any combination of button operation, position parameters, and rotation parameters.

[0118] Only appropriate modifications are required: the prompt range trigger and the multi-select event trigger will float separately in front of the body, and the multi-select event trigger itself can also add interactive functions. Through a single controller, interactions with the multi-select event trigger, such as clicking and long pressing, can simulate the pinch function and gesture change function of multiple players, thereby achieving operations such as switching display states (simulating palm-up and palm-down states) and ending multi-selection.

[0119] In summary, the embodiments of the present invention have the following technical advantages:

[0120] Compared with the existing solution one: First, the embodiment of the present invention has simple steps, and there is no need to pre-select multiple selection objects. Each selection of an object can be used as a condition for starting a multiple selection operation or adding or deleting objects in a multiple selection operation, and the steps are as simple as possible. Second, the embodiment of the present invention does not require too much UI for the basic function of multiple selection, which effectively reduces the user's cognitive load and reduces the cost of learning and operation. Third, the embodiment of the present invention separates the left and right hands, and there is no need to keep the hands raised. If the hand in the multi-selection state is a handle, it can be put down at will. If it is gesture recognition, it only needs to keep the hand within the recognition range, and there is no need to keep raising the hand to select. The hand responsible for selection can also make selections or rest at will, and there is no situation where the user is forced to keep raising his hand in order to complete the operation.

[0121] Compared with the existing solution 2: First, the embodiment of the present invention only needs the most basic rays or gestures to select the target, and there is no need to deliberately adjust the range. The process is much simpler than the existing solution 2. It can also accurately select objects that are blocked, overlapped, and densely arranged. As long as the objects are supported by the most basic operations such as rays and gestures, they are within the scope of application. This almost covers all current mainstream augmented reality operation methods and has a wide range of applications. Second, the embodiment of the present invention is very simple. In the process of implementing multiple selections, simple basic operations such as selection, merging, selection, and merging will be repeated. Objects can be selected without so many operation steps in the existing solution 2. Third, unlike the existing solution 2, the embodiment of the present invention does not require more computing power to render and process the physical calculations of 3D collision bodies. Relying on the most basic selection interaction, it is more cost-effective than the existing solution 2 in terms of performance. Fourth, this embodiment of the present invention is particularly suitable for handling occluded objects. Based on rays or gestures, the unobstructed portion of an obstructed candidate can be more accurately selected. For completely obstructed candidates, the occluding object can be moved normally before selecting the candidate, or the previously obstructed object can be moved from a different angle for precise selection. This process is much simpler than the existing second solution. Fifth, this embodiment of the present invention is applicable to controllers, gesture tracking, and even head control. It has low device requirements and can be used as long as basic ray or gesture selection is possible. Its versatility is wider than the existing second solution.

[0122] The embodiment of the present invention has a wide range of uses. Whether it is a 2D object or a 3D object, this solution can operate. Whether it is densely or sparsely arranged, blocked or overlapping, this solution is also applicable, and it conforms to the user's cognition. The embodiment of the present invention does not require too many UI controls, and basic operations such as starting, adding, removing objects, and ending multiple selections are performed directly through simple operations. Since the embodiment of the present invention uses two hands to cooperate, selection and multiple selection are separate operations. The selecting hand can perform multiple selection operations, and can also perform other operations such as selection, movement, clicking, long pressing, etc., without affecting its original operation. Even in the case of a single input source such as one hand or head control, the auxiliary multi-selection component enters the multi-selection mode, and a single input source such as one hand or head control can also perform other operations such as selection, movement, clicking, long pressing, etc., and the overall smoothness of the operation is not affected by the multiple selection operation.

[0123] In addition, in the multi-select state, the embodiment of the present invention also has the advantage of convenient display and operation of multi-select objects. For example, the default state that is convenient to move and does not occupy too much field of view and the display state that allows browsing more details can be switched by gestures. Both states are designed to facilitate selection and improve usability. In addition, the multi-select objects can move with the hand, which is also convenient for users to operate multi-select objects in a large space or across spaces, or to operate multi-select objects as a whole. Compared with existing solutions, these advantages of the embodiment of the present invention in the multi-selection process can facilitate users to better perform other operations based on multi-selection and better browse and select multi-select objects. If faced with the need to add or remove multi-select objects, the embodiment of the present invention is also more concise and fast, without the need to expand too many UI interfaces or adjust a certain selection area. The embodiment of the present invention uses the most basic selection and movement operations to achieve the above purpose. Therefore, it has higher universality in augmented reality, can save users' operation time, and can also reduce learning costs and increase the usability of the solution, thereby optimizing the user's operating experience.

[0124] The present invention also provides a three-dimensional multi-selection device, as described in the following embodiments. Since the principle of the device is similar to that of the three-dimensional multi-selection method, the implementation of the device can refer to the implementation of the three-dimensional multi-selection method, and the repeated parts will not be repeated.

[0125] Figure 9 is a schematic diagram of a multiple selection device in a three-dimensional space according to an embodiment of the present invention. Figure 9 As shown, the apparatus 900 includes:

[0126] Prompt range display module 901 is configured to display a prompt range to the user when, in an augmented reality scenario, the user's first hand holds a first object, the second hand holds a second object, and the distance between the first hand and the second hand is less than a first threshold; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scenario;

[0127] The multi-select holding module 902 is used to control the other hand to hold the first object and the second object when any hand within the prompt range releases the object held. The any hand is the first hand or the second hand, when the any hand is the first hand, the other hand is the second hand, when the any hand is the second hand, the other hand is the first hand, and the object is the first object or the second object, and the object represents an element in the augmented reality scene.

[0128] In one embodiment, the number of objects held by any one hand may be one or more, and the number of objects held by the other hand may also be one or more.

[0129] In one embodiment, either hand holds the object in a first specified posture.

[0130] In one embodiment, the multi-select holding module 902 is specifically configured to: when within the prompt range and the distance between the two hands is less than a second threshold, after either hand releases the object it is holding, control the other hand to hold the first object and the second object; the second threshold is less than the first threshold.

[0131] In one embodiment, the multiple objects held by any one hand have different data types and different interaction expressions.

[0132] Figure 10 FIG. 1 is a specific example diagram of a multiple selection device in a three-dimensional space according to an embodiment of the present invention. Figure 10 As shown, the apparatus 900 includes a display processing module 1003, which is specifically configured to:

[0133] When any hand is detected to be in a second specified posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second specified posture includes the first specified posture and the palm of the hand faces the first direction; the folded display includes front-to-back arrangement;

[0134] When any hand is detected to be in the third specified posture, each object held by the hand is controlled to be of uniform size and displayed flatly in the augmented reality scene; the third specified posture includes the first specified posture and the palm faces the second direction; the flat display includes one of the flat display within a rectangular range, an arc range, and an annular range.

[0135] In one embodiment, the apparatus 900 includes a multiple-selection operation module, which is specifically configured to:

[0136] When it is detected that any hand holds multiple objects, a variety of multi-object selection operations are provided; the multi-object selection operation is: all objects held by any hand are operated as a whole.

[0137] In one embodiment, the multi-object selection operation includes one or any combination of the following:

[0138] Group, ungroup, copy, paste, delete, close, minimize, send to, rename, auto arrange, auto layout, create shortcut, edit, open with app.

[0139] In one embodiment, the display processing module 1003 is specifically used to: when any hand is detected to be performing the second specified posture or the third specified posture: in the augmented reality scene, the specified object selected by the other hand ray from the multiple objects displayed: hover highlighting, and / or, performing a drag operation after hover highlighting.

[0140] In one embodiment, when any hand ends the first designated gesture, the multiple objects held by the hand are released.

[0141] In one embodiment, when the multi-object selection operation is completed, the multiple objects held by any one hand are released.

[0142] In one embodiment, the hand posture and operation are monitored by a gesture sensor or a handle; the gesture sensor includes one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyographic gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

[0143] In one embodiment, any operation of the first hand and the second hand is replaced by monitoring data of a hardware device; the monitoring data of the hardware device includes one or any combination of button operation, position parameters, and rotation parameters.

[0144] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multiple selection method in three-dimensional space when executing the computer program.

[0145] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned multiple selection method in three-dimensional space.

[0146] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned multiple selection method in three-dimensional space.

[0147] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0151] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for multiple selection operations in three-dimensional space, characterized in that: include: When, in an augmented reality scenario, a user's first hand holds a first object and a second hand holds a second object, and a distance between the first hand and the second hand is less than a first threshold, a prompt range is displayed to the user; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scenario; When any hand within the prompt range releases the object held, the other hand is controlled to hold the first object and the second object; the either hand is the first hand or the second hand, when the either hand is the first hand, the other hand is the second hand, when the either hand is the second hand, the other hand is the first hand, the object is the first object or the second object, and the object represents an element in the augmented reality scene; either hand holds the object in a first specified posture; Multiple objects held in either hand have different data types and different forms of interaction; When any hand is detected to be in a second specified posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second specified posture includes the first specified posture and the palm of the hand faces the first direction; the folded display includes front-to-back arrangement; When any hand is detected to be in a third specified posture, each object held by the hand is controlled to be uniform in size and displayed flatly in the augmented reality scene; the third specified posture includes the first specified posture and the palm of the hand faces the second direction; the flat display includes flat display within one of a rectangular range, an arc range, and an annular range; When any hand is detected holding multiple objects, a variety of multi-select object operations are provided; the multi-select object operations are: all objects held by any hand are operated as a whole; the multi-select object operations include one or any combination of the following common operations: group, ungroup, copy, cut, paste, delete, close, minimize, send to, rename, auto arrange, auto layout, create shortcut, edit, and open with application; When any hand is detected to be in the second designated posture or the third designated posture: in the augmented reality scene, the other hand is directed to a designated object selected from a plurality of displayed objects: hovering to highlight the designated object, and / or performing a dragging operation after hovering to highlight the designated object; When the multi-object selection operation is completed, the multiple objects held by any one hand are released.

2. The method according to claim 1, wherein The number of objects held by the first hand or the second hand is one or more.

3. The method according to claim 1, wherein When any hand ends the first designated gesture, the multiple objects held by the any hand are released.

4. The method according to any one of claims 1 to 3, characterized in that The hand posture and operation are monitored through a gesture sensor or a handle; the gesture sensor includes one of an optical gesture sensor, an inertial gesture sensor, a capacitive gesture sensor, an electromyographic gesture sensor, an ultrasonic gesture sensor, a millimeter wave sensor, and a magnetic field gesture sensor.

5. The method according to any one of claims 1 to 3, characterized in that: Any operation of any hand is replaced by the monitoring data of the hardware device; the monitoring data of the hardware device includes one or any combination of button operation, position parameters, and rotation parameters.

6. A multi-selection operation device in three-dimensional space, characterized in that: include: A prompt range display module, configured to display a prompt range to the user when, in an augmented reality scenario, the user's first hand holds a first object, the second hand holds a second object, and the distance between the first hand and the second hand is less than a first threshold; the prompt range represents a trigger range for triggering a multi-select operation in the augmented reality scenario; A multi-select holding module, configured to control the other hand to hold a first object and a second object when any hand within the prompt range releases the object held; the either hand is the first hand or the second hand, the other hand is the second hand when the either hand is the first hand, and the other hand is the first hand when the either hand is the second hand; the object is the first object or the second object, and the object represents an element in an augmented reality scene; the either hand holds the object in a first specified posture; The multiple objects held by any one hand have different data types and different interaction expressions; The device includes a display processing module, which is specifically used to: When any hand is detected to be in a second specified posture, each object held by the hand is controlled to be folded and displayed in a uniform size in the augmented reality scene; the second specified posture includes the first specified posture and the palm of the hand faces the first direction; the folded display includes front-to-back arrangement; When any hand is detected to be in a third specified posture, each object held by the hand is controlled to be uniform in size and displayed flatly in the augmented reality scene; the third specified posture includes the first specified posture and the palm of the hand faces the second direction; the flat display includes flat display within one of a rectangular range, an arc range, and an annular range; The device includes a multi-selection operation module, which is specifically used to: when it is detected that any hand holds multiple objects, provide multiple multi-selection object operations; the multi-selection object operations are: operating all objects held by any hand as a whole; the multi-selection object operations include one or any combination of the following common operations: group, ungroup, copy, cut, paste, delete, close, minimize, send to, rename, auto arrange, auto layout, create shortcut, edit, and open with application; The display processing module is specifically configured to: when any hand is detected to be in the second designated posture or the third designated posture: in the augmented reality scene, highlight the designated object selected by the other hand ray from the multiple displayed objects by hovering, and / or, perform a drag operation after hovering and highlighting; When any hand ends the first designated gesture, the multiple objects held by the any hand are released.

7. The device according to claim 6, characterized in that The number of objects held by the first hand or the second hand is one or more.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images