Gesture interaction method, electronic equipment, storage medium and program product
By using different times and subsequent actions of one-hand pinching operation in mixed reality technology, the operation of virtual objects is simplified, the problems of cumbersome interaction and reduced reality in the existing technology are solved, and efficient and realistic gesture interaction is achieved .
Patent Information
- Application Number
- CN202410814715.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-08
AI Technical Summary
In mixed reality technology, the existing gesture interaction methods have problems such as cumbersome interaction and inconvenient operation, especially the interaction based on sliding bars reduces the sense of interaction and deviates from the original intention of augmented reality technology.
By continuously performing different times of one-hand pinching operation, and in conjunction with the subsequent actions of the last one-hand pinching operation, different operations are achieved on virtual objects without single-hand switching, simplifying operations, and not relying on other virtual controls in the virtual screen , all rely on user gestures to enhance the sense of reality.
It realizes simple operation of virtual objects, improves interaction efficiency and sense of reality, and solves the problems of gesture behavior conflict and auxiliary control distortion.
Smart Images

Figure CN118915906A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of machine vision and intelligent systems, and in particular to a gesture interaction method, electronic device, storage medium and program product. Background Art
[0002] Mixed Reality (MR) is an augmented reality technology that mixes the real world with virtual objects. In MR technology, virtual objects are rendered based on environmental information such as lighting and shadows in the real world to enhance the realism of virtual objects. In addition, it also supports users to smoothly operate virtual objects through interactive gestures in the real world, further increasing the sense of reality.
[0003] In some MR scenarios, virtual objects are moved by one-hand gestures, rotated by two-hand relative rotation gestures, and scaled by two-hand relative motion gestures. In other MR scenarios, a slider is added to the virtual screen, and the virtual object is moved by sliding gestures on the slider, rotated by one-hand movement gestures, and scaled by two-hand relative motion gestures.
[0004] Virtual objects can be operated through the above-mentioned interactive gestures, but there are problems such as cumbersome interaction and inconvenient operation. In addition, interactive operations based on sliders will reduce the realism of the interaction and deviate from the original intention of MR. Summary of the invention
[0005] Multiple aspects of the present application provide a gesture interaction method, electronic device, storage medium and program product to simplify interactive operations without the need for other virtual controls, relying entirely on user gesture operations to enhance realism.
[0006] An embodiment of the present application provides a gesture interaction method, which is applied to an XR device, the method comprising: displaying a target virtual screen, the target virtual screen including at least one virtual object; responding to N consecutive one-hand pinch operations initiated by a user on the target virtual object in the real world, following a first action of the Nth one-hand pinch operation in a pinch state, and performing a first operation on the target virtual object; responding to M consecutive one-hand pinch operations initiated by the user on the target virtual object in the real world, following a second action of the Mth one-hand pinch operation in a pinch state, and performing a second operation on the target virtual object; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are respectively different from the second action and the second operation; when N and M are the same, the first action and the first operation are respectively the same as the second action and the second operation.
[0007] An embodiment of the present application also provides a gesture interaction method, which is applied to an XR device, and the method includes: displaying a target virtual screen, wherein the target virtual screen includes at least one virtual object; in response to N consecutive one-handed interaction operations initiated by a user on the target virtual object in the real world, following a first action of the Nth one-handed interaction operation in an interaction state, performing a first operation on the target virtual object; in response to M consecutive one-handed interaction operations initiated by the user on the target virtual object in the real world, following a second action of the Mth one-handed interaction operation in the interaction state, performing a second operation on the target virtual object; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are respectively different from the second action and the second operation; when N and M are the same, the first action and the first operation are respectively the same as the second action and the second operation.
[0008] An embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the gesture interaction method provided in the embodiment of the present application.
[0009] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements each step in the gesture interaction method provided in the embodiment of the present application.
[0010] The embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement each step in the gesture interaction method provided in the embodiment of the present application.
[0011] In the embodiment of the present application, different operations are performed on the virtual object by continuously performing one-hand pinch operations for different numbers of times and coordinating the subsequent actions of the last one-hand pinch operation. On this basis, different operations on the same virtual object are accurately distinguished by continuously performing one-hand pinch operations for different numbers of times. There is no need to switch between one hand and two hands, which simplifies the operation. There is no need to rely on other virtual controls in the virtual screen (such as an operation bar). Everything depends on the user's gesture operation, which has a strong sense of reality. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0013] Figure 1 A flowchart of a gesture interaction method provided for an exemplary embodiment of the present application;
[0014] Figure 2A schematic diagram of an operation flow of a moving and rotating operation provided for an exemplary embodiment of the present application;
[0015] Figure 3 A flowchart of another gesture interaction method provided for an exemplary embodiment of the present application;
[0016] Figure 4 A schematic diagram of the structure of a gesture interaction device provided for an exemplary embodiment of the present application;
[0017] Figure 5 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0019] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.
[0020] In an embodiment of the present application, the spatial computing device is a new type of device with a wearable display screen that can sense the surrounding area and has a user interaction interface designed for the user's hands, eyes, and voice. The spatial computing device can be implemented as glasses or other head-mounted display devices. The spatial computing device can use mixed reality technology (MixedReality, MR), augmented reality (AR) or virtual reality (VR) technology to display 3D images. The spatial computing device can also be called an XR device, which is a general term for MR devices, AR devices, and VR devices. For example, when the spatial computing device uses MR technology to display the picture, the spatial computing device can be called an MR device, when the spatial computing device uses AR technology to display the picture, the spatial computing device can be called an AR device, and when the spatial computing device uses VR technology to display the picture, the spatial computing device can be called a VR device.
[0021] In the embodiments of the present application, the structure of the spatial computing device is not limited. For example, the spatial computing device includes: a display screen, various sensors, a camera, etc., and further includes: an audio device (such as headphones or a microphone), a controller (such as a handle or a touchpad), and a battery and charging system, etc. Depending on the different implementation forms of the spatial computing device, the various components included will also be different, and the above are only examples. For example, a head-mounted device may not include components such as a handle or a touchpad. Among them, 1. Display. It can provide a high-resolution visual experience for the spatial computing device and provide a co-immersive experience for the user. 2. Sensors. The sensors in the spatial computing device are crucial for capturing the user's movements, position, and posture. The sensors in the spatial computing device may include but are not limited to: optical sensors, motion sensors, depth sensors, environmental sensors, etc. Among them, optical sensors are often used to track the user's eye movements to achieve eye tracking functions. Eye tracking contributes to more natural and intuitive user interactions, for example, controlling a virtual interface or selecting an object through the user's line of sight. The optical sensor also includes: a camera that can capture images of the real world and accurately superimpose virtual elements on these images. The camera can also be used for environmental recognition, helping the device understand the user's surroundings in order to provide a more intelligent and personalized experience. Motion sensors are used to capture the user's head and hand movements in real time and convert them into corresponding operations in the virtual environment, thereby providing a basis for the interaction between the spatial computing device and the user. For example, the motion sensor captures whether the user initiates a one-hand pinch operation, a one-hand release operation, or whether the user initiates N or M consecutive one-hand pinch operations, etc., where N and M are positive integers greater than or equal to 1. Depth sensors can measure the distance between the user and objects in the real world, which is critical for accurately superimposing virtual objects into the real world. Depth sensors help spatial computing devices understand the user's surroundings by measuring the depth information of the surface of an object and provide a more realistic and natural experience. Environmental sensors include temperature sensors, humidity sensors, and light sensors, which are used to monitor the environmental conditions around the spatial computing device. Environmental sensors help the device automatically adjust parameters such as display brightness and temperature control to provide a more comfortable user experience.
[0022] In an embodiment of the present application, a spatial computing device can display a virtual screen, and the virtual screen includes virtual objects. The virtual screen can be a screen presented by the spatial computing device running various programs. The program can be various applications installed on the spatial computing device (such as various 3D games, 3D shopping software, 3D smart home software, etc.), or it can be a desktop program or other system program of the spatial computing device, etc., which is not limited to this. Furthermore, depending on the virtual screen and its application scenario, the implementation method of the virtual object is also different. For example, in a home scene, virtual objects may include but are not limited to: sofas, tables, chairs, and murals. For another example, in a shopping scene, virtual objects may include but are not limited to: virtual clothing, virtual electronic devices, virtual toys, and virtual sports products. It should be noted that the virtual object can be a three-dimensional virtual object or a two-dimensional virtual object.
[0023] In the embodiments of the present application, several implementation methods are provided for users to perform process operations on virtual objects through interactive gestures, wherein the process operations include moving, rotating and scaling for example, but are not limited to this. The user's interactive operation can be a one-handed interactive operation or a two-handed interactive operation. Among them, whether it is a one-handed interactive operation or a two-handed interactive operation, the interactive operation can include but is not limited to: a pinch operation, a click operation, a palm contact operation, a fist operation, etc. The following is an example of an interactive operation that is a one-handed pinch operation or a two-handed pinch operation. Other interactive operations are the same or similar and will not be repeated here.
[0024] Method A1:
[0025] 1. Move: Perform a one-hand pinch operation (pinch) on the virtual object, and move the hand in the pinching state, and the virtual object moves along with the movement of the pinch point of the one-hand. The one-hand pinch operation can be a pinch operation of the index finger and thumb.
[0026] 2. Rotation: Pinch the virtual object with both hands. For example, both the left and right hands pinch the virtual object. At this time, the two hands in the pinching state begin to rotate relative to each other, and the rotation direction and angle of the virtual object are controlled according to the angle change of the line between the pinching points of the two hands. The rotation direction can be along the X-axis, Y-axis or Z-axis in the world coordinate system. For example, the Z-axis is the direction corresponding to the front of the eyes, the Y-axis is the vertical direction, and the X-axis is the horizontal direction. The rotation angle can be any angle between 0 and 360 degrees.
[0027] 3. Zoom: Pinch the virtual object with both hands. The two hands in the pinching state start to move relative to each other. The zoom multiple is controlled according to the change in the distance between the pinching points of the two hands. For example, if the distance between the pinching points of the two hands after the movement is twice the initial distance, the virtual object is enlarged by 2 times; if the distance between the pinching points of the two hands after the movement is 1 / 3 of the initial distance, the virtual object is reduced by 1 / 3; the initial distance is the distance between the pinching points before the two hands move.
[0028] Among them, the above-mentioned method A1 has problems such as cumbersome interaction and inconvenient operation. For example, the rotation operation and the zoom operation need to be implemented through the interaction of both hands, which is cumbersome and inconvenient. For another example, when both hands perform rotation and zoom operations at the same time, it is difficult to perform a single operation, because when the two hands are actually operated, the distance between the pinch points of the two hands may move at the same time, and the rotation angle may be made, making it difficult to implement a single rotation operation or a single zoom operation.
[0029] Method A2:
[0030] 1. Move: Add a virtual operation bar at the bottom of the virtual object, and move the operation bar by pinching with one hand to make the virtual object move with it.
[0031] 2. Rotation: Pinch the virtual object with one hand, drag the virtual object while pinching, and calculate the direction and distance of the virtual object's rotation based on the direction and distance of the drag.
[0032] 3. Zoom: Pinch the virtual object with both hands. The two hands start to move relative to each other, and the zoom factor is controlled according to the change in the distance between the pinch points of the two hands.
[0033] For detailed introduction of the above-mentioned one-hand pinching operation and two-hand pinching operation, please refer to the aforementioned method A1, which will not be repeated here.
[0034] Among them, in method A2, a virtual operation bar is used to reduce the user's sense of real operation of directly grasping the virtual object. In addition, when performing operations such as plane placement on the virtual object, the operation bar is below the plane, affecting the sense of realism of placement.
[0035] In response to the above technical problems, the embodiment of the present application proposes a method A3, which implements different operations on virtual objects by continuously performing one-hand pinch operations for different numbers of times and coordinating the subsequent actions of the last one-hand pinch operation. On this basis, different operations on the same virtual object are accurately distinguished by continuously performing one-hand pinch operations for different numbers of times. There is no need to switch between one hand and two hands, which simplifies the operation. There is no need to rely on other virtual controls in the virtual screen (such as an operation bar). Everything depends on the user's gesture operation, which has a strong sense of reality.
[0036] The following is a detailed description of method A3 provided in an embodiment of the present application in conjunction with the accompanying drawings.
[0037] Figure 1 This is a flow chart of a gesture interaction method provided by an exemplary embodiment of the present application. The method is applied to an XR device. For an introduction to the XR device, please refer to the aforementioned embodiment and will not be repeated here. Figure 1 As shown, the method includes:
[0038] 101. Displaying a target virtual screen, where the target virtual screen includes at least one virtual object;
[0039] 102. In response to N consecutive one-hand pinch operations initiated by the user on the target virtual object in the real world, following a first action of the Nth one-hand pinch operation in the pinch state, a first operation is performed on the target virtual object;
[0040] 103. In response to M consecutive one-hand pinch operations initiated by a user on a target virtual object in the real world, following a second action of the M-th one-hand pinch operation in the pinch state, perform a second operation on the target virtual object; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are different from the second action and the second operation, respectively; when N and M are the same, the first action and the first operation are the same as the second action and the second operation, respectively.
[0041] In this embodiment, the target virtual screen is implemented differently according to the different implementation methods of the XR device. For example, if the XR device is implemented as an AR device, the target virtual screen is an AR virtual screen displayed in a real physical environment, and the virtual screen includes at least one virtual object; if the XR device is implemented as an MR device, the target virtual screen can be an MR screen, including highly fidelity environmental information redrawn with reference to the surrounding environment, and in addition, the target virtual screen also includes at least one virtual object; if the XR device is implemented as a VR device, the target virtual screen is a completely fictitious VR virtual screen that is convincing enough to be true, and the target virtual screen includes a virtual environment and at least one virtual object.
[0042] The detailed introduction of virtual objects can be found in the above embodiments, which will not be described here. The user can perform interactive operations on a target virtual object in at least one virtual object. The target virtual object is any one of the at least one virtual object.
[0043] In this embodiment, the user can initiate N consecutive one-hand pinch operations on the target virtual object in the real world, and perform the first action in the Nth pinch state. Wherein, N ≥ 1, and N is a positive integer. The real world refers to the user initiating N one-hand pinch operations in the spatial environment, which is different from clicking the display screen to initiate N one-hand pinch operations. Wherein, the first action can be movement, rotation, or other actions.
[0044] In this embodiment, the single hand in the one-hand pinch operation can be the left hand or the right hand. Usually, the same hand is used for N one-hand pinch operations. The number of fingers used in the pinch operation can be two, three or more. For example, the one-hand pinch operation can be a pinch of the index finger and the thumb, or a pinch of the index finger, the middle finger, and the thumb, or a pinch of the thumb and other four fingers, without limitation.
[0045] In this embodiment, the XR device can capture and identify N consecutive single-handed pinch operations initiated by the user on the target virtual object in the real world. Among them, the method of identifying the single-handed pinch operation is not limited. For example, the XR device can collect multiple target images containing the user's hand posture, and adopt a traditional image recognition scheme, or combine artificial intelligence to recognize the collected images containing the user's hand posture. For example, the collected multiple target images are input into a pre-trained action recognition model, and the action recognition model is used to identify whether the multiple target images contain "single-handed pinch operation". If the action recognition model recognizes "single-handed pinch operation" in the multiple target images, it is determined that the user has initiated one single-handed pinch operation. It should be noted that there may be multiple stages of one single-handed pinch operation corresponding to multiple consecutive target images, and the consecutive multiple images correspond to one single-handed pinch operation. The XR device can collect continuous image frames to identify one or more single-handed pinch operations initiated by the user.
[0046] In this embodiment, the XR device can respond to N consecutive one-hand pinch operations initiated by the user, follow the first action of the N-th one-hand pinch operation in the pinch state, and perform a first operation on the target virtual object. The first operation corresponds to the first action. For example, the first action is implemented as a movement operation of the N-th one-hand pinch operation in the pinch state, and the first operation is implemented as a movement operation performed on the target virtual object following the movement operation in the pinch state. Further, the movement distance and direction of the first operation are the same as those of the first action. For another example, the first action is implemented as a rotation operation of the N-th one-hand pinch operation in the pinch state, and the first operation is implemented as a rotation operation performed on the target virtual object following the rotation operation in the pinch state. Further, the rotation direction of the first operation and the first action is the same.
[0047] In this embodiment, the user can initiate M consecutive single-handed pinching operations on the target virtual object in the real world, and perform the second action in the Mth pinching state. Wherein, M ≥ 1, and M is a positive integer. For the introduction of the real world, please refer to the above embodiment, which will not be repeated here. Wherein, the second action can be moving, rotating or other actions. Other actions can be, for example, flipping, shaking, etc.
[0048] In this embodiment, the XR device can respond to M consecutive one-hand pinch operations initiated by the user, and follow the second action of the M-th one-hand pinch operation in the pinch state to perform a second operation on the target virtual object. The second operation corresponds to the second action. For example, the second action is implemented as a movement operation of the M-th one-hand pinch operation in the pinch state, and the second operation is implemented as a movement operation performed on the target virtual object following the movement operation in the pinch state. Further, the movement distance and direction of the second action and the second operation are the same. For another example, the second action is implemented as a rotation operation of the M-th one-hand pinch operation in the pinch state, and the second operation is implemented as a rotation operation performed on the target virtual object following the rotation operation in the pinch state. Further, the second action and the second operation have the same rotation direction.
[0049] In this embodiment, when N and M are different, the first action is different from the second action, and the first operation is different from the second operation. In this case, it means that two different operations are performed on the same target virtual object through a one-hand pinch operation. For example, the target virtual object is first moved through a one-hand pinch operation + a moving action, and then the target virtual object is rotated through a one-hand pinch operation + a rotation action. For another example, the target virtual object is first moved through a one-hand pinch operation + a moving action, and then the target virtual object is scaled through a one-hand pinch operation + a moving action. For another example, the target virtual object is first rotated through a one-hand pinch operation + a rotation action, and then the target virtual object is moved through a one-hand pinch operation + a moving action. It should be noted that when N and M are different, N=2, M=3, N=2, M=1, or N=3, M=5, and there is no limitation on this. The following takes N=1 and M=2 as an example for explanation, and other value combinations of N and M are the same or similar, and are not repeated here.
[0050] For example, N=1, M=2, the first action may be a moving operation of the Nth one-hand pinch operation in the pinch state, and the first operation may be a moving operation of the target virtual object following the moving operation in the pinch state; the second action may be a rotation operation of the Mth one-hand pinch operation in the pinch state, and the second operation may be a rotation operation of the target virtual object following the rotation operation in the pinch state. Wherein, in the case of N=1, M=2, it can be considered that the user first performs a one-hand pinch operation, performs a moving operation in the pinch state corresponding to the one-hand pinch operation, and the VR device responds to the user performing a one-hand pinch operation, and performs a moving operation on the target virtual object according to the moving operation in the pinch state; the user performs two one-hand pinch operations, and performs a rotation operation in the pinch state corresponding to the second one-hand pinch operation, and the VR device responds to the user performing two one-hand pinch operations, and performs a rotation operation on the target virtual object according to the rotation operation performed in the pinch state corresponding to the second one-hand pinch operation.
[0051] Wherein, when N=1, M=2, the first action is implemented as a moving operation, and the second action is implemented as a rotating operation, a moving operation is represented by a one-handed pinching operation, and a rotating operation is represented by a one-handed pinching operation. Wherein, when N is different from M, for the same one-handed interactive operation (e.g., a one-handed pinching operation), different interactive operations (e.g., a moving operation or a rotating operation) for the target virtual object are distinguished according to the number of interactions. During the whole process, the interactive operation is simple and easy to switch, and does not rely on interface components (e.g., an operation bar), and the interactive reality is strong.
[0052] In this embodiment, when N and M are the same, the first action is the same as the second action, and the first operation is the same as the second operation. In this case, it means that the same operation is performed twice on the same target virtual object through a one-hand pinch operation. For example, the target virtual object is first moved through a one-hand pinch operation + a moving action, and then the target virtual object is moved a second time through a one-hand pinch operation + a moving action. For another example, the target virtual object is first scaled through a one-hand pinch operation + a moving action, and then the target virtual object is scaled a second time through a one-hand pinch operation + a moving action. For another example, the target virtual object is first rotated through a one-hand pinch operation + a rotation action, and then the target virtual object is rotated a second time through a one-hand pinch operation + a rotation action. When N and M are the same, the values of N and M are not limited, and the values of N and M can also be N=M=1, N=M=3, or N=M=5, etc. Regarding the case where N and M have the same values, N=M=1 is used as an example for explanation, and the implementation methods of other values are the same or similar, and will not be repeated here.
[0053] For example, in the case of N=M=1, the first action and the second action are both implemented as a movement operation of a one-handed pinch operation in a pinched state, that is, the user initiates two movement operations on the target virtual object through interactive gestures. That is, the VR device responds to a one-handed pinch operation initiated by the user on the target virtual object in the real world, follows the first action (e.g., movement operation) of the one-handed pinch operation in the pinched state, and performs a first operation (e.g., movement operation) on the target virtual object; in response to a one-handed pinch operation initiated by the user on the target virtual object in the real world, follows the second action (e.g., movement operation) of the one-handed pinch operation in the pinched state, and performs a second operation (e.g., movement operation) on the target virtual object.
[0054] In the embodiment of the present application, different operations are performed on the virtual object by continuously performing one-hand pinch operations for different numbers of times and coordinating the subsequent actions of the last one-hand pinch operation. On this basis, different operations on the same virtual object are accurately distinguished by continuously performing one-hand pinch operations for different numbers of times. There is no need to switch between one hand and two hands, which simplifies the operation. There is no need to rely on other virtual controls in the virtual screen (such as an operation bar). Everything depends on the user's gesture operation, which has a strong sense of reality.
[0055] Furthermore, different operations on the same virtual object can be achieved by pinching one hand at different times without switching between one hand and two hands. This allows for more rapid and flexible switching between movement and rotation, thereby improving operational efficiency.
[0056] In the present embodiment, the target virtual object corresponds to state information, and the state information can be understood as the position information of the target virtual object, that is, the position information and direction information. Furthermore, the target virtual object may be static or dynamic. When the target virtual object is dynamic, the target virtual object can automatically change the state information as time changes. For the convenience of distinction and description, when the first operation is performed on the target virtual object, the state of the target virtual object is called the first state, and when the second operation is performed on the target virtual object, the state of the target virtual object is called the second state. If the target virtual object returns to its original state after the first operation is performed on the target virtual object, and then continues to perform the second operation, then the first state and the second state are the same. If after the first operation is performed on the target virtual object, the target virtual object does not return to its original state, and then the second operation is continued to be performed on the target virtual object, then the first state and the second state are different; or, after the first operation is performed on the target virtual object, the second operation is performed on the target virtual object after a certain interval, during which time the target virtual object changes state due to other reasons, for example, the target virtual object changes state by itself, or changes state due to the influence of other virtual objects in the virtual screen, etc., then the first state and the second state are different.
[0057] In an optional embodiment, following the first action of the Nth one-handed pinching operation in the pinching state, the implementation method of performing the first operation on the target virtual object is not limited. Among them, the first action can be a moving operation or a rotating operation. Depending on the implementation method of N, the implementation method of the first action is also different. For example, when N is the first numerical value, the first action is implemented as a moving operation. For another example, when N is the second numerical value, the first action is implemented as a rotating operation. The first numerical value is less than the second numerical value. For example, if the first numerical value is 1, the second numerical value can be 2, 3 or 5, etc. If the first numerical value is 2, the second numerical value is 3, 4 or 6, etc. Specifically, when N is the first numerical value, following the movement of the Nth one-handed pinching operation in the pinching state, the target virtual object is moved; when N is the second numerical value, following the rotation of the Nth one-handed pinching operation in the pinching state, the target virtual object is rotated.
[0058] Optionally, the implementation method of performing a second operation on the target virtual object following the second action of the Mth one-hand pinching operation in the pinching state is not limited. The first action may be a moving operation or a rotation operation. For example, when M is the first numerical value, the first action is implemented as a moving operation, and when N is the second numerical value, the first action is implemented as a rotation operation. The first numerical value is smaller than the second numerical value. For example, if the first numerical value is 1, the second numerical value may be 2, 3 or 5, etc. If the first numerical value is 2, the second numerical value may be 3, 4 or 6, etc. Specifically, when M is the first numerical value, following the movement of the Mth one-hand pinching operation in the pinching state, the target virtual object is moved; when M is the second numerical value, following the rotation of the Mth one-hand pinching operation in the pinching state, the target virtual object is rotated.
[0059] For the above-mentioned implementation, N can be the first value and the second value, M can also be the first value and the second value, and there are many combinations of values of N and M. For example, if N is the first value and M is also the first value, it is equivalent to the user performing two movement operations on the target virtual object. For another example, if N is the first value and M is also the second value, it is equivalent to the user performing one movement operation and one rotation operation on the target virtual object. For another example, if N is the second value and M is also the first value, it is equivalent to the user performing one rotation operation and one movement operation on the target virtual object. For another example, if N is the second value and M is also the second value, it is equivalent to the user performing two rotation operations on the target virtual object.
[0060] Optionally, in addition to performing a move operation and a rotation operation on the target virtual object, a scaling operation may be performed on the target virtual object. The implementation method of performing the scaling operation on the target virtual object is not limited. An exemplary description is given below.
[0061] Example B1, when N is the third value, in response to the user's N consecutive one-hand pinch operations on the target virtual object in the real world, the target virtual object is scaled following the movement of the Nth one-hand pinch operation in the pinch state. The third value is greater than the second value, for example, the third value is 3 and the second value is 2, and for another example, the third value is 5 and the second value is 2, etc.
[0062] Example B2, when M is the third value, in response to the user's M consecutive one-hand pinch operations on the target virtual object in the real world, the target virtual object is scaled following the movement of the Mth one-hand pinch operation in the pinch state. The introduction to the third value can be found in the aforementioned example B1, which will not be repeated here.
[0063] Example B3, a user initiates a two-handed zoom operation on a target virtual object in the real world, and the XR device responds to the two-handed zoom operation initiated by the user on the target virtual object in the real world, and zooms the target virtual object. For example, the two-handed zoom operation is implemented as an operation of pinching and moving both hands relative to each other, and the zoom multiple can be controlled according to the change in the distance of the line between the pinch points of the two hands. For example, the initial distance between the two pinch points corresponding to the two hands is D1, and after the two hands move relative to each other, the distance between the pinch points corresponding to the two hands is D2, and the zoom multiple is expressed as: D2 / D1, if D2 / D1>1, it means that the target virtual object is enlarged, if D2 / D1<1, it means that the target virtual object is reduced, and if D2 / D1=1, it means that the size of the target virtual object remains unchanged.
[0064] Among them, zooming is achieved by two-handed operation, and rotation is achieved by one-handed operation. Different gestures are used to distinguish zooming and rotation, which can accurately distinguish the two operations and improve the user experience.
[0065] The operational convenience provided by the embodiment of the present application refers to the user's operating behavior habits, and the operational convenience decreases in the order of move -> rotate -> zoom. An innovative shortcut gesture is provided for temporarily switching from move to rotation, without the need to click on external controls (such as turning on / off controls), greatly improving the efficiency and experience of XR device operation.
[0066] The following is an exemplary description of the detailed implementation of moving, rotating and scaling operations on the target virtual object.
[0067] Example C1, whether following the movement of the target virtual object in the pinch state by the Nth one-hand pinch operation, or following the movement of the target virtual object in the pinch state by the Mth one-hand pinch operation, the above implementation method of moving the target virtual object is not limited. The following is an illustrative description of the movement of the target virtual object by the Nth one-hand pinch operation in the pinch state. The implementation method of moving the target virtual object by the Mth one-hand pinch operation in the pinch state is the same or similar, and will not be repeated here. For details, please refer to the implementation method of moving the target virtual object by the Nth one-hand pinch operation in the pinch state.
[0068] For example, in response to the movement of the Nth one-handed pinching operation in the pinching state, the movement trajectory of the Nth one-handed pinching operation is generated; with the goal of fixing the pinching point generated by the Nth one-handed pinching operation on the target virtual object, the target virtual object is controlled to move along the movement trajectory. Among them, the pinching point can be a point on the target virtual object. Among them, the implementation method of generating the movement trajectory of the Nth one-handed fitting operation is not limited. For example, the image information of the movement of a single hand in the pinching state in the real space (three-dimensional space) is captured by a camera, and the image information is input into the neural network model. In the neural network model, the position information of the single hand in the pinching state is identified by color segmentation, shape recognition, edge detection or deep learning. After the position information of the single hand is detected, the position of the single hand can be tracked to obtain the movement trajectory of the single hand in continuous images or video frames. Among them, the position tracking can use the optical flow method, which estimates the movement direction and speed of the single hand by comparing the movement of pixels between adjacent image frames, thereby determining the movement trajectory of the single hand based on the movement direction and speed.
[0069] Example C2, whether the target virtual object is rotated following the rotation of the Nth one-hand pinch operation in the pinch state, or the target virtual object is rotated following the rotation of the Mth one-hand pinch operation in the pinch state, the above implementation method of rotating the target virtual object is not limited. The following is an illustrative description of the rotation of the target virtual object by the Nth one-hand pinch operation in the pinch state. The implementation method of rotating the target virtual object by the Mth one-hand pinch operation in the pinch state is the same or similar, and will not be repeated here. For details, please refer to the implementation method of rotating the target virtual object by the Nth one-hand pinch operation in the pinch state.
[0070] For example, in response to the rotation of the Nth one-hand pinch operation in the pinch state, a rotation trajectory of the Nth one-hand pinch operation is generated; the rotation trajectory is decomposed into the rotation angle of the target virtual object on the corresponding coordinate axis, and the target virtual object is controlled to rotate on the corresponding coordinate axis according to the rotation angle.
[0071] The implementation method of generating the rotation trajectory of the Nth single-hand pinch operation is not limited. For example, the XR device collects images or video streams of the user's hands through a camera, and preprocesses the collected images or video streams, which may include noise reduction, color space conversion, contrast enhancement, etc., to improve the accuracy of subsequent processing. In the preprocessed image, the Nth single-hand pinch operation is detected using skin color modeling, edge detection, or deep learning, and the position of the hand in the pinch state is determined. By segmenting the hand area, the hand is separated from the background or other objects to facilitate subsequent trajectory recognition. The features of the hand are extracted, which may include the contour, edge, image moment, image feature vector, and regional histogram features of the hand. Based on feature change information such as the area change of the hand area and the change of the center of mass position, the rotation trajectory of the hand is identified. Among them, for the area change, the size of the hand area in the continuous image frames can be compared to determine whether the rotation operation occurs. For the change of the center of mass position, the direction and speed of the rotation can be determined by calculating the center of mass of the hand area and observing its moving trajectory in the continuous frames.
[0072] Among them, the implementation method of decomposing the rotation trajectory into the rotation angle of the target virtual object on the corresponding coordinate axis is not limited. Taking the world coordinate axis as an example, the world coordinate axis includes the X-axis, the Y-axis and the Z-axis. Among them, the definitions of the X-axis, the Y-axis and the Z-axis are not limited. For example, the Z-axis is the direction corresponding to the front of the eye, the Y-axis is the vertical direction, and the X-axis is the horizontal direction. Among them, the rotation trajectory can be represented by the Euler angle (Roll-Pitch-Yaw), and the barrel roll (Roll) represents the angle of rotation around the Z-axis. Pitch (Pitch) represents the angle of rotation around the Y-axis, that is, the angle in the vertical direction up and down. Yaw (Yaw), also known as the heading angle, rotates around the X-axis and refers to the angle of deflection to the left or right. Among them, the rotation direction can be limited by the rotation angle. For example, the starting angle of rotation around any axis is set to 0 degrees, and rotating 360 degrees around any axis can return to the original position. For another example, the initial rotation angle is set to 0 degrees, rotating 180 degrees in the clockwise direction, and rotating 180 degrees in the counterclockwise direction can traverse all directions of the axis.
[0073] The target virtual object can be controlled to rotate on the corresponding coordinate axis according to the rotation angle. For example, the target virtual object is rotated according to the rotation angles in the X-axis, Y-axis and Z-axis directions respectively. For example, the rotation trajectory indicates that the target virtual object is rotated 90 degrees around the Y-axis direction and 45 degrees around the X-axis direction. Accordingly, the target virtual object is controlled to rotate 90 degrees around the Y-axis direction and 45 degrees around the X-axis direction.
[0074] Example C3 provides an exemplary description of scaling a target virtual object.
[0075] 1) Detailed implementation corresponding to Example B1. In response to the movement of the Nth one-hand pinch operation in the pinch state, a movement trajectory of the Nth one-hand pinch operation is generated; the pinch point generated by the Nth one-hand pinch operation on the target virtual object is used as the zoom starting point, and the target virtual object is zoomed along the movement trajectory.
[0076] Among them, the implementation method of generating the moving trajectory of the Nth one-handed pinching operation can refer to the aforementioned embodiment, which will not be repeated here. Among them, the implementation method of scaling the target virtual object along the moving trajectory is not limited to the pinching point generated by the Nth one-handed pinching operation on the target virtual object as the zoom starting point. For example, the moving trajectory is moving along the pinching point, and it can be defined that moving to the right is zooming in, and moving to the left is zooming out. Determine the zoom-in ratio or the zoom-out ratio based on the moving trajectory. The zoom-in ratio or the zoom-out ratio can be the ratio of the first distance from the center point of the target virtual object to the initial pinching point to the second distance corresponding to the center point to the end point of the moving trajectory. For example, the first distance from the center point of the target virtual object to the initial pinching point is 10cm, and the Nth one-handed pinching operation moves the initial pinching point to the right along the moving trajectory. The distance between the end point of the movement and the center point is 20cm, then the target virtual object is zoomed in 2 times.
[0077] 2) Detailed implementation of Example B2. In response to the movement of the Mth one-hand pinch operation in the pinch state, a movement trajectory of the Mth one-hand pinch operation is generated; the pinch point generated by the Mth one-hand pinch operation on the target virtual object is used as the zoom starting point, and the target virtual object is zoomed along the movement trajectory. The detailed description can be found in the above-mentioned embodiment, which will not be repeated here.
[0078] In an optional embodiment, when a target virtual object is moved, rotated or scaled, the target virtual object can be kept stationary at a current position, which is the position after the move, rotation or scaling operation is performed, and the target virtual object can also be controlled to return from the current position to the original position.
[0079] An exemplary description is given below for a case where a user initiates N consecutive single-handed pinch operations on a target virtual object in the real world.
[0080] For example, when the target virtual object is moved, in response to the release operation of the Nth one-hand pinch operation, the target virtual object is kept stationary at the current position, or in response to the release operation of the Nth one-hand pinch operation, the target virtual object is controlled to return from the current position to the original position. The release operation corresponds to the one-hand pinch operation. For example, the one-hand pinch operation is implemented as: the index finger and the thumb are pinched, and correspondingly, the release operation is the index finger and the thumb are opened. For another example, the one-hand pinch operation is implemented as: the thumb and the other four fingers are pinched, and correspondingly, the release operation is the thumb and the other four fingers are opened.
[0081] For another example, considering that objects in the real world are affected by gravity, the rotation of the target virtual object on the X-axis and Z-axis may cause the target virtual object to appear inconsistent with the gravity scene, affecting the user experience. In addition, considering that the user has a high sensitivity in adjusting the target virtual object manually, it is difficult to restore the rotation of the target virtual object in the X-axis and Z-axis directions to the original posture. The original posture is the initial position and direction, and the original state may also be a state where the plane formed by the X-axis and Z-axis is parallel to the horizontal plane. Based on this, when the target virtual object is rotated, in response to the release operation of the Nth one-handed pinch operation, the current posture of the target virtual object on the X-axis and Z-axis is restored to the original posture, and the current posture on the Y-axis is maintained.
[0082] For another example, when scaling the target virtual object, in response to the Nth release operation of the one-hand pinch operation, the target virtual object is kept in the current scaled state, or, in response to the Nth release operation of the one-hand pinch operation, the target virtual object is controlled to restore from the current scaled state to the original state.
[0083] An exemplary description is given below for a case where a user initiates M consecutive single-handed pinch operations on a target virtual object in the real world.
[0084] For example, when the target virtual object is moved, in response to the Mth release operation of the one-hand pinch operation, the target virtual object is kept stationary at the current position, or, in response to the Mth release operation of the one-hand pinch operation, the target virtual object is controlled to return from the current position to the original position.
[0085] For another example, when the target virtual object is rotated, in response to the release operation of the Mth single-handed pinch operation, the current posture of the target virtual object on the X-axis and Z-axis is restored to the original posture, and the current posture on the Y-axis is maintained.
[0086] For another example, when scaling the target virtual object, in response to the Mth release operation of the one-hand pinch operation, the target virtual object is kept in the current scaled state, or, in response to the Mth release operation of the one-hand pinch operation, the target virtual object is controlled to restore from the current scaled state to the original state.
[0087] Among them, the detailed introduction of this embodiment can be found in the aforementioned implementation manner, which will not be repeated here.
[0088] It should be noted that when the target virtual object is rotated, in response to the release operation of the Nth or Mth one-hand pinch operation, the target virtual object returns to a movable state. The next time the user initiates a one-hand pinch operation, the XR device starts to accumulate the number of one-hand pinch operations again.
[0089] In an optional embodiment, a video screen of a user initiating N or M consecutive one-handed pinching operations on a target virtual object in the real world is collected; a one-handed pinching operation is identified based on the video screen, and when two adjacent one-handed pinching operations are identified, it is determined whether the time interval between the two adjacent one-handed pinching operations is within a set time interval range; wherein the set time interval is not limited, for example, the set time interval can be 400ms, 500ms or 600ms, etc.; if the time interval between the two adjacent one-handed pinching operations is within the set time interval range, it is determined that the two adjacent one-handed pinching operations belong to the same continuous one-handed pinching operation process. For example, a user initiates two consecutive one-handed pinching operations on a target virtual object in the real world, and if the time interval between the two one-handed pinching operations is greater than the set time interval (e.g., 400ms), the two one-handed pinching operations are regarded as the same continuous one-handed pinching operation process, and the target virtual object can be rotated following the rotation operation of the second one-handed pinching operation in the pinching state. If the time interval between two one-hand pinch operations is less than or equal to the set time interval (e.g., 400ms), the two one-hand pinch operations are considered to be independent one-hand pinch operations, and the rotation operation of the second one-hand pinch operation in the pinch state is considered to be a move operation, and the move operation is performed on the target virtual object following the move operation.
[0090] In an optional embodiment, the target virtual screen includes at least one virtual object, and the implementation method of how to determine which virtual object to initiate the one-hand pinch operation is not limited.
[0091] For example, eye movement selection of virtual objects is combined with manual control of virtual objects. In response to the user's eye movement operation, the user's gaze point on the target virtual screen is determined; based on the positional relationship between the gaze point and at least one virtual object, the target virtual object is determined. For example, the distance between the virtual object and the gaze point is determined, and the virtual object closest to the gaze point is used as the target virtual object. For another example, the virtual object that overlaps with the gaze point is used as the target virtual object, that is, the gaze point falls on the target virtual object. When the target virtual object is determined, the user can initiate N or M single-handed pinch operations on the target virtual object in the real world.
[0092] Among them, the optical sensor on the XR device can track the user's eye movement to determine the user's gaze point on the target virtual screen, and the motion sensor on the XR device can capture the user's hand movements to determine whether the user initiates a one-hand pinch operation. For detailed introduction, please refer to the aforementioned embodiment, which will not be repeated here.
[0093] As another example, manual selection of virtual objects is combined with manual control of virtual objects. For example, in response to a user's one-hand selection operation, such as a one-hand pinch operation, the target virtual object selected by the user is determined based on the position information of the pinch point of the user's one-hand pinch operation. For example, the virtual object closest to the pinch point is used as the target virtual object, or the virtual object that overlaps with the pinch point is used as the target virtual object. The user can initiate N or M consecutive one-hand pinch operations, and the XR device can capture N consecutive one-hand pinch operations through a motion sensor or camera, and follow the first action of the Nth one-hand pinch operation in the pinch state, and perform a first operation on the target virtual object; or, the XR device can capture M consecutive one-hand pinch operations through a motion sensor or camera, and follow the second action of the Mth one-hand pinch operation in the pinch state, and perform a second operation on the target virtual object.
[0094] For another example, the XR device can collect the user's action pictures in the real world, including but not limited to: the user's hand movements, body movements, and eye movements, etc.; based on the action pictures collected in the real world, determine the user's hand position or gaze point position, and determine the target virtual object accordingly. For the detailed implementation of the hand position or gaze point, and the determination of the target virtual object, please refer to the above embodiment, which will not be repeated here.
[0095] Scenario-based implementation example:
[0096] The following takes the user's eye movement operation to select the target virtual object, N=1, M=2 as an example to illustrate the user-initiated movement and rotation operations.
[0097] like Figure 2As shown, the user gazes at the target virtual object in the real world, and the XR device responds to the gaze operation and takes the virtual object overlapping with the gaze point as the target virtual object; responds to a single-handed pinch operation initiated for the target virtual object and moves in the pinch state, determines the movement trajectory, controls the target virtual object to move along the movement trajectory, and responds to the release operation of the single-handed pinch operation to keep the target virtual object stationary at the current position. The user gazes at the target virtual object again in the real world, and the XR device responds to the gaze operation and determines the target virtual object; responds to a single-handed pinch operation initiated for the target virtual object, starts timing, and generates a rotation trajectory in response to detecting a second single-handed pinch operation within a set time interval (such as 400ms) and rotating in the pinch state; determines the rotation angle along each coordinate axis based on the rotation trajectory, and controls the target virtual object to rotate on the corresponding coordinate axis according to the rotation angle of each coordinate axis; responds to the release operation of the second single-handed pinch operation, restores the current posture of the target virtual object on the X-axis and Z-axis to the original posture, and maintains the current posture on the Y-axis; keeps the target virtual object stationary.
[0098] The three-dimensional gesture interaction method provided in this solution refers to user behavior habits and quickly switches between movement and rotation operations through the number of original one-hand pinch operations. It can switch from movement operation to rotation in one operation, and then return to movement operation from rotation operation, thereby maximizing the efficiency of interactive operations and effectively solving problems such as gesture behavior conflicts and distortion of auxiliary controls.
[0099] In addition to the above-mentioned gesture interaction method, another gesture interaction method is also provided in the embodiment of the present application. The gesture interaction method is applied to the XR device. Figure 1 The difference between the gesture interaction methods shown is that the gesture interaction operations are different. For detailed description, please refer to the subsequent embodiments.
[0100] like Figure 3 The method shown includes:
[0101] 301. Displaying a target virtual screen, where the target virtual screen includes at least one virtual object;
[0102] 302. In response to N consecutive one-hand interaction operations initiated by the user on the target virtual object in the real world, following a first action of the Nth one-hand interaction operation in the interaction state, perform a first operation on the target virtual object;
[0103] 303. In response to M consecutive one-handed interaction operations initiated by the user on the target virtual object in the real world, following a second action of the M-th one-handed interaction operation in the interaction state, perform a second operation on the target virtual object; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are different from the second action and the second operation, respectively; when N and M are the same, the first action and the first operation are the same as the second action and the second operation, respectively.
[0104] In this embodiment, the one-handed interactive operation may be any interactive operation that can be performed with one hand. Preferably, the one-handed interactive operation is implemented as an operation that is in contact with the target virtual object, such as a one-handed fist operation or a one-handed lifting operation, etc. Of course, the one-handed interactive operation may also be a non-contact operation, such as a close-range hook finger or sliding gesture, push-pull gesture, etc., as long as it can be determined which virtual object the non-contact one-handed interactive operation is performed on.
[0105] In this embodiment, after the one-hand interaction operation, different interaction actions (such as the first action or the second action) are connected, and accordingly, different operations can be performed on the target virtual object.
[0106] in, Figure 1 The method shown is described by taking the one-hand interaction operation implemented as a one-hand pinch operation as an example. Figure 2 In the illustrated embodiment, the detailed implementation of other one-hand interactive operations is the same or similar to the introduction of the one-hand pinch operation. For details, see Figure 1 The embodiments shown will not be described in detail later.
[0107] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0108] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.
[0109] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.
[0110] Figure 4 The following is a schematic diagram of a gesture interaction device provided by an exemplary embodiment of the present application. Figure 4 As shown, the device includes: a display module 41 and an operation module 42.
[0111] A display module 41, used for displaying a target virtual screen, wherein the target virtual screen includes at least one virtual object;
[0112] An operation module 42 is used to respond to N consecutive single-hand pinch operations initiated by the user on the target virtual object in the real world, and perform a first operation on the target virtual object following a first action of the Nth single-hand pinch operation in the pinch state;
[0113] The operation module 42 is also used to respond to M consecutive one-hand pinch operations initiated by the user on the target virtual object in the real world, and follow the second action of the Mth one-hand pinch operation in the pinching state to perform a second operation on the target virtual object; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are different from the second action and the second operation, respectively; when N and M are the same, the first action and the first operation are the same as the second action and the second operation, respectively.
[0114] In an optional embodiment, the operation module is specifically used to: when N is the first value, follow the movement of the Nth one-hand pinching operation in the pinching state to move the target virtual object; when N is the second value, follow the rotation of the Nth one-hand pinching operation in the pinching state to rotate the target virtual object; wherein the first value is smaller than the second value.
[0115] In an optional embodiment, the operation module is specifically used to: when M is the first value, follow the movement of the Mth one-hand pinching operation in the pinching state to move the target virtual object; when M is the second value, follow the rotation of the Mth one-hand pinching operation in the pinching state to rotate the target virtual object.
[0116] Optionally, the device also includes a zoom module; the zoom module is used to zoom the target virtual object following the movement of the Nth or Mth one-handed pinching operation in the pinching state when N or M is the third value; wherein the third value is greater than the second value; or to zoom the target virtual object in response to a two-handed zooming operation initiated by the user on the target virtual object in the real world.
[0117] Optionally, the operation module is specifically used to: generate a movement trajectory of the Nth or Mth one-hand pinch operation in response to the movement of the Nth or Mth one-hand pinch operation in the pinch state; control the target virtual object to move along the movement trajectory with the pinch point generated by the Nth or Mth one-hand pinch operation on the target virtual object fixed.
[0118] Optionally, the operation module is specifically used to: generate a rotation trajectory of the Nth or Mth one-hand pinch operation in response to the rotation of the Nth or Mth one-hand pinch operation in the pinch state; decompose the rotation trajectory into the rotation angle of the target virtual object on the corresponding coordinate axis, and control the rotation of the target virtual object on the corresponding coordinate axis according to the rotation angle.
[0119] Optionally, the zoom module is specifically used to: generate a movement trajectory of the Nth or Mth one-hand pinch operation in response to the movement of the Nth or Mth one-hand pinch operation in the pinch state; use the pinch point generated by the Nth or Mth one-hand pinch operation on the target virtual object as the zoom starting point, and zoom the target virtual object along the movement trajectory.
[0120] In an optional embodiment, the device also includes: a processing module; a processing module, which is used to respond to the release operation of the Nth or Mth one-hand pinch operation when the target virtual object is moved, keep the target virtual object stationary at the current position, or control the target virtual object to return from the current position to the original position; respond to the release operation of the Nth or Mth one-hand pinch operation when the target virtual object is rotated, restore the current posture of the target virtual object on the X-axis and Z-axis to the original posture, and maintain the current posture on the Y-axis; respond to the release operation of the Nth or Mth one-hand pinch operation when the target virtual object is scaled, keep the target virtual object in the current scaled state, or control the target virtual object to be restored from the current scaled state to the original state.
[0121] In an optional embodiment, the first value is 1 and the second value is 2.
[0122] In an optional embodiment, the device also includes: a collection module, a judgment module and a determination module; the collection module is used to collect video images of a user initiating N or M consecutive one-hand pinch operations on a target virtual object in the real world; the judgment module is used to identify the one-hand pinch operation based on the video image, and when two adjacent one-hand pinch operations are identified, determine whether the time interval between the two adjacent one-hand pinch operations is within a set time interval range; the determination module is used to determine that the two adjacent one-hand pinch operations belong to the same continuous one-hand pinch operation process if the time interval between the two adjacent one-hand pinch operations is within the set time interval range.
[0123] In an optional embodiment, before responding to the user initiating N or M consecutive single-hand pinch operations on the target virtual object in the real world, the determination module is also used to: respond to the user's eye movement operation to determine the user's gaze point on the target virtual screen; determine the target virtual object based on the positional relationship between the gaze point and at least one virtual object.
[0124] The embodiment of the present application also provides another gesture interaction device, which corresponds to an XR device, and includes: a display module and an operation module;
[0125] A display module, used for displaying a target virtual screen, wherein the target virtual screen includes at least one virtual object;
[0126] An operation module, configured to respond to N consecutive one-hand interaction operations initiated by a user on a target virtual object in the real world, and perform a first operation on the target virtual object following a first action of the Nth one-hand interaction operation in an interaction state;
[0127] The operation module is also used to respond to M consecutive one-handed interaction operations initiated by the user on the target virtual object in the real world, and perform a second operation on the target virtual object following the second action of the M-th one-handed interaction operation in the interaction state; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are respectively different from the second action and the second operation; when N and M are the same, the first action and the first operation are respectively the same as the second action and the second operation.
[0128] The detailed implementation and beneficial effects of each step in the above-mentioned device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0129] Figure 5 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application, the electronic device corresponds to an XR device, such as Figure 5 As shown, the device includes: a memory 54 and a processor 55.
[0130] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.
[0131] The memory 54 can be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0132] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54, so as to: display a target virtual screen, which includes at least one virtual object; respond to N consecutive one-hand pinch operations initiated by a user on the target virtual object in the real world, follow the first action of the Nth one-hand pinch operation in the pinching state, and perform a first operation on the target virtual object; respond to M consecutive one-hand pinch operations initiated by the user on the target virtual object in the real world, follow the second action of the Mth one-hand pinch operation in the pinching state, and perform a second operation on the target virtual object; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are respectively different from the second action and the second operation; when N and M are the same, the first action and the first operation are respectively the same as the second action and the second operation.
[0133] In an optional embodiment, when the processor 55 follows the first action of the Nth one-hand pinching operation in the pinching state and performs the first operation on the target virtual object, it is specifically used to: when N is the first value, follow the movement of the Nth one-hand pinching operation in the pinching state to move the target virtual object; when N is the second value, follow the rotation of the Nth one-hand pinching operation in the pinching state to rotate the target virtual object; wherein the first value is smaller than the second value.
[0134] In an optional embodiment, when the processor 55 performs a second operation on the target virtual object following the second action of the Mth one-hand pinching operation in the pinching state, it is specifically used to: when M is the first value, follow the movement of the Mth one-hand pinching operation in the pinching state to move the target virtual object; when M is the second value, follow the rotation of the Mth one-hand pinching operation in the pinching state to rotate the target virtual object.
[0135] Optionally, the processor 55 is further used to: when N or M is a third value, scale the target virtual object following the movement of the Nth or Mth one-handed pinching operation in the pinching state; wherein the third value is greater than the second value; or scale the target virtual object in response to a two-handed pinching operation initiated by the user on the target virtual object in the real world.
[0136] Optionally, when the processor 55 follows the movement of the N or Mth one-hand pinch operation in the pinch state to move the target virtual object, it is specifically used to: generate a movement trajectory of the N or Mth one-hand pinch operation in response to the movement of the N or Mth one-hand pinch operation in the pinch state; and control the target virtual object to move along the movement trajectory with the pinch point generated by the N or Mth one-hand pinch operation on the target virtual object fixed.
[0137] Optionally, when the processor 55 rotates the target virtual object following the rotation of the Nth or Mth one-hand pinch operation in the pinch state, it is specifically used to: generate a rotation trajectory of the Nth or Mth one-hand pinch operation in response to the rotation of the Nth or Mth one-hand pinch operation in the pinch state; decompose the rotation trajectory into rotation angles of the target virtual object on the corresponding coordinate axes, and control the rotation of the target virtual object on the corresponding coordinate axes according to the rotation angles.
[0138] Optionally, when the processor 55 follows the movement of the Nth or Mth one-hand pinch operation in the pinch state to scale the target virtual object, it is specifically used to: generate a movement trajectory of the Nth or Mth one-hand pinch operation in response to the movement of the Nth or Mth one-hand pinch operation in the pinch state; use the pinch point generated on the target virtual object by the Nth or Mth one-hand pinch operation as the scaling starting point, and scale the target virtual object along the movement trajectory.
[0139] Further optionally, the processor 55 is also used to: in the case of moving the target virtual object, respond to the release operation of the Nth or Mth one-hand pinch operation, keep the target virtual object stationary at the current position, or control the target virtual object to return from the current position to the original position; in the case of rotating the target virtual object, respond to the release operation of the Nth or Mth one-hand pinch operation, restore the current posture of the target virtual object on the X-axis and Z-axis to the original posture, and maintain the current posture on the Y-axis; in the case of scaling the target virtual object, respond to the release operation of the Nth or Mth one-hand pinch operation, keep the target virtual object in the current scaling state, or control the target virtual object to recover from the current scaling state to the original state.
[0140] In an optional embodiment, the first value is 1 and the second value is 2.
[0141] In an optional embodiment, the processor 55 is also used to: collect video footage of a user initiating N or M consecutive one-hand pinch operations on a target virtual object in the real world; identify the one-hand pinch operation based on the video footage, and when two adjacent one-hand pinch operations are identified, determine whether the time interval between the two adjacent one-hand pinch operations is within a set time interval range; if the time interval between the two adjacent one-hand pinch operations is within the set time interval range, determine that the two adjacent one-hand pinch operations belong to the same continuous one-hand pinch operation process.
[0142] In an optional embodiment, before responding to the user initiating N or M consecutive single-hand pinching operations on the target virtual object in the real world, the processor 55 is also used to: respond to the user's eye movement operation to determine the user's gaze point on the target virtual screen; and determine the target virtual object based on the positional relationship between the gaze point and at least one virtual object.
[0143] Regarding the embodiments of the present application Figure 5 The detailed implementation and beneficial effects of each step in the electronic device have been described in detail in the aforementioned embodiments, and will not be elaborated here.
[0144] Further, if Figure 5 As shown, the electronic device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.
[0145] The embodiment of the present application also provides an electronic device, the implementation structure of the electronic device is Figure 5 The implementation structure of the electronic device shown is the same or similar, and can refer to Figure 5 The electronic device provided in this embodiment is implemented as follows: Figure 5The difference between the electronic devices in the illustrated embodiments mainly lies in that the functions implemented by the processor executing the computer program stored in the memory are different. For the electronic device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: display a target virtual screen, which includes at least one virtual object; respond to N consecutive one-handed interaction operations initiated by the user on the target virtual object in the real world, follow the first action of the Nth one-handed interaction operation in the interaction state, and perform a first operation on the target virtual object; respond to M consecutive one-handed interaction operations initiated by the user on the target virtual object in the real world, follow the second action of the Mth one-handed interaction operation in the interaction state, and perform a second operation on the target virtual object; wherein N and M are positive integers, and when N and M are different, the first action and the first operation are respectively different from the second action and the second operation; when N and M are the same, the first action and the first operation are respectively the same as the second action and the second operation.
[0146] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0147] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by a processor in the above method embodiment.
[0148] Accordingly, the computer program product includes a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement each step that can be executed by the processor in the above method embodiment.
[0149] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0150] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.
[0151] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0152] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.
[0153] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.
[0154] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.
[0155] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0158] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.
[0159] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0160] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0161] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0162] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A gesture interaction method, applied to an XR device, characterized in that: The method comprises: Displaying a target virtual screen, wherein the target virtual screen includes at least one virtual object; In response to N consecutive one-hand pinch operations initiated by a user on a target virtual object in the real world, following a first action of the Nth one-hand pinch operation in a pinch state, a first operation is performed on the target virtual object; In response to M consecutive one-hand pinch operations initiated by the user on the target virtual object in the real world, following a second action in the pinch state of the Mth one-hand pinch operation, a second operation is performed on the target virtual object; Wherein, N and M are positive integers. When N and M are different, the first action and the first operation are different from the second action and the second operation respectively; when N and M are the same, the first action and the first operation are the same as the second action and the second operation respectively.
2. The method according to claim 1, characterized in that Following the first action of the Nth one-hand pinching operation in the pinching state, performing a first operation on the target virtual object includes: When N is the first value, following the movement of the Nth single-hand pinching operation in the pinching state, the target virtual object is moved; When N is the second value, following the rotation of the Nth single-hand pinching operation in the pinching state, the target virtual object is rotated; The first value is smaller than the second value.
3. The method according to claim 2, characterized in that Following the second action of the Mth one-hand pinching operation in the pinching state, performing a second operation on the target virtual object includes: When M is the first value, following the movement of the Mth single-hand pinching operation in the pinching state, the target virtual object is moved; When M is the second value, the target virtual object is rotated following the rotation of the Mth single-hand pinching operation in the pinching state.
4. The method according to claim 3, characterized in that: Also includes: When N or M is a third value, scaling the target virtual object following the movement of the Nth or Mth single-handed pinching operation in the pinching state; wherein the third value is greater than the second value; or In response to a two-handed scaling operation initiated by the user on the target virtual object in the real world, scaling the target virtual object.
5. The method according to claim 3, characterized in that: Following the movement of the Nth or Mth single-hand pinching operation in the pinching state, moving the target virtual object includes: In response to the movement of the Nth or Mth one-hand pinch operation in the pinch state, generating a movement trajectory of the Nth or Mth one-hand pinch operation; With the goal of keeping the pinch point generated by the Nth or Mth single-hand pinch operation on the target virtual object fixed, the target virtual object is controlled to move along the movement trajectory.
6. The method according to claim 3, characterized in that: Following the rotation of the Nth or Mth single-hand pinching operation in the pinching state, rotating the target virtual object includes: In response to the rotation of the Nth or Mth one-hand pinch operation in the pinch state, generating a rotation trajectory of the Nth or Mth one-hand pinch operation; The rotation trajectory is decomposed into a rotation angle of the target virtual object on the corresponding coordinate axis, and the target virtual object is controlled to rotate on the corresponding coordinate axis according to the rotation angle.
7. The method according to claim 4, characterized in that Following the movement of the Nth or Mth single-hand pinching operation in the pinching state, scaling the target virtual object includes: In response to the movement of the Nth or Mth one-hand pinch operation in the pinch state, generating a movement trajectory of the Nth or Mth one-hand pinch operation; The target virtual object is scaled along the movement trajectory using a pinch point generated by the Nth or Mth single-hand pinch operation on the target virtual object as a zoom starting point.
8. The method according to claim 4, characterized in that The method further comprises: When the target virtual object is moved, in response to a release operation of the Nth or Mth one-hand pinch operation, the target virtual object is kept stationary at a current position, or the target virtual object is controlled to return from the current position to an original position; When the target virtual object is rotated, in response to a release operation of the Nth or Mth single-handed pinch operation, the current posture of the target virtual object on the X-axis and the Z-axis is restored to the original posture, and the current posture on the Y-axis is maintained; When the target virtual object is scaled, in response to a release operation of the Nth or Mth single-hand pinch operation, the target virtual object is kept in a current scaled state, or the target virtual object is controlled to be restored from the current scaled state to an original state.
9. The method according to claim 2, characterized in that: The first number value is 1, and the second number value is 2.
10. The method according to any one of claims 1 to 9, characterized in that: Also includes: Capturing a video screen of the user initiating N or M consecutive single-handed pinching operations on a target virtual object in the real world; recognizing a one-hand pinch operation based on the video image, and when two adjacent one-hand pinch operations are recognized, determining whether a time interval between the two adjacent one-hand pinch operations is within a set time interval range; If the time interval between the two adjacent one-hand pinching operations is within the set time interval range, it is determined that the two adjacent one-hand pinching operations belong to the same continuous one-hand pinching operation process.
11. The method according to any one of claims 1 to 9, characterized in that: Before responding to the user initiating N or M consecutive single-hand pinch operations on the target virtual object in the real world, the method further includes: In response to the eye movement operation of the user, determining the gaze point of the user on the target virtual screen; The target virtual object is determined according to a positional relationship between the gaze point and the at least one virtual object.
12. A gesture interaction method, applied to an XR device, characterized in that: The method comprises: Displaying a target virtual screen, wherein the target virtual screen includes at least one virtual object; In response to N consecutive one-handed interaction operations initiated by a user on a target virtual object in the real world, following a first action of the Nth one-handed interaction operation in an interaction state, a first operation is performed on the target virtual object; In response to M consecutive one-handed interaction operations initiated by the user on the target virtual object in the real world, following a second action of the M-th one-handed interaction operation in the interaction state, a second operation is performed on the target virtual object; Wherein, N and M are positive integers. When N and M are different, the first action and the first operation are different from the second action and the second operation respectively; when N and M are the same, the first action and the first operation are the same as the second action and the second operation respectively.
13. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method of any one of claims 1 to 11 and claim 12.
14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 1 to 11 and claim 12.
15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps in the method of any one of claims 1 to 11 and claim 12.
