Interaction method, device, equipment, medium and product
By mapping the user's gestures into the operation of the virtual mouse and controlling the mouse cursor on the screen to perform corresponding operations, the problem of complex eye-hand interaction process in the extended real environment is solved, and the user's interactive experience is improved.
Patent Information
- Application Number
- CN202510295279.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-12
- Publication Date
- 2025-06-27
AI Technical Summary
In an extended reality environment, the eye-hand interaction process is complex and is greatly affected by the differences in user eye characteristics, resulting in poor user interaction experience.
By capturing the user's gestures, mapping them to the target manipulation behavior of the virtual mouse, and controlling the mouse cursor displayed on the screen to perform corresponding operations, simplifying the interaction process.
It realizes simplified interaction with XR devices, improves user interaction experience, and avoids interaction complexity problems caused by differences in eye characteristics.
Smart Images
Figure CN120215710A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of human-computer interaction, and particularly relates to an interaction method, device, equipment, medium and product. Background Art
[0002] In an Extended Reality (XR) environment, eye-hand interaction has gradually become a trend, and there are more and more scenarios of collaborative interaction using eye tracking and gesture tracking technologies. However, since eye movement conditions vary among users, for example, problems such as myopia and non-myopia, and pupil distance differences can lead to complex eye movement conditions, which in turn make the process of interacting with an XR device using eye and hand rather complicated. Summary of the Invention
[0003] The purpose of the embodiments of this application is to provide an interaction method, device, equipment, medium and product, which can solve the problem of complex process in the related technology when interacting with an XR device.
[0004] In a first aspect, the embodiments of this application provide an interaction method, which is applied to an extended reality device. The method includes:
[0005] When a first gesture is captured, map the first gesture to a target manipulation behavior of the user on a virtual mouse;
[0006] According to the target manipulation behavior, control the mouse cursor displayed on the screen to perform an operation corresponding to the target manipulation behavior.
[0007] In a second aspect, the embodiments of this application provide an interaction device, which is applied to an extended reality device. The device includes:
[0008] A mapping module, configured to map a first gesture to a target manipulation behavior of the user on a virtual mouse when the first gesture is captured;
[0009] A control module, configured to control the mouse cursor displayed on the screen to perform an operation corresponding to the target manipulation behavior according to the target manipulation behavior.
[0010] In a third aspect, the embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.
[0011] In a fourth aspect, the embodiments of this application provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.
[0012] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run programs or instructions to implement the steps of the method described in the first aspect.
[0013] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.
[0014] In an embodiment of the present application, when a first gesture is captured, the first gesture is mapped to a target manipulation behavior of the user on the virtual mouse; according to the target manipulation behavior, the mouse cursor displayed on the screen is controlled to perform an operation corresponding to the target manipulation behavior. That is, the embodiment of the present application uses gestures to simulate the target manipulation behavior of the virtual mouse, realizes the interaction with the XR device, simplifies the interaction process, and improves the user's interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a flowchart of an interaction method provided by an embodiment of the present application;
[0016] Figure 2 is a flowchart of another interaction method provided by an embodiment of the present application;
[0017] Figure 3 is a schematic diagram of a first gesture provided by an embodiment of the present application;
[0018] Figure 4 is a schematic diagram of another first gesture provided by an embodiment of the present application;
[0019] Figure 5 is a flowchart of another interaction method provided by an embodiment of the present application;
[0020] Figure 6 is a schematic diagram of simulating the left button of a virtual mouse provided by an embodiment of the present application;
[0021] Figure 7 is a schematic diagram of simulating the right button of a virtual mouse provided by an embodiment of the present application;
[0022] Figure 8 is a schematic diagram of the contact between the right index finger and the left palm provided by an embodiment of the present application;
[0023] Figure 9 is a schematic diagram of the contact between the right index finger and the left palm and the index finger moving from position A to position B provided by an embodiment of the present application;
[0024] Figure 10 is a schematic diagram of object selection provided by an embodiment of the present application;
[0025] Figure 11 Flow chart of another interaction method provided by an embodiment of the present application;
[0026] Figure 12 Schematic diagram of simulating a virtual mouse scroll wheel key provided by an embodiment of the present application;
[0027] Figure 13 Schematic diagram of a screen displaying multiple windows provided by an embodiment of the present application;
[0028] Figure 14 Flow chart of another interaction method provided by an embodiment of the present application;
[0029] Figure 15 Schematic diagram of the pose of a palm provided by an embodiment of the present application;
[0030] Figure 16 Schematic diagram of a mouse cursor in a free state provided by an embodiment of the present application;
[0031] Figure 17 Schematic diagram of the structure of an interaction device provided by an embodiment of the present application;
[0032] Figure 18 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application;
[0033] Figure 19 Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners
[0034] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art fall within the protection scope of the present application.
[0035] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.
[0036] When interacting with an XR device, currently, the interaction mainly relies on the coordination of eye movements and hand movements with the XR device. However, due to different eye characteristics of different users, such as myopia, non-myopia, and pupil distance differences, etc., all will affect the interaction results between the user and the XR device. To avoid the influence of eye characteristics on the interaction results, when using the eye-hand coordination to interact with the XR device, a relatively complex interaction process often needs to be set up, resulting in a poor interaction experience for the user.
[0037] Moreover, considering that users are already accustomed to operating the computer screen with a mouse, on this basis, the embodiments of the present application provide an interaction method, which uses gesture simulation to perform target manipulation behaviors on a virtual mouse to achieve interaction with the XR device, simplifies the interaction process, and improves the user's interaction experience.
[0038] Next, in conjunction with the accompanying drawings, through specific embodiments and their application scenarios, the interaction methods, devices, equipment, media, and products provided by the embodiments of the present application will be described in detail.
[0039] Figure 1 FIG. is a flowchart of an interaction method provided by an embodiment of the present application. This interaction method can be applied to an XR device, and the XR device can include, for example, but not limited to, an Augmented Reality (AR) device, a Virtual Reality (VR) device, a Mixed Reality (MR) device, etc. In some embodiments, the XR device can include a head-mounted device. As Figure 1 shown, this interaction method can include the following steps:
[0040] S110. When a first gesture is captured, map the first gesture to a target manipulation behavior of the user on the virtual mouse.
[0041] S120. According to the target manipulation behavior, control the mouse cursor displayed on the screen to perform an operation corresponding to the target manipulation behavior.
[0042] The embodiments of the present application use gesture simulation to perform target manipulation behaviors on a virtual mouse to achieve interaction with the XR device, simplifies the interaction process, and improves the user's interaction experience.
[0043] The above steps will be described in detail below, as specifically shown below:
[0044] In S110, the first gesture is used to simulate the target manipulation behavior of the user on the virtual mouse, and the virtual mouse is used to achieve the interaction between the user and the XR device. That is, the embodiments of the present application can use gestures to simulate a virtual mouse, and then use the virtual mouse to perform different operations on the screen of the XR device, simplifying the interaction process between the user and the XR device.
[0045] Exemplarily, the first gesture can be static or dynamic. The first gesture can be a gesture formed by a single hand, for example, a gesture formed by the left hand or a gesture formed by the right hand. That is, the embodiments of the present application can use a single hand to simulate a virtual mouse and interact with the XR device.
[0046] Exemplarily, the first gesture can also be a gesture formed by the cooperation of two hands. That is, the embodiments of the present application can use the gesture formed by the cooperation of the left and right hands to simulate the target manipulation behavior of the user on the virtual mouse and achieve interaction with the XR device.
[0047] That is, the embodiments of the present application can use different gestures to simulate the manipulation behavior of the user on the virtual mouse and interact with the XR device, without the need to cooperate with eye movements, thereby simplifying the interaction process.
[0048] Different gestures can simulate different manipulation behaviors of the user on the virtual mouse, and the mapping relationship between the gesture and the manipulation behavior can be established and saved according to requirements.
[0049] Exemplarily, the XR device can capture the user's gesture through a camera and identify the gesture captured by the camera. When it is determined that the captured gesture is the first gesture, the first gesture can be mapped to the target manipulation behavior of the user on the virtual mouse in combination with the saved mapping relationship between the gesture and the manipulation behavior.
[0050] In S120, the mouse cursor here is the mouse cursor of the virtual mouse, also known as the mouse pointer. Exemplarily, the mouse cursor can be displayed on the screen when the XR device captures the first gesture; it can also be displayed on the screen when the XR device captures the second gesture; it can also be displayed on the screen when the first gesture and the second gesture are captured. The second gesture is different from the first gesture.
[0051] According to the target manipulation behavior, the XR device can control the mouse cursor on the screen to perform an operation corresponding to the target manipulation behavior.
[0052] For example, if the target manipulation behavior is the left key click behavior of the user on the virtual mouse, the XR device can control the mouse cursor on the screen to perform a click operation. Another example is that if the target manipulation behavior is the scrolling behavior of the user on the scroll wheel key of the virtual mouse, the XR device can control the content on the screen to perform a scrolling operation.
[0053] That is, the embodiments of the present application can use a virtual mouse on the XR device to interact with the user, effectively solving the problems of complex technical implementation and poor experience brought by eye-hand interaction.
[0054] In some embodiments, the first gesture includes a gesture in which the positional relationship between the first part of one hand and the second part of the other hand satisfies the first condition.
[0055] Exemplarily, the first part may be the palm, and the second part may be at least one finger. For example, the first gesture may include a gesture in which the positional relationship between the palm of the left hand and at least one finger of the right hand satisfies the first condition. Additionally, the first gesture may also include a gesture in which the positional relationship between the palm of the right hand and at least one finger of the left hand satisfies the first condition.
[0056] Taking the first part as the palm of the right hand and the second part as one finger of the left hand as an example, exemplarily, the first condition may be that the finger of the left hand touches the palm of the right hand, or that the finger of the left hand touches the palm of the right hand and the finger moves within the palm.
[0057] Taking the first part as the palm of the left hand and the second part as at least one finger of the right hand as an example, exemplarily, the range of the palm can be used to simulate a virtual mouse pad, and the fingers of the right hand can be used to simulate the left button, right button, and scroll wheel button of a virtual mouse. Through the cooperation of the fingers of the right hand and the palm of the left hand, the user's manipulation behavior of the virtual mouse can be simulated, thereby realizing interaction with the XR device. In this embodiment, only the range of the left hand palm is used as the activity range of the fingers of the right hand.
[0058] For example, the index finger of the right hand can be used to simulate the left button of the virtual mouse, the middle finger of the right hand can be used to simulate the right button of the virtual mouse, the index finger and middle finger of the right hand can be used to simulate the scroll wheel button of the virtual mouse, and the thumb of the right hand can be used to simulate the base of the virtual mouse.
[0059] In the embodiment of the present application, the left and right hands are used to simulate a virtual mouse pad and a virtual mouse, and the user's manipulation behavior of the virtual mouse is simulated through the cooperation of the left and right hands, enabling the XR device to perform corresponding operations on the content on the screen according to this manipulation behavior, simplifying the interaction process, and enhancing the user's interaction experience.
[0060] Figure 2 It is a flowchart of another interaction method provided by the embodiment of the present application. Figure 2 Different from Figure 1 is that Figure 1 S110 in Figure 2 can be refined into S210 in
[0061] S210. When the first gesture is that the first finger in the second part moves in the target area associated with the palm, map the first gesture to the user's movement behavior of the virtual mouse.
[0062] The first finger here can be at least one finger in the second part, that is, the first gesture in this embodiment is a gesture formed by the cooperation of both hands. By simulating the virtual mouse and mouse pad through the cooperation of the left and right hands, it is more in line with the user's usage habits and more convenient for the user to operate.
[0063] The target area can be an area on the palm, for example, it can be the center area or the edge area of the palm. As Figure 3 shown, Figure 3 Taking the left hand 300 palm as an example, the first finger is the thumb of the right hand 301, and the thumb of the right hand 301 can move in the center area of the left hand 300 palm.
[0064] Exemplarily, the target area can also include both an area on the palm and an area outside the palm at the same time, that is, the first finger can move both on the palm and outside the palm, or some fingers can move on the palm and some fingers can move outside the palm. As Figure 4 shown, Figure 4 Taking the case where some fingers move on the palm and some fingers move outside the palm as an example. For example, the thumb of the right hand 301 moves outside the left hand 300 palm, and the index finger of the right hand 301 moves on the left hand 300 palm. In this way, the movement range of the fingers of the right hand 301 can be expanded, and more interaction operations can be realized.
[0065] Exemplarily, when the first finger moves on the palm, the first finger is used to manipulate an object at a first depth, and when the first finger moves outside the palm, the first finger is used to manipulate an object at a second depth, and the first depth and the second depth are different.
[0066] Exemplarily, in the case where the XR device detects that the right thumb moves in the target area associated with the left palm, the first gesture can be mapped to the user's movement behavior of the virtual mouse. At this time, the XR device can control the movement of the mouse cursor on the screen.
[0067] The embodiment of the present application can use the cooperation of the fingers of one hand and the palm of the other hand to simulate the user's behavior of moving the virtual mouse, so that the mouse cursor on the screen can move to any position, simplifying the interaction process.
[0068] Figure 5 It is a flowchart of another interaction method provided by the embodiment of the present application. Figure 5 Different from Figure 1 is that Figure 1 S110 in Figure 5 can be refined into
[0069] S510. When the first gesture is that the second finger in the second part contacts the palm and the contact parameter meets the second condition, map the first gesture to the user's key pressing behavior on the target key of the virtual mouse.
[0070] Exemplarily, the second finger may be one or more fingers in the second part. For example, the second finger may include one or more of the index finger, middle finger, ring finger, and little finger. The second finger in this embodiment may be the same as or different from the first finger in the above embodiment. When the second finger is the same as the first finger, the contact parameters between the first finger and the palm are different from the contact parameters between the second finger and the palm, so that different interaction operations can be realized.
[0071] Whether the second finger contacts the palm can be detected by a depth sensor in the XR device. The specific detection process is not limited in the embodiments of this application.
[0072] The contact parameters may include but are not limited to contact duration and contact position. Taking the contact parameter including contact duration as an example, the contact duration satisfying the second condition may be, for example, that the contact duration is greater than or equal to the duration threshold, or the contact duration is less than the duration threshold. Taking the contact parameter including contact position as an example, the contact position satisfying the second condition may be, for example, that the contact position is on the left side of the palm center, or the contact position is on the right side of the palm center, or the contact position is at the palm center.
[0073] Exemplarily, when the fingers of one hand contact the palm of the other hand and the contact parameters satisfy the second condition, the first gesture can be mapped to the base key behavior of the left or right button of the virtual mouse by the user.
[0074] For example, as Figures 6 - 7 shown, the index finger of the right hand 301 can be used to simulate the left button 600 of the virtual mouse, and the middle finger of the right hand 301 can be used to simulate the right button 601 of the virtual mouse.
[0075] Exemplarily, in the case where the index finger of the right hand 301 contacts the palm of the left hand 300, the contact duration is less than the duration threshold, and the contact position is on the left side of the palm center, the first gesture can be mapped to the click behavior of the left button of the virtual mouse by the user.
[0076] At this time, the XR device can map the contact position to the screen, obtain the mapped position of the contact position on the screen, and perform a click operation on the object at the mapped position.
[0077] Exemplarily, in the case where the index finger of the right hand 301 contacts the palm of the left hand 300, the contact duration is less than the duration threshold, and the contact position is on the left side of the palm center, the first gesture can be mapped to the click behavior of the left button of the virtual mouse by the user.
[0078] Exemplarily, when the index finger of the right hand 301 touches the palm of the left hand 300, the contact duration is greater than the duration threshold, the index finger of the right hand 301 moves within the palm of the left hand 300, and the thumb of the right hand 301 is outside the palm of the left hand 300, the first gesture can be mapped to the long - press behavior of the user on the left button of the virtual mouse and the movement behavior of the virtual mouse.
[0079] At this time, the XR device can map the contact position to the screen and perform a selection operation on the object at the mapped position.
[0080] Exemplarily, as Figures 8 - 10 , assuming that the index finger of the right hand 301 initially touches the position A on the palm of the left hand 300, the thumb is outside the palm, the index finger of the right hand 301 moves on the palm of the left hand 300, moves from position A to position B, and at position B, the index finger of the right hand 301 leaves the palm of the left hand 300, then the XR device can determine the mapped position A1 of position A on the screen, and then start from the object at the mapped position A1 and follow the movement of the index finger to control the mouse cursor to perform an object selection operation until reaching the mapped position B1 corresponding to position B. Thus, the XR device can control the mouse cursor to select the object between the mapped position A1 and the mapped position B1. For example, in Figure 10 , assuming that the text corresponding to the mapped position A1 is "person" and the text corresponding to the mapped position B1 is "study", then the XR device can select the text "Artificial intelligence is a very broad science".
[0081] In the embodiment of the present application, by using the cooperation of the palm of one hand and the fingers of the other hand, any key - pressing behavior of the user on the left and right buttons of the virtual mouse can be simulated, effectively avoiding the situation that due to the limitation of the user's arm length, the user cannot operate the object on the screen beyond the arm length, and improving the user's interaction experience.
[0082] Figure 11 It is a flowchart of another interaction method provided by the embodiment of the present application. Figure 11 Different from Figure 1 , the S110 in Figure 1 can be refined into Figure 11 the S1100 in
[0083] S1100. When the first gesture is that at least two fingers in the first part touch and slide with the palm, map the first gesture to the scrolling behavior of the user on the scroll wheel button of the virtual mouse.
[0084] Taking two fingers as an example, these two fingers can be two adjacent fingers in the first part or two non - adjacent fingers in the first part. Taking the index finger and the middle finger as an example, exemplarily, as Figure 12 shown, the index finger and the middle finger of the right hand 301 can be used to simulate the scroll wheel button 603 of the virtual mouse.
[0085] Exemplarily, when the index finger and middle finger of the right hand 301 contact the palm of the left hand 300 and slide within the palm, the first gesture can be mapped to the scrolling behavior of the scroll wheel key 603 of the virtual mouse by the user. At this time, the XR device can control the content in the window to scroll. For example, when the index finger and middle finger slide downward within the palm, the XR device can control the content in the window to scroll downward; when the index finger and middle finger slide upward within the palm, the XR device can control the content in the window to scroll upward. The window here can be a window on the screen.
[0086] The embodiment of the present application uses the combination of some fingers of one hand and the palm of the other hand to simulate the behavior of the user scrolling the scroll wheel key of the mouse, so that the content on the screen can scroll in a certain direction, simplifies the interaction process, and improves the user's interaction experience.
[0087] In practical applications, one or more windows can be displayed on the screen, which is specifically related to the user's needs. When multiple windows are displayed on the screen, the sizes of these multiple windows can be the same or different.
[0088] Figure 13 Exemplarily, two windows are displayed, and the sizes of the two windows are the same, both being full-screen displays. What the screen currently shows to the user is window 1301. The coordinate system here is constructed according to the XR device. Figure 13 Taking the coordinate system constructed with the horizontal direction as the x-axis, the vertical direction as the y-axis, and the user's direct front as the z-axis as an example.
[0089] The depth of each window is different. For example, the depth of the window can be positively correlated with the time when the user opens the window, that is, the earlier the user opens the window, the larger the depth value corresponding to the window. Here, the depth value can be represented by the z-axis. That is, the larger the depth value, the farther the window is from the origin in the z-axis direction.
[0090] For example, if the opening time of window 1300 is earlier than that of window 1301, then the depth value of window 1300 is greater than that of window 1301, that is, window 1300 is farther from the user.
[0091] On this basis, as Figure 14 shown, Figure 14 is a flowchart of another interaction method provided by the embodiment of the present application. Figure 14 Different from Figure 1 is that Figure 14 also includes S1410 - S1420, Figure 1 The S120 in Figure 14 can be refined into S1430 in
[0092] S1410. When the first gesture is captured, obtain the pose information of the first part.
[0093] The pose information here is used to characterize the position and pose of the first part. Taking the palm of the left hand 300 as an example of the first part, exemplarily, as Figure 15 shown, the angles between the palm and the x-axis and y-axis can be determined as the pose information of the palm.
[0094] S1420. Determine the target window from each window according to the pose information.
[0095] Different pose information can correspond to windows with different depths. In actual applications, users can adjust the pose information of the palm to select the target window and perform corresponding operations on the objects in the target window.
[0096] For example, the user can rotate along the x-axis direction and determine the target window according to the angle between the palm and the y-axis. For example, the larger the angle with the y-axis, the greater the depth value of the selected window.
[0097] S1430. According to the target manipulation behavior, control the mouse cursor displayed in the target window to perform an operation corresponding to the target manipulation behavior.
[0098] After the target window is determined, the XR device can control the mouse cursor displayed in the target window to perform an operation corresponding to the target manipulation behavior.
[0099] In the embodiments of the present application, when the user opens multiple windows, the target window can be dynamically selected by adjusting the pose information of the palm, and the objects in the target window can be operated by the cooperation of the left and right hands, greatly improving the success rate of interaction.
[0100] In some embodiments, before mapping the first gesture to the target manipulation behavior of the user on the virtual mouse, the interaction method may further include the following steps:
[0101] When the second gesture is captured, display a floating mouse cursor on the screen.
[0102] Exemplarily, the second gesture can be a gesture corresponding to the right hand, and the specific gesture is not limited in the embodiments of the present application.
[0103] In this embodiment, as Figure 16 shown, when the XR device captures the first gesture and the second gesture, it can display a floating mouse cursor 1601 on the screen 1600. That is, at this time, the mouse cursor 1601 will be displayed on the screen 1600, but the mouse cursor 1601 does not select any object and cannot move any object.
[0104] Exemplarily, in the case where the first gesture is not captured, the mouse cursor 1601 is no longer displayed on the screen 1600. That is, when the virtual mouse pad does not appear, the mouse cursor 1601 will not be displayed on the screen 1600 again.
[0105] The embodiments of the present application can use gestures to call out the mouse cursor, providing a basis for subsequently controlling the mouse cursor to perform operations corresponding to target manipulation behaviors. That is, the embodiments of the present application can use the gestures of the left and right hands to simulate the operations of the user on the virtual mouse, and then map them to the screen to achieve the operation of the screen, following the user's habit of using the mouse, simplifying the interaction process, and enhancing the user's interaction experience.
[0106] The embodiments of the present application use the fingers of one hand to simulate the virtual mouse and the palm of the other hand to simulate the virtual mouse pad. By cooperating the fingers of one hand with the palm of the other hand, it is possible to simulate the operation of the user on the virtual mouse on the virtual mouse pad and map the operation to the screen to perform corresponding operations on the content in any window on the screen, which can not only simplify the user's interaction process but also effectively solve the problem that the user cannot operate objects at a relatively long distance due to insufficient arm length, enhancing the user's interaction experience.
[0107] It should be noted that for the interaction method provided by the embodiments of the present application, the execution subject can be an interaction device or a processing module in the interaction device for executing the interaction method. In the embodiments of the present application, taking the interaction device executing the interaction method as an example, the interaction device provided by the embodiments of the present application is described.
[0108] Figure 17 It is a schematic structural diagram of an interaction device provided by the embodiments of the present application.
[0109] As Figure 17 shown, the interaction device 1700 may include:
[0110] A mapping module 1701, configured to map the first gesture to a target manipulation behavior of the user on the virtual mouse in the case where the first gesture is captured;
[0111] A control module 1702, configured to control the mouse cursor displayed on the screen to perform an operation corresponding to the target manipulation behavior according to the target manipulation behavior.
[0112] In the embodiments of the present application, in the case where the first gesture is captured, the first gesture is mapped to a target manipulation behavior of the user on the virtual mouse; according to the target manipulation behavior, the mouse cursor displayed on the screen is controlled to perform an operation corresponding to the target manipulation behavior. That is, the embodiments of the present application use gestures to simulate the target manipulation behavior of the virtual mouse to achieve interaction with the XR device, simplify the interaction process, and enhance the user's interaction experience.
[0113] In some possible implementations of the embodiments of the present application, the first gesture includes a gesture in which the positional relationship between the first part of one hand and the second part of the other hand satisfies a first condition.
[0114] In some possible implementations of the embodiments of the present application, the first part is a palm and the second part is at least one finger.
[0115] In some possible implementations of the embodiments of the present application, the mapping module 1701 is specifically configured to:
[0116] When the first gesture is that the first finger in the second part moves in the target area associated with the palm, map the first gesture to the user's movement behavior of the virtual mouse.
[0117] In some possible implementations of the embodiments of the present application, the mapping module 1701 is specifically configured to:
[0118] When the first gesture is that the second finger in the second part contacts the palm and the contact parameter satisfies a second condition, map the first gesture to the user's keystroke behavior of the target key of the virtual mouse, where the target key includes the left key or the right key of the virtual mouse.
[0119] In some possible implementations of the embodiments of the present application, the mapping module 1701 is specifically configured to:
[0120] When the first gesture is that at least two fingers in the second part contact the palm and slide, map the first gesture to the user's scrolling behavior of the scroll wheel key of the virtual mouse.
[0121] In some possible implementations of the embodiments of the present application, multiple windows are displayed on the screen;
[0122] The interaction device 1700 may further include:
[0123] An acquisition module, configured to acquire the pose information of the first part before the mapping module 1701 maps the first gesture to the user's target manipulation behavior of the virtual mouse;
[0124] A determination module, configured to determine a target window from each window according to the pose information;
[0125] The control module 1702 is specifically configured to:
[0126] According to the target manipulation behavior, control the mouse cursor displayed in the target window to perform an operation corresponding to the target manipulation behavior.
[0127] In some possible implementations of the embodiments of the present application, the interaction device 1700 may further include:
[0128] A display module, configured to display a floating mouse cursor on a screen when a second gesture is captured before the mapping module 1701 maps the first gesture to a target manipulation behavior of the user on the virtual mouse.
[0129] In the embodiment of the present application, the fingers of one hand are used to simulate a virtual mouse, and the palm of the other hand is used to simulate a virtual mouse pad. By cooperating the fingers of one hand with the palm of the other hand, it is possible to simulate the user's operation on the virtual mouse on the virtual mouse pad and map the operation to the screen, so as to perform corresponding operations on the content in any window on the screen. This can not only simplify the user's interaction process, but also effectively solve the problem that the user cannot operate objects at a relatively long distance due to insufficient arm length, improving the user's interaction experience.
[0130] The interaction device in the embodiment of the present application can be a device or a component in an electronic device, such as an integrated circuit or a chip. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc. The embodiment of the present application does not make specific limitations.
[0131] The electronic device in the embodiment of the present application can be a terminal with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems. The embodiment of the present application does not make specific limitations.
[0132] The interaction device provided by the embodiment of the present application can implement Figures 1 - 16 each process in the interaction method embodiment described above and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.
[0133] Such as Figure 18As shown in the figure, an embodiment of the present application further provides an electronic device 1800, which may be an XR device. The electronic device 1800 may include a processor 1801 and a memory 1802. The memory 1802 stores programs or instructions that can run on the processor 1801. When the programs or instructions are executed by the processor 1801, the respective steps of the above-mentioned interaction method embodiment are implemented, and the same technical effects can be achieved. To avoid repetition, details are not described herein again.
[0134] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.
[0135] Figure 19 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.
[0136] The electronic device 1900 includes, but is not limited to: a radio frequency unit 1901, a network module 1902, an audio output unit 1903, an input unit 1904, a sensor 1905, a display unit 1906, a user input unit 1907, an interface unit 1908, a memory 1909, and a processor 1910 and other components.
[0137] Those skilled in the art can understand that the electronic device 1900 may further include a power supply (such as a battery) for supplying power to each component. The power supply can be logically connected to the processor 1910 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 19 The structure of the electronic device 1900 shown in the figure does not limit the electronic device 1900. The electronic device 1900 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements, which will not be elaborated herein.
[0138] Among them, the processor 1910 is configured to map the first gesture to a target control behavior of the user on the virtual mouse when the first gesture is captured; and control the mouse cursor displayed on the screen to perform an operation corresponding to the target control behavior according to the target control behavior.
[0139] In an embodiment of the present application, when the first gesture is captured, the first gesture is mapped to a target control behavior of the user on the virtual mouse; and the mouse cursor displayed on the screen is controlled to perform an operation corresponding to the target control behavior according to the target control behavior. That is, the embodiment of the present application uses gestures to simulate the target control behavior of the virtual mouse to realize the interaction with the XR device, simplifies the interaction process, and improves the user's interaction experience.
[0140] In some possible implementations of the embodiment of the present application, the first gesture includes a gesture in which the positional relationship between the first part of one hand and the second part of the other hand satisfies a first condition.
[0141] In some possible implementations of the embodiments of the present application, the first part is the palm, and the second part is at least one finger.
[0142] In some possible implementations of the embodiments of the present application, the processor 1910 is specifically configured to:
[0143] When the first gesture is that the first finger in the second part moves in the target area associated with the palm, map the first gesture to the movement behavior of the user on the virtual mouse.
[0144] In some possible implementations of the embodiments of the present application, the processor 1910 is specifically configured to:
[0145] When the first gesture is that the second finger in the second part contacts the palm and the contact parameter satisfies the second condition, map the first gesture to the keystroke behavior of the user on the target key of the virtual mouse, where the target key includes the left key or the right key of the virtual mouse.
[0146] In some possible implementations of the embodiments of the present application, the processor 1910 is specifically configured to:
[0147] When the first gesture is that at least two fingers in the second part contact and slide on the palm, map the first gesture to the scrolling behavior of the user on the scroll wheel key of the virtual mouse.
[0148] In some possible implementations of the embodiments of the present application, multiple windows are displayed on the screen;
[0149] The processor 1910 is further configured to obtain the pose information of the first part before mapping the first gesture to the target control behavior of the user on the virtual mouse; and determine the target window from each window according to the pose information.
[0150] In some possible implementations of the embodiments of the present application, the processor 1910 is specifically configured to:
[0151] Control the mouse cursor displayed in the target window to perform an operation corresponding to the target control behavior according to the target control behavior.
[0152] In some possible implementations of the embodiments of the present application, the display unit 1906 is configured to display a floating mouse cursor on the screen when capturing the second gesture before the processor 1910 maps the first gesture to the target control behavior of the user on the virtual mouse.
[0153] In the embodiment of the present application, the fingers of one hand are used to simulate a virtual mouse, and the palm of the other hand is used to simulate a virtual mouse pad. By cooperating the fingers of one hand with the palm of the other hand, it is possible to simulate the operation of a virtual mouse on the virtual mouse pad and map the operation to the screen, so as to perform corresponding operations on the content in any window on the screen. This can not only simplify the user interaction process, but also effectively solve the problem that the user cannot operate objects at a relatively long distance due to insufficient arm length, thus improving the user interaction experience.
[0154] It should be understood that, in the embodiment of the present application, the input unit 1904 may include a Graphics Processing Unit (GPU) 19041 and a microphone 19042. The graphics processor 19041 processes the image data of static pictures or videos obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1906 may include a display panel 19061, and the display panel 19061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1907 includes at least one of a touch panel 19071 and other input devices 19072. The touch panel 19071 is also called a touch screen. The touch panel 19071 may include two parts: a touch detection device and a touch controller. The other input devices 19072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, which will not be elaborated here.
[0155] The memory 1909 can be used to store software programs and various data. The memory 1909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1909 may include a volatile memory or a non-volatile memory, or the memory 1909 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 1909 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.
[0156] The processor 1910 may include one or more processing units; optionally, the processor 1910 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 1910.
[0157] The embodiments of the present application also provide a readable storage medium. A program or instructions are stored on the readable storage medium. When the program or instructions are executed by a processor, each process of the above interaction method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, it will not be elaborated here.
[0158] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.
[0159] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above embodiment of the interaction method, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.
[0160] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.
[0161] The embodiments of the present application provide a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement each process of the above embodiment of the interaction method, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.
[0162] It should be noted that in this article, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.
[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions to enable a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.
[0164] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.
Claims
1. An interactive method, characterized in that: Applied to an extended reality device, the method comprises: When the first gesture is captured, mapping the first gesture to a target manipulation behavior of the user on the virtual mouse; According to the target manipulation behavior, a mouse cursor displayed on the screen is controlled to perform an operation corresponding to the target manipulation behavior.
2. The method according to claim 1, characterized in that The first gesture includes a gesture in which a positional relationship between a first part of one hand and a second part of the other hand satisfies a first condition.
3. The method according to claim 2, characterized in that The first part is a palm, and the second part is at least one finger.
4. The method according to claim 3, characterized in that When the first gesture is captured, mapping the first gesture to a target manipulation behavior of the user on the virtual mouse includes: When the first gesture is a movement of a first finger in the second part in a target area associated with the palm, the first gesture is mapped to a user's movement behavior of the virtual mouse.
5. The method according to claim 3, characterized in that: When the first gesture is captured, mapping the first gesture to a target manipulation behavior of the user on the virtual mouse includes: When the first gesture is the contact between the second finger in the second part and the palm, and the contact parameters meet the second condition, the first gesture is mapped to the user's keystroke behavior on the target key of the virtual mouse, and the target key includes the left button or the right button of the virtual mouse.
6. The method according to claim 3, characterized in that When the first gesture is captured, mapping the first gesture to a target manipulation behavior of the user on the virtual mouse includes: When the first gesture is at least two fingers in the first part contacting and sliding with the palm, the first gesture is mapped to a user's scrolling behavior of the scroll wheel button of the virtual mouse.
7. The method according to claim 3 or 4, characterized in that: A plurality of windows are displayed on the screen; Before mapping the first gesture to a target manipulation behavior of the user on the virtual mouse, the method further includes: Acquire the position and posture information of the first part; Determining a target window from each of the windows according to the position information; According to the target manipulation behavior, controlling a mouse cursor displayed on the screen to perform an operation corresponding to the target manipulation behavior includes: According to the target manipulation behavior, a mouse cursor displayed in the target window is controlled to perform an operation corresponding to the target manipulation behavior.
8. The method according to claim 3 or 4, characterized in that: Before mapping the first gesture to a target manipulation behavior of the user on the virtual mouse, the method further includes: When the second gesture is captured, a mouse cursor in a free state is displayed on the screen.
9. An interactive device, characterized in that: Applied to an extended reality device, the device comprises: A mapping module, configured to map the first gesture to a target manipulation behavior of the virtual mouse by the user when the first gesture is captured; The control module is used to control the mouse cursor displayed on the screen to perform an operation corresponding to the target manipulation behavior according to the target manipulation behavior.
10. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.
11. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
12. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method according to any one of claims 1 to 8.