Augmented reality interaction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an augmented reality interaction method and device, electronic equipment and a storage medium. The method comprises the steps that the augmented reality equipment obtains eye movement data of a target user; determining watching position information from the body of the augmented reality equipment according to the eye movement data; if the watching position information is matched with the preset position information, generating a target control instruction corresponding to the preset position information, and executing a target action in response to the target control instruction; wherein the preset position corresponding to the preset position information is located in a target area on the body of the augmented reality device, and the target area is an area, except the main view area, in areas which can be browsed by the target user from the body of the augmented reality device. Eye movement interaction processing is performed through the area except the main view area in the browsable area, so that the false touch rate is reduced, and the accuracy of eye movement interaction and the user experience are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to an extended reality interaction method, device, electronic device, and storage medium. Background Art

[0002] Extended Reality (XR) is a comprehensive term that covers virtual reality (VR), augmented reality (AR) and mixed reality (MR). XR technology expands human perception and interaction methods, interacting with the real world through digital technology to create virtual, augmented and mixed reality experiences. XR provides a more intuitive and convenient content presentation experience through virtual reality scenes or extended reality scenes, and is developing rapidly in entertainment, games, education, medical care, enterprise production and social communication, showing great potential and influence.

[0003] Among them, in the field of extended reality or in extended reality devices, eye tracking technology has become an important way of interaction. Eye tracking technology is used to capture the user's eye data, and the user's gaze point is analyzed based on the eye data, and then control instructions are generated according to the gaze point, and the device is interacted with according to the control instructions.

[0004] However, since the user's eyes are constantly changing, there is always a problem of false triggering in the process of analyzing gaze points based on eye movement data and device interaction, resulting in a poor eye movement interaction experience. Summary of the invention

[0005] The embodiments of the present application provide an extended reality interaction method, device, electronic device and storage medium, which can improve the accuracy of eye movement interaction and enhance user experience.

[0006] The present application provides an extended reality interaction method, which is applied to an extended reality device. The method includes: Obtain eye movement data of target users; Determine gaze position information from the main body of the extended reality device according to the eye movement data; If the gaze position information matches the preset position information, a target control instruction corresponding to the preset position information is generated, and a target action is performed in response to the target control instruction; Among them, the preset position corresponding to the preset position information is located in the target area on the body of the extended reality device, and the target area is the area other than the main viewing area in the area that the target user can browse from the body of the extended reality device when using the extended reality device.

[0007] Accordingly, an embodiment of the present application further provides an extended reality interaction device, which is applied to an extended reality device, and the device includes: An acquisition module is used to acquire the eye movement data of the target user; A determination module, configured to determine gaze position information from a body of the extended reality device according to the eye movement data; A generating module, configured to generate a target control instruction corresponding to the preset position information if the gaze position information matches the preset position information, and execute a target action in response to the target control instruction; Among them, the preset position corresponding to the preset position information is located in the target area on the body of the extended reality device, and the target area is the area other than the main viewing area in the area that the target user can browse from the body of the extended reality device when using the extended reality device.

[0008] Optionally, in some embodiments of the present application, the gaze position information includes at least two within a preset time, and if the gaze position information matches the preset position information, generating a target control instruction corresponding to the preset position information includes: If at least two of the gaze position information match at least two preset position information, a target control instruction is generated according to the at least two preset position information.

[0009] Optionally, in some embodiments of the present application, generating a target control instruction according to the at least two preset position information includes: Determine gaze timing information of each of the preset position information according to the eye movement data, and generate a target control instruction according to the gaze timing information; The gaze timing information and the gaze number information of each preset position information are determined according to the eye movement data, and a target control instruction is generated according to the gaze timing information and the gaze number information.

[0010] Optionally, in some embodiments of the present application, each preset position information corresponds to a preset position, and each preset position is respectively provided with a mark component for guiding the target user's gaze direction, and the field of view angle corresponding to each mark component is greater than an angle threshold, and the mark component is at least one of circular, rectangular, triangular or diamond-shaped.

[0011] Optionally, in some embodiments of the present application, the sign component includes a reflective component or a self-luminous component, and the method further includes: If the sign component is a self-luminous component, obtaining current ambient illumination information, and adjusting display brightness information of the self-luminous component according to the ambient illumination information; and / or, If the sign component is a self-luminous component, the display brightness information of the self-luminous component is adjusted after matching the preset position information.

[0012] Optionally, in some embodiments of the present application, the step of obtaining the eye movement data of the target user includes: Acquire the target user's eye movement data at a preset sampling rate; and / or, Identify the current scene type, determine a target sampling rate according to the current scene type, and obtain eye movement data of the target user according to the target sampling rate; Among them, the current scene type includes the scene type identified based on the virtual scene screen of the extended reality device, or the scene type identified based on the real scene screen browsed by the extended reality device, or the scene type identified based on the virtual scene screen and the real scene screen.

[0013] Optionally, in some embodiments of the present application, before acquiring the eye movement data of the target user, the method further includes: Collecting eye movement data of the target user; Calibrate the eye movement acquisition device according to the eye movement data and the preset position information to obtain a calibrated eye movement acquisition device; The step of obtaining the eye movement data of the target user includes: The eye movement data of a target user who uses the extended reality device is collected by the calibrated eye movement collection device.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned extended reality interaction method when executing the computer program.

[0015] And, the electronic device includes an extended reality device, which includes a main body, and a browsing area is set on the surface of the main body, and the browsing area is an area that can be browsed when using the extended reality device; the browsing area is also divided into a main viewing area and a target area, wherein the target area is an area in the browsing area other than the main viewing area; the extended reality device performs the steps in the above-mentioned extended reality interaction method based on the main viewing area and the target area.

[0016] In a fourth aspect, an embodiment of the present application further provides a storage medium, which includes a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned extended reality interaction method are implemented.

[0017] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0018] An embodiment of the present application obtains eye movement data of a target user using an extended reality device, determines gaze position information from the main body of the extended reality device based on the eye movement data, generates a target control instruction corresponding to the preset position information if the gaze position information matches preset position information, and executes a target action in response to the target control instruction, wherein the preset position corresponding to the preset position information is located in a target area on the main body of the extended reality device, and the target area is an area that the target user can browse from the main body of the extended reality device except for the main viewing area.

[0019] Among them, eye movement interaction between the target user and the extended reality device is achieved by matching the gaze position information corresponding to the eye movement data with the preset position information on the extended reality device body, generating target control instructions, and controlling the extended reality device based on the target control instructions.

[0020] Furthermore, by matching the preset position information on the extended reality device body with the gaze position information and generating target control instructions for interaction, eye movement interaction is independent of the virtual content of the extended reality device, and the extended reality device can be controlled even when the extended reality device does not display a virtual image, thereby improving the operability of the extended reality device.

[0021] Also, by setting preset position information in other areas that are not easily browsed outside the main viewing area, the preset position information is relatively biased in the line of sight, which increases the difficulty of the line of sight falling on the preset position information during eye movement and reduces the false touch rate during eye movement interaction. This makes the line of sight falling on the preset position information more likely to be an active and conscious behavior of the user, and also improves the accuracy of eye movement interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1is a schematic diagram of a scenario in which an extended reality device provided in an embodiment of the present application executes the extended reality interaction method; Figure 2 is a flowchart of an extended reality interaction method provided in an embodiment of the present application; Figure 3 is a schematic diagram of the structure of an extended reality device provided in an embodiment of the present application; Figure 4 is another structural schematic diagram of an extended reality device provided in an embodiment of the present application; Figure 5 is another structural schematic diagram of the extended reality device provided in an embodiment of the present application; Figure 6 is a schematic diagram of the structure of an extended reality interaction device provided in an embodiment of the present application; Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0024] Description of Figure Numbers: 10-extended reality device; 201-acquisition module; 202-determination module; 203-generation module; 301-processor; 302-memory; 303-power supply; 304-input unit. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] The embodiments of the present application provide an extended reality interaction method, device, electronic device and computer-readable storage medium. Specifically, the embodiments of the present application provide an extended reality interaction device suitable for an electronic device (such as an extended reality device), and the extended reality device includes but is not limited to a head-mounted display, wearable glasses, and the extended reality device can also be an integrated extended reality device built into a computing processing unit, or a split extended reality device external to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, an extended reality application applied to handheld mobile devices such as mobile phones, laptops, and tablets (such as extended reality games, virtual tourism, telemedicine, or virtual experiments, etc.), and a near-eye display system applied to wearable devices such as head-mounted displays and smart glasses.

[0027] See also Figure 1 , Figure 1 is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application performs the extended reality interaction method, such as Figure 1 Taking the extended reality device as optical transmission wearable smart AR glasses as an example, the specific execution process of the extended reality device executing the extended reality interaction method is as follows: The extended reality device 10 obtains the eye movement data of the target user, and determines the gaze position information from the main body of the extended reality device based on the eye movement data. If the gaze position information matches the preset position information, a target control instruction corresponding to the preset position information is generated, and a target action is executed in response to the target control instruction.

[0028] Among them, in an embodiment of the present application, the preset position corresponding to the preset position information is located in a target area on the body of the extended reality device, and the target area is an area that the target user can browse from the body of the extended reality device except the main viewing area.

[0029] It can be understood that the target area is an area on the body of the extended reality device that is difficult to browse or directly view, such as an area that the user needs to squint to see, or an area that does not correspond to the front of the user.

[0030] The embodiments of the present application achieve eye movement interaction between the target user and the extended reality device by matching the gaze position information corresponding to the eye movement data with the preset position information on the extended reality device body, generating target control instructions, and controlling the extended reality device based on the target control instructions.

[0031] Furthermore, by matching the preset position information on the extended reality device body with the gaze position information and generating target control instructions for interaction, eye movement interaction is independent of the virtual content of the extended reality device, and the extended reality device can be controlled even when the extended reality device does not display a virtual image, thereby improving the operability of the extended reality device.

[0032] Also, by setting preset position information in other areas that are not easily browsed outside the main viewing area, the preset position information is relatively biased in the line of sight, which increases the difficulty of the line of sight falling on the preset position information during eye movement and reduces the false touch rate during eye movement interaction. This makes the line of sight falling on the preset position information more likely to be an active and conscious behavior of the user, and also improves the accuracy of eye movement interaction.

[0033] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0034] See also Figure 2 , Figure 2 A flowchart of an extended reality interaction method provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the extended reality interaction method is applied to an extended reality device, and the specific process of the method is as follows: 101. Obtain the target user’s eye movement data.

[0035] It should be noted that eye movement data refers to eye movement data, such as the position of the pupil and cornea of ​​the eye during eye gaze, direct gaze, squinting, scanning, blinking, closing the eyes or opening the eyes. In the embodiment of the present application, the eye movement data of the user using the extended reality device can be collected by an eye movement collection device (such as an eye tracker) configured in the extended reality device.

[0036] For example, when the extended reality device is in the form of a smart glasses product, the target user is the user wearing the extended reality device, and the eye movement data is information obtained by collecting the user's eye data through the eye tracker in the extended reality device. For example, the eye tracker is an eye tracker, which captures the user's eye image through the camera of the eye tracker, and then analyzes the image through image processing technology to determine the position of the pupil and cornea.

[0037] 102. Determine gaze position information from the main body of the extended reality device based on the eye movement data.

[0038] It can be understood that the gaze position information includes the gaze point position. After obtaining the eye movement data, the gaze position information of the user's eyes can be determined by geometric calculation. For example, based on the position of the pupil and cornea, combined with the known parameters of the eye tracker (such as camera focal length, user head position, etc.), geometric calculations are performed to determine the position of the gaze point in the extended reality device body.

[0039] The main body refers to the extended reality device itself, such as when the extended reality device is a smart glasses, the main body refers to the smart glasses itself. It should be noted that in the embodiment of the present application, the gaze position information is mainly analyzed based on the surface of the main body of the extended reality device to analyze the position of the eye movement data on the surface of the main body, such as analyzing whether the gaze point of the eye movement data falls on the tail of the temple of the smart glasses or on the four sides of the frame. The tail of the temple refers to the side of the temple close to the frame, and the frame is the frame structure that wraps or fixes the optical lenses of the glasses.

[0040] 103. If the gaze position information matches the preset position information, a target control instruction corresponding to the preset position information is generated, and a target action is executed in response to the target control instruction.

[0041] Among them, the preset position information is predetermined position information, which is a pre-marked position, such as the four corners of the frame or the end of the temple. In the embodiment of the present application, in order to realize the intelligent interaction between the user and the extended reality device, the corresponding trigger function is matched for each preset position information. After analyzing that the user's gaze point falls on the preset position information, the corresponding control instruction is generated to control the extended reality device to perform the corresponding action to trigger the function of the extended reality device. For example, these functions include but are not limited to unlocking, locking, taking pictures, playing music, recording or scanning recognition.

[0042] It should be emphasized that due to the high reliance on user eye movement data when interacting based on eye movement data, and the high flexibility of the human eye, the eyeball will move constantly, therefore, the probability of false triggering is high, especially when the preset position is in an area that is easily browsed by the eyes, it is more likely to generate false triggering operations. For example, when the preset position is set in the middle of the front of the glasses, that is, facing the direction of the user's eyes, the probability of being triggered is high, and accordingly, the probability of false triggering will also increase.

[0043] Based on this, in an embodiment of the present application, in order to improve the accuracy of eye movement interaction and prevent users from triggering the extended reality device to perform actions too easily, the preset position information is set at a position that is not easy to browse. On the basis of ensuring that it is triggerable and operable, the functional action corresponding to each preset position is what the user expects when it is triggered, and meets the user's expected effect.

[0044] That is, in the embodiment of the present application, the preset position corresponding to the preset position information is located in the target area on the body of the extended reality device, and the target area is the area that the target user can browse from the body of the extended reality device except the main viewing area.

[0045] Among them, the main viewing area is the area that the user mainly browses when using the extended reality device. It can also be understood as the area that the user can easily browse when using the extended reality device, for example, the middle of the front of the extended reality device, that is, the area facing the user's eyes. For example, if the extended reality device is smart glasses, the main viewing area is the middle of the optical lens of the smart glasses; if the extended reality device is a vehicle-mounted AR-HUD, the main viewing area is the middle of the AR-HUD projection area.

[0046] It can be understood that the target area where the preset position is located is the area in the browsing area except the main viewing area, that is, the area that cannot be easily browsed, for example, the edge part of the area that can be browsed by the extended reality device. For example, if the extended reality device is smart glasses, the target area is the edge area of ​​the optical lens, the frame around the optical lens, the tail area of ​​the temple, etc.

[0047] In other words, the target area is the area that can be viewed by the user's peripheral vision when the user uses the extended reality device normally and looks straight ahead (for example, the area 30 degrees to the left, 30 degrees to the right, 30 degrees above, and 50 degrees below the straight ahead direction when looking straight ahead), or the area that the user can only view by squinting when using the extended reality device, for example, for an in-vehicle AR-HUD, when the user browses the middle of the projection area of ​​the in-vehicle AR-HUD, the edge area of ​​the projection area that can be viewed by the user's peripheral vision, or when the user wears smart glasses and looks straight ahead, the edge of the optical lens, the frame, or the end of the temple that can be viewed by the peripheral vision.

[0048] In summary, the embodiment of the present application sets the preset position information used to generate control instructions and control the extended reality device to perform actions to complete the interaction in the non-primary viewing area of ​​the extended reality device, that is, the target area, thereby reducing the frequency of the user's line of sight falling on the preset position corresponding to the preset position information when using the extended reality device, and reducing the frequency of accidental touches. In an embodiment of the present application, the user can squint to place the gaze point at the preset position at the edge of the extended reality device to trigger the extended reality device to perform an action to complete the eye movement interaction. It is understandable that the duration of the squint can be set to be relatively short, which is related to the logic of whether to generate a target control instruction. For example, if it exceeds 1s, the generation of the target control instruction is triggered without causing excessive eye use.

[0049] Among them, in the embodiments of the present application, the function triggered by each preset position can be flexibly configured. For example, the function to be triggered corresponding to each preset position can be set based on user habits or preferences, and whether to trigger to further avoid accidental touch can be comprehensively judged by parameters such as the distance error between the gaze position and the preset position, the gaze duration and the gaze frequency. For example, when the distance error between the gaze position and the preset position is less than the set distance threshold, the gaze duration is greater than the set duration threshold, and the gaze frequency is greater than the set frequency threshold, the target control instruction corresponding to the preset position is triggered.

[0050] Optionally, in order to enrich the diversity of the interactive content between the user and the extended reality device, the user can be interacted with based on multiple preset position information in the target area of ​​the extended reality device, and the multiple preset position information can trigger the generation of target control instructions separately or jointly, thereby improving operability and the richness of interactive content. For example, see Figure 3 As shown, Figure 3 : is a structural diagram of an extended reality device provided in an embodiment of the present application, wherein the extended reality device takes smart glasses as an example, wherein the preset position information is the positions of the four corners of the left and right frames of the extended reality device as a whole (such as Figure 3 The black dot in the middle) can also be Figure 4 As shown in the four corners of a single frame (such as Figure 4 black dot in the middle).

[0051] Accordingly, matching multiple preset position information requires multiple gaze position information, that is, optionally, in some embodiments of the present application, the gaze position information includes at least two within a preset time, and the step of "if the gaze position information matches the preset position information, generating a target control instruction corresponding to the preset position information" includes: If at least two pieces of gaze position information match at least two pieces of preset position information, a target control instruction is generated according to the at least two pieces of preset position information.

[0052] For example, eye movement data is collected over a continuous period of time, and multiple gaze position information is obtained from the eye movement data through analysis. Furthermore, multiple preset position information is obtained by matching from the main body of the extended reality device.

[0053] Among them, when there are multiple preset position information, the target control instructions required to be generated by the extended reality device or the functions required to be triggered are controlled according to the multi-point trigger logic. For example, the multi-point trigger logic includes: a generation strategy based on preset position information and a generation strategy based on timing, etc.

[0054] Among them, the generation strategy based on the preset position information is specifically: simply control the generation of target control instructions based on the included preset position information. For example, if the preset position information involved includes position 1 and position 2, the target control instructions are generated according to the correspondence between position 1 and position 2 and the target control instructions. If the preset position information involved includes position 1 and position 3, the target control instructions are generated according to the correspondence between position 1 and position 3 and the target control instructions.

[0055] Among them, the generation strategy based on timing includes: on the basis of the specific preset position information involved, the gaze order of each preset position information is also considered. For example, if the preset position information involved includes position 1, position 2 and position 3, and the gaze order is position 1, position 2 and position 3, then the target control instruction is generated according to the correspondence between the order of position 1, position 2 and position 3 and the target control instruction. If the gaze order is position 1, position 3 and position 2, then the target control instruction is generated according to the correspondence between the order of position 1, position 3 and position 2 and the target control instruction. That is, when multiple preset position information is involved and the target control instruction is generated based on the annotation order, different gaze orders control the generation of different target control instructions. That is, the step of "generating target control instructions according to at least two preset position information" includes: Determine the gaze timing information of each preset position information according to the eye movement data, and generate a target control instruction according to the gaze timing information; The gaze timing information and the gaze number information of each preset position information are determined according to the eye movement data, and the target control instruction is generated according to the gaze timing information and the gaze number information.

[0056] The fixation timing information is the above-mentioned fixation sequence.

[0057] Optionally, in order to improve the smoothness of eye movement interaction, a marker component may be set at a preset position of the extended reality device to guide the user to accurately place the gaze point at the preset position through the marker component, such as Figure 3 and Figure 4 As shown, the black dots represent the logo components. It should be understood that Figure 3 and Figure 4 The black dots shown in the figure mainly represent the position of the logo component, but do not limit the specific shape of the logo component. For example, see Figure 5 As shown, the shape of the logo component can also be a rectangle, such as a rounded rectangle. Similarly, the logo component can also be a triangle or a rhombus.

[0058] Among them, in order to simplify the difficulty of user eye movement operations and facilitate matching to the desired preset position or the desired sign component, in an embodiment of the present application, the field of view angle between the preset positions or sign components is controlled to be above the set angle. For example, the position between the sign components must satisfy the field of view angle of more than 30 degrees.

[0059] In the embodiment of the present application, in order to enhance the eye-catching effect of the sign component, the sign component may be a reflective component or a self-luminous component, wherein the reflective component includes but is not limited to a reflective strip, a warning strip, etc., and may also be different colors or materials adapted to different ambient light. The self-luminous component includes but is not limited to LED and other lights.

[0060] Among them, if the sign component is a self-luminous component, the self-luminous component can adjust the display brightness according to the ambient illumination or switch the display brightness according to whether the landing point is at a preset position, that is, if the sign component is a self-luminous component, the current ambient illumination information is obtained, and the display brightness information of the self-luminous component is adjusted according to the ambient illumination information; and / or, if the sign component is a self-luminous component, the display brightness information of the self-luminous component is adjusted after matching the preset position information.

[0061] For example, when the ambient illumination is high, the display brightness is appropriately increased to improve the display effect and facilitate user perception. If the ambient illumination is low, the display brightness is appropriately reduced to avoid glare problems.

[0062] For another example, the self-luminous component can be set to three states: sleep / shutdown state, monitoring state and activation state, where the shutdown state: when the function is not activated or the extended reality device is in sleep mode or shutdown mode, the self-luminous component does not emit light, such as the LED light is not lit; monitoring state: the state when the system is operating normally, no gaze position information matching the preset position information is detected, and the self-luminous component displays a brightness or color, such as the LED light is displayed in color A; activation state: when the system detects gaze position information matching the preset position information, it displays another brightness or color, such as the LED switches to color B to inform the user that the preset position or trigger point has been activated. In addition, in addition to the change in color and brightness as a reminder, a prompt sound can also be set to prompt the user to activate or match the preset position.

[0063] Optionally, in an embodiment of the present application, the sampling rate of collecting the user's eye movement data may be controlled to reduce the collection frequency, reduce the amount of calculation, and reduce power consumption. That is, optionally, in some embodiments of the present application, the step of "obtaining the eye movement data of the target user" includes: Acquire the target user's eye movement data at a preset sampling rate; and / or, Identify the current scene type, determine the target sampling rate according to the current scene type, and obtain the eye movement data of the target user according to the target sampling rate; Among them, the current scene type includes the scene type identified based on the virtual scene screen of the extended reality device, or the scene type identified based on the real scene screen browsed by the extended reality device, or the scene type determined comprehensively based on the virtual scene screen and the real scene screen.

[0064] The preset sampling rate is a preset sampling rate, which may be determined by statistical analysis in advance. For example, the preset sampling rate meets the accuracy requirement of acquisition and can also reduce power consumption as much as possible.

[0065] Among them, for different usage scenarios, the response speed of eye movement interaction also has different requirements. Therefore, in an embodiment of the present application, the current scene type is analyzed and the sampling rate for collecting eye movement data is adjusted based on the scene type.

[0066] The current scene type is the current scene type, which includes but is not limited to game scenes, learning scenes, conversation scenes, shopping scenes, and film and television scenes. The scene type can be obtained by collecting virtual scene images provided by the extended reality device and analyzing the real scene images currently in or currently browsed, for example, by analyzing the virtual scene images and the real scene images through a scene recognition model based on machine learning to identify their corresponding scene types.

[0067] Optionally, before using the extended reality device, the accuracy of the line of sight analysis and position matching may be improved by calibration, that is, optionally, in some embodiments of the present application, before the step of “obtaining the eye movement data of the target user”, the method further includes: Collect eye movement data of target users; Calibrate the eye movement acquisition device according to the eye movement data and the preset position information to obtain a calibrated eye movement acquisition device; Obtain the target user's eye movement data, including: The eye movement data of the target user using the extended reality device is collected through the calibrated eye movement collection device.

[0068] Among them, calibration includes lighting up each logo component in turn, and recording the user's gaze position for each logo component, and calibrating the eye movement acquisition equipment with the gaze position as a reference to meet the needs of people with different faces and eye distances. At the same time, when wearing smart glasses, if wearing drift occurs (such as the position of the smart glasses moves relative to the eyes), reducing data quality, the eye movement acquisition equipment can be optimized through calibration to adapt to the current drift situation.

[0069] In summary, the embodiments of the present application achieve eye movement interaction between the target user and the extended reality device by matching the gaze position information corresponding to the eye movement data with the preset position information on the extended reality device body, generating target control instructions, and controlling the extended reality device based on the target control instructions.

[0070] Furthermore, by matching the preset position information on the extended reality device body with the gaze position information and generating target control instructions for interaction, eye movement interaction is independent of the virtual content of the extended reality device, and the extended reality device can be controlled even when the extended reality device does not display a virtual image, thereby improving the operability of the extended reality device.

[0071] Also, by setting preset position information in other areas that are not easily browsed outside the main viewing area, the preset position information is relatively biased in the line of sight, which increases the difficulty of the line of sight falling on the preset position information during eye movement and reduces the false touch rate during eye movement interaction. This makes the line of sight falling on the preset position information more likely to be an active and conscious behavior of the user, and also improves the accuracy of eye movement interaction.

[0072] In order to better implement the extended reality interaction method of the present application, the present application also provides an extended reality interaction device based on the above extended reality interaction method. The meanings of the terms are the same as those in the above extended reality interaction method, and the specific implementation details can refer to the description in the method embodiment.

[0073] See also Figure 6 , Figure 6 : is a schematic diagram of the structure of an extended reality interaction device provided in an embodiment of the present application, wherein the extended reality interaction device is applied to an extended reality device, and the device may be specifically as follows: An acquisition module 201 is used to acquire eye movement data of a target user; A determination module 202, configured to determine gaze position information from a body of an extended reality device according to eye movement data; A generating module 203, configured to generate a target control instruction corresponding to the preset position information if the gaze position information matches the preset position information, and execute a target action in response to the target control instruction; Among them, the preset position corresponding to the preset position information is located in the target area on the body of the extended reality device. The target area is the area other than the main viewing area in the area that the target user can browse from the body of the extended reality device when using the extended reality device.

[0074] Optionally, in some embodiments of the present application, the gaze position information includes at least two within a preset time, and if the gaze position information matches the preset position information, a target control instruction corresponding to the preset position information is generated, including: If at least two pieces of gaze position information match at least two pieces of preset position information, a target control instruction is generated according to the at least two pieces of preset position information.

[0075] Optionally, in some embodiments of the present application, generating a target control instruction according to at least two preset position information includes: Determine the gaze timing information of each preset position information according to the eye movement data, and generate a target control instruction according to the gaze timing information; The gaze timing information and the gaze number information of each preset position information are determined according to the eye movement data, and the target control instruction is generated according to the gaze timing information and the gaze number information.

[0076] Optionally, in some embodiments of the present application, each preset position information corresponds to a preset position, and each preset position is respectively provided with a mark component for guiding the target user's gaze direction, the field of view angle corresponding to each mark component is greater than an angle threshold, and the mark component is at least one of circular, rectangular, triangular or diamond-shaped.

[0077] Optionally, in some embodiments of the present application, the sign component includes a reflective component or a self-luminous component, and the method further includes: If the sign component is a self-luminous component, obtaining current ambient illumination information, and adjusting display brightness information of the self-luminous component according to the ambient illumination information; and / or, If the sign component is a self-luminous component, the display brightness information of the self-luminous component is adjusted after matching the preset position information.

[0078] Optionally, in some embodiments of the present application, obtaining the eye movement data of the target user includes: Acquire the target user's eye movement data at a preset sampling rate; and / or, Identify the current scene type, determine the target sampling rate according to the current scene type, and obtain the eye movement data of the target user according to the target sampling rate; Among them, the current scene type includes the scene type identified based on the virtual scene screen of the extended reality device, or the scene type identified based on the real scene screen browsed by the extended reality device, or the scene type identified based on the virtual scene screen and the real scene screen.

[0079] Optionally, in some embodiments of the present application, before obtaining the eye movement data of the target user, the method further includes: Collect eye movement data of target users; Calibrate the eye movement acquisition device according to the eye movement data and the preset position information to obtain a calibrated eye movement acquisition device; Obtain the target user's eye movement data, including: The eye movement data of the target user using the extended reality device is collected through the calibrated eye movement collection device.

[0080] In the embodiment of the present application, the acquisition module 201 first acquires the eye movement data of the target user, the determination module 202 determines the gaze position information from the body of the extended reality device based on the eye movement data, and the generation module 203 generates a target control instruction corresponding to the preset position information if the gaze position information matches the preset position information, and executes the target action in response to the target control instruction.

[0081] Among them, the preset position corresponding to the preset position information is located in the target area on the body of the extended reality device. The target area is the area other than the main viewing area in the area that the target user can browse from the body of the extended reality device when using the extended reality device.

[0082] Among them, the embodiment of the present application achieves eye movement interaction between the target user and the extended reality device by matching the gaze position information corresponding to the eye movement data with the preset position information on the extended reality device body, generating target control instructions, and controlling the extended reality device based on the target control instructions.

[0083] Furthermore, by matching the preset position information on the extended reality device body with the gaze position information and generating target control instructions for interaction, eye movement interaction is independent of the virtual content of the extended reality device, and the extended reality device can be controlled even when the extended reality device does not display a virtual image, thereby improving the operability of the extended reality device.

[0084] Also, by setting preset position information in other areas that are not easily browsed outside the main viewing area, the preset position information is relatively biased in the line of sight, which increases the difficulty of the line of sight falling on the preset position information during eye movement and reduces the false touch rate during eye movement interaction. This makes the line of sight falling on the preset position information more likely to be an active and conscious behavior of the user, and also improves the accuracy of eye movement interaction.

[0085] In addition, the present application also provides an electronic device, such as Figure 7 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically: The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0086] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0087] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0088] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0089] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in any one of the extended reality interaction methods provided in the embodiments of the present application.

[0090] Further, the electronic device includes an extended reality device, which includes a body, a browsing area is set on the surface of the body, and the browsing area is an area that can be browsed when using the extended reality device; the browsing area is further divided into a main viewing area and a target area, wherein the target area is an area other than the main viewing area in the browsing area; the extended reality device performs the steps in the above-mentioned extended reality interaction method based on the main viewing area and the target area, specifically: The extended reality device of the embodiment of the present application obtains eye movement data of a target user using the extended reality device, determines gaze position information from the main body of the extended reality device based on the eye movement data, generates a target control instruction corresponding to the preset position information if the gaze position information matches the preset position information, and executes a target action in response to the target control instruction, wherein the preset position corresponding to the preset position information is located in a target area on the main body of the extended reality device, and the target area is an area that the target user can browse from the main body of the extended reality device except the main viewing area.

[0091] Among them, eye movement interaction between the target user and the extended reality device is achieved by matching the gaze position information corresponding to the eye movement data with the preset position information on the extended reality device body, generating target control instructions, and controlling the extended reality device based on the target control instructions.

[0092] Furthermore, by matching the preset position information on the extended reality device body with the gaze position information and generating target control instructions for interaction, eye movement interaction is independent of the virtual content of the extended reality device, and the extended reality device can be controlled even when the extended reality device does not display a virtual image, thereby improving the operability of the extended reality device.

[0093] Also, by setting preset position information in other areas that are not easily browsed outside the main viewing area, the preset position information is relatively biased in the line of sight, which increases the difficulty of the line of sight falling on the preset position information during eye movement and reduces the false touch rate during eye movement interaction. This makes the line of sight falling on the preset position information more likely to be an active and conscious behavior of the user, and also improves the accuracy of eye movement interaction.

[0094] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0095] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0096] To this end, the present application provides a storage medium, which includes a computer-readable storage medium, on which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any extended reality interaction method provided in the present application.

[0097] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0098] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0099] Since the instructions stored in the computer-readable storage medium can execute the steps in any extended reality interaction method provided in the present application, the beneficial effects that can be achieved by any extended reality interaction method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0100] The above is a detailed introduction to an extended reality interaction method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An extended reality interaction method, characterized in that: Applied to an extended reality device, the method comprises: Obtain eye movement data of target users; Determine gaze position information from the main body of the extended reality device according to the eye movement data; If the gaze position information matches the preset position information, a target control instruction corresponding to the preset position information is generated, and a target action is performed in response to the target control instruction; The preset position corresponding to the preset position information is located in a target area on the main body of the extended reality device, and the target area is an area that the target user can browse from the main body of the extended reality device except the main viewing area.

2. The extended reality interaction method according to claim 1, characterized in that: The gaze position information includes at least two within a preset time, and if the gaze position information matches the preset position information, generating a target control instruction corresponding to the preset position information includes: If at least two of the gaze position information match at least two preset position information, a target control instruction is generated according to the at least two preset position information.

3. The extended reality interaction method according to claim 2, characterized in that: The generating the target control instruction according to the at least two preset position information comprises: Determine gaze timing information of each of the preset position information according to the eye movement data, and generate a target control instruction according to the gaze timing information; The gaze timing information and the gaze number information of each preset position information are determined according to the eye movement data, and a target control instruction is generated according to the gaze timing information and the gaze number information.

4. The extended reality interaction method according to any one of claims 1 to 3, characterized in that: Each preset position information corresponds to a preset position, and each preset position is respectively provided with a mark component for guiding the target user's gaze direction, and the field of view angle corresponding to each mark component is greater than an angle threshold, and the mark component is at least one of circular, rectangular, triangular or diamond-shaped.

5. The extended reality interaction method according to claim 4, characterized in that: The sign assembly includes a reflective assembly or a self-luminous assembly, and the method further includes: If the sign component is a self-luminous component, obtaining current ambient illumination information, and adjusting display brightness information of the self-luminous component according to the ambient illumination information; and / or, If the sign component is a self-luminous component, after matching the preset position information, the display brightness information or display color information of the self-luminous component is adjusted.

6. The extended reality interaction method according to claim 1, characterized in that: The step of obtaining the eye movement data of the target user includes: Acquire the target user's eye movement data at a preset sampling rate; and / or, Identify the current scene type, determine a target sampling rate according to the current scene type, and obtain eye movement data of the target user according to the target sampling rate; Among them, the current scene type includes the scene type identified based on the virtual scene screen of the extended reality device, or the scene type identified based on the real scene screen browsed by the extended reality device, or the scene type determined comprehensively based on the virtual scene screen and the real scene screen.

7. The extended reality interaction method according to claim 1, characterized in that: Before acquiring the eye movement data of the target user, the method further includes: Collecting eye movement data of the target user; Calibrate the eye movement acquisition device according to the eye movement data and the preset position information to obtain a calibrated eye movement acquisition device; The step of obtaining the eye movement data of the target user includes: The eye movement data of a target user who uses the extended reality device is collected by the calibrated eye movement collection device.

8. An extended reality interaction device, characterized in that: Applied to an extended reality device, the device comprises: An acquisition module is used to acquire the eye movement data of the target user; A determination module, configured to determine gaze position information from a body of the extended reality device according to the eye movement data; A generating module, configured to generate a target control instruction corresponding to the preset position information if the gaze position information matches the preset position information, and execute a target action in response to the target control instruction; The preset position corresponding to the preset position information is located in a target area on the main body of the extended reality device, and the target area is an area that the target user can browse from the main body of the extended reality device except the main viewing area.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the extended reality interaction method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the extended reality interaction method according to any one of claims 1 to 7 are implemented.

Images