Display method and device, equipment, medium and product

Abstract

The invention discloses a display method and device, equipment, a medium and a product, and belongs to the technical field of virtual reality. The display method is applied to the augmented reality device. The method comprises the following steps: acquiring first eye movement data of a user; according to the first eye movement data, determining a first fixation position corresponding to a fixation point of the user on a display screen of the augmented reality device; under the condition that a target gesture of the user is captured, a first perspective area corresponding to the target gesture is displayed in an area associated with the first watching position in the display screen, and the first perspective area is used for displaying a real scene.

Description

Technical Field

[0001] The present application belongs to the field of virtual reality technology, and specifically relates to a display method, device, equipment, medium and product. Background Art

[0002] With the development of technology, Extended Reality (XR) devices support the function of starting Video See-Through (VST) in a virtual environment. In the perspective mode, users can see the scenes of the real world and interact with the real world.

[0003] Currently, displaying the perspective area in a virtual environment is mainly achieved through the following methods: turning on the full perspective mode with one click by tapping the XR device or touching a virtual button.

[0004] This method is a complete switch from the virtual world to the real world, which can easily destroy the user's sense of immersion in immersive experiences such as watching movies and playing games. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a display method, device, equipment, medium and product, which can flexibly display a local perspective area without affecting the user's immersion to meet the user's interaction needs with the real world.

[0006] In a first aspect, an embodiment of the present application provides a display method, which is applied to an extended reality device, and the method includes:

[0007] Get the user's first glance data;

[0008] Determine, according to the first eye movement data, a first gaze position corresponding to a gaze point of the user on the display screen of the extended reality device;

[0009] When the target gesture of the user is captured, a first perspective area corresponding to the target gesture is displayed in an area associated with the first gaze position in the display screen, and the first perspective area is used to display the real scene.

[0010] In a second aspect, an embodiment of the present application provides a display device, which is applied to an extended reality device, and the device includes:

[0011] An acquisition module, used to acquire the user's first gaze movement data;

[0012] A determination module, configured to determine a first gaze position corresponding to a gaze point of a user on a display screen of an extended reality device according to the first eye movement data;

[0013] The display module is used to display a first perspective area corresponding to the target gesture in an area associated with a first gaze position in a display screen when a target gesture of the user is captured, and the first perspective area is used to display a real scene.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0018] In an embodiment of the present application, the user's first eye movement data is obtained; based on the first eye movement data, the first gaze position corresponding to the user's gaze point on the display screen of the extended reality device is determined; when the user's target gesture is captured, the first perspective area corresponding to the target gesture is displayed in the area associated with the first gaze position in the display screen, and the first perspective area is used to display the real scene. That is, the embodiment of the present application can display a local perspective area according to the user's gaze position and target gesture, and display the real scene through the local perspective area, and display the virtual scene through other areas, which will not affect the user's sense of immersion, and there is no need for the user to operate the physical buttons of the extended reality device, so that the user will not feel fatigued. In addition, since the user's eye movement data will change, the gaze position determined based on the eye movement data will also change accordingly, that is, the local perspective area can change dynamically, so that the local perspective area can be flexibly updated. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flow chart of a display method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a first gaze position provided in an embodiment of the present application;

[0021] Figure 3 A flowchart of another display method provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a movement trajectory provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a pinch gesture provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of a first partial perspective area provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a confirmation gesture provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of a second partial perspective area provided in an embodiment of the present application;

[0027] Fig. 9 A logical schematic diagram of a display method provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of the structure of a display device provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0030] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0032] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0033] As mentioned above, currently the display of perspective areas in virtual environments is mainly achieved through the following methods: turning on the full perspective mode with one click by tapping the XR device or touching a virtual button. This method is a complete switch from the virtual world to the real world, which can easily destroy the user's sense of immersion in immersive experiences such as watching movies and games.

[0034] To this end, the embodiments of the present application provide a display method, device, equipment, medium and product, which can flexibly display a local perspective area without affecting the user's immersion to meet the user's interaction needs with the real world.

[0035] The display method, device, equipment, medium and product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0036] The display method provided in the embodiments of the present application can be applied to XR devices, which may include, for example, augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, etc. XR devices may include, for example, head-mounted devices. In some embodiments, XR devices may include VR glasses.

[0037] Figure 1 A flow chart of a display method provided in an embodiment of the present application, such as Figure 1 As shown, the display method may include the following steps:

[0038] S110: Acquire the user's first gaze movement data.

[0039] S120. Determine, based on the first eye movement data, a first gaze position corresponding to a gaze point of the user on the display screen of the extended reality device.

[0040] S130: When a target gesture of the user is captured, a first perspective area corresponding to the target gesture is displayed in an area associated with the first gaze position in the display screen.

[0041] The first perspective area is used to display the real scene.

[0042] In an embodiment of the present application, the user's first eye movement data is obtained; based on the first eye movement data, the first gaze position corresponding to the user's gaze point on the display screen of the extended reality device is determined; when the user's target gesture is captured, the first perspective area corresponding to the target gesture is displayed in the area associated with the first gaze position in the display screen, and the first perspective area is used to display the real scene. That is, the embodiment of the present application can display a local perspective area according to the user's gaze position and target gesture, and display the real scene through the local perspective area, and display the virtual scene through other areas, which will not affect the user's sense of immersion, and there is no need for the user to operate the physical buttons of the extended reality device, so that the user will not feel fatigued. In addition, since the user's eye movement data will change, the gaze position determined based on the eye movement data will also change accordingly, that is, the local perspective area can change dynamically, so that the local perspective area can be flexibly updated.

[0043] The above steps are described in detail below:

[0044] In S110, the first eye movement data is the user's eye movement data. The eye movement data can be collected by an eye tracking sensor built into the XR device. The eye movement data may include, for example, the user's pupil size, eye movement time, eye saccade direction, blinking and other data.

[0045] In some embodiments, before using the eye tracking sensor to collect the user's eye movement data, the eye tracking sensor may be used to perform eye movement calibration to improve the accuracy and reliability of the first eye movement data. The specific calibration process is not limited in the embodiments of the present application.

[0046] In actual application, the eye tracking function of the XR device can be turned on by default, that is, the eye tracking sensor is turned on by default. In order to save power consumption, the eye tracking sensor can be turned on when needed. In this case, the user can turn on the XR eye tracking sensor through touch, voice, gesture, etc. After the eye tracking sensor is turned on, the user's eye movement data can be collected.

[0047] In S120, the first gaze position is the position of the user's gaze point on the display screen of the XR device. It is understandable that when the user needs to interact with the real world through a certain area, it can be achieved by gazing at a certain area. When the XR device detects that the user is gazing at a certain area, it can determine the position of the gaze point on the display screen based on the user's first eye movement data, providing a basis for the subsequent display of the local perspective area. In this way, the user does not need to manually operate the physical knob of the XR, so as not to produce a sense of operation fatigue.

[0048] Exemplarily, the XR device may determine the spatial position information of the gaze point based on the first eye movement data, and then perform coordinate transformation on the spatial position information of the gaze point to obtain the position of the gaze point on the display screen.

[0049] In S130, the target gesture may be a gesture associated with parameters such as the size and shape of the local perspective area, that is, the user may control the size and shape of the local perspective area through the target gesture to meet the user's needs. The target gesture may be a static gesture or a dynamic gesture.

[0050] Different target gestures may correspond to different local perspective areas, that is, the local perspective area in the embodiment of the present application may be updated as the target gesture changes, thereby improving the flexibility of the local perspective area.

[0051] For example, for a target gesture similar to "C" where the palm of one hand is bent inwards and the thumb and the other four fingers are facing each other, the shape of the corresponding local perspective area is a circle, an ellipse, or a semicircle, etc. For another example, for a target gesture similar to "triangle" where the left thumb touches the right thumb and the left index finger touches the right index finger, the shape of the corresponding local perspective area is a triangle.

[0052] The user's gesture can be obtained by identifying the user's hand image, and the specific identification process is not limited in the embodiments of this application. The hand image can be collected by a camera, which can be a camera on an XR device, or it can exist independently of the XR device and communicate with the XR device. Taking the camera as an example, in actual application, the camera can be turned on by default, or it can be turned on when needed. In this case, the user can turn on the XR camera through touch, voice, etc. After the camera is turned on, the user's hand image can be collected.

[0053] The area associated with the first gaze position may be an area on the display screen that includes the first gaze position, or may be an area that does not include the first gaze position but whose distance from the first gaze position satisfies a preset distance condition. The preset distance condition may, for example, be that the farthest distance from the first gaze position is not greater than a distance threshold, and the size of the distance threshold may be set according to actual needs.

[0054] When the region includes the first gaze position, the first gaze position may be the center point of the region, or may be a point in the region other than the center point.

[0055] The first perspective area is a partial perspective area on the display screen, that is, the display screen displays other areas in addition to the first perspective area. The user can see the scene of the real world through the first perspective area, so that he can interact with the real world, and can see the virtual scene of the virtual world through other areas. In this way, the interaction between the user and the real world is achieved without destroying the user's immersion in the virtual world.

[0056] In some embodiments, the above S120 may include the following steps:

[0057] determining an eye movement behavior of the user according to the first eye movement data;

[0058] When the eye movement behavior includes gaze behavior, the gaze point is converted from the spatial coordinate system to the screen coordinate system according to the spatial position information of the gaze point and the conversion relationship between the spatial coordinate system and the screen coordinate system to obtain the first gaze position of the gaze point on the display screen. The screen coordinate system is the coordinate system where the display screen is located.

[0059] The user's eye movement behavior can include fixation behavior and non-fixation behavior, such as Figure 2 As shown, the gaze behavior is the behavior of the user gazing at a certain area of ​​the display screen 200. Exemplarily, the user's eye movement spatial position information can be determined according to the first eye movement data, and the user's eye movement behavior can be determined according to the eye movement spatial position information.

[0060] For example, when the time duration of the user's eye movement spatial position information remaining unchanged is not less than the time duration threshold, the user's eye movement behavior can be determined to be a fixation behavior; when the time duration of the user's eye movement spatial position information remaining unchanged is less than the time duration threshold, the user's eye movement behavior can be determined to be a non-fixation behavior. The time duration threshold can be set according to actual needs, and the unit is ms.

[0061] The above-mentioned spatial coordinate system is the coordinate system where the spatial position information of the gaze point is located, and the screen coordinate system is the coordinate system where the display screen is located. The conversion relationship between the spatial coordinate system and the screen coordinate system can be determined during eye movement calibration. According to the conversion relationship between the spatial coordinate system and the screen coordinate system, and in combination with the position information of the gaze point in one of the coordinate systems, the position information of the gaze point in the other coordinate system can be obtained. For example, in an embodiment of the present application, according to the spatial position information of the gaze point, combined with the above-mentioned conversion relationship, the position of the gaze point in the screen coordinate system can be obtained, that is, the first gaze position A of the gaze point on the display screen 200.

[0062] The embodiments of the present application can determine whether the user has a gaze behavior based on the user's eye movement data, and when the user has a gaze behavior, the gaze point in the spatial coordinate system is converted to the screen coordinate system according to the spatial position information of the gaze point and the conversion relationship between the spatial coordinate system and the screen coordinate system, so as to obtain the position information of the gaze point on the display screen, thereby providing a basis for the subsequent display of the local perspective area. Moreover, because the user's gaze area can change dynamically, the local perspective area finally displayed can also change dynamically, thereby improving the flexibility of the local perspective area.

[0063] Figure 3 A flowchart of another display method provided in an embodiment of the present application, Figure 3 and Figure 1 The difference is that Figure 1 The S130 in can be refined into Figure 3 S310-S320 in.

[0064] S310: In response to the target gesture, display the movement trajectory of the hand on the display screen.

[0065] Exemplarily, the target gesture may be a gesture that allows the user to draw a certain area on the display screen, such as but not limited to a tap gesture, a slide gesture, a knuckle circle gesture, etc. When the user draws an area based on the target gesture, the user does not need to look at the first gaze position determined above.

[0066] After the XR device detects the user's target gesture, it can obtain and display the movement trajectory of the hand. Figure 4 As shown, the XR device can display a movement track 401 on the display screen 200 in response to the user's tap gesture. Figure 4 The moving track 401 shown may form a closed area, and the closed area does not include the first gaze position A. In actual application, the closed area may also include the first gaze position A, and the moving track 401 may also form a non-closed area.

[0067] In some embodiments, when displaying the movement trajectory 401, the XR device may also display the drawing order of the movement trajectory 401, for example Figure 4 In the example, if the user moves in a clockwise direction, the XR device can display an arrow representing the clockwise direction.

[0068] S320: Displaying a first perspective area centered on the first gaze position on the display screen according to initial area parameters of the area formed by the movement trajectory.

[0069] The initial region parameters are region parameters of the region formed by the moving track, and are used to characterize the region characteristics of the region. Exemplarily, the initial region parameters may include, but are not limited to, parameters such as the area and shape of the region.

[0070] It can be understood that when the area formed by the moving track is a closed area, the initial area parameter can be the area parameter of the closed area, and when the area formed by the moving track is a non-closed area, the initial area parameter can be the area parameter corresponding to the circumscribed figure of the non-closed area, where the circumscribed figure can be the minimum circumscribed rectangle, circle, etc. of the non-closed area, or a figure obtained by expanding the minimum circumscribed figure according to a certain step length. The step length here can be set according to actual needs.

[0071] After the regional parameters of the area formed by the moving track are determined, the first perspective area corresponding to the regional parameters can be displayed with the first gaze position determined as the center.

[0072] For example, a first perspective area having the same area parameters as the hand-drawn area may be displayed, that is, parameters such as shape and area of ​​the first perspective area are the same as those of the user's hand-drawn area.

[0073] For another example, a first perspective area having a mapping relationship with area parameters of the hand-drawn area may be displayed. The mapping relationship may be used to adjust parameters such as the shape and area of ​​the hand-drawn area. For example, the first perspective area may have a more regular shape and a larger area than the hand-drawn area.

[0074] The embodiment of the present application can display the movement trajectory of the hand based on the user's target gesture, so as to facilitate the user to understand the information of the hand-drawn graphics. At the same time, the first perspective area can be redrawn and displayed based on the area parameters of the hand-drawn graphics with the first gaze position as the center. That is, the embodiment of the present application can dynamically update the first perspective area according to the user's gaze point position and target gesture, thereby improving the flexibility of the first perspective area.

[0075] It is understandable that only when the local perspective function of the XR device is turned on, the XR device can display the local perspective area, and the user can interact with the real world through the local perspective area. Therefore, in order to improve the user's interaction experience with the real world, it is necessary to turn on the local perspective function of the XR device first, then display the user's movement trajectory, and then display the local perspective area.

[0076] Based on this, in some embodiments, the target gesture may include a first gesture and a second gesture, where the first gesture is used to turn on the local perspective function of the XR device. Exemplarily, the first gesture may include a pinch gesture, that is, the tips of the thumb and index finger are in contact, and the remaining fingers can be spread out, similar to an "OK" gesture. Of course, the first gesture may also be other gestures.

[0077] The second gesture is used to draw a certain area on the display screen. Exemplarily, the second gesture may be a tap gesture, a slide gesture, a knuckle circle gesture, or the like.

[0078] Exemplarily, the above S310 may include the following steps:

[0079] In response to the first gesture, when the first gaze position remains unchanged for a preset time period, enabling a local perspective function of the extended reality device;

[0080] In response to the second gesture, a movement track corresponding to the second gesture is displayed on the display screen.

[0081] The fact that the first gaze position remains unchanged within the preset time period indicates that the user has been gazing at the first gaze position.

[0082] Exemplarily, the local perspective function of the XR device is turned off by default. When the XR device captures the user's first gesture and the user is still looking at the above-mentioned first gaze position, it means that the user has the need to interact with the real world. At this time, the XR device can turn on the local perspective function to provide a basis for the subsequent display of the local perspective area.

[0083] If the first gesture is not captured, or only the user's first gesture is captured but the user is not looking at the first gaze position, the local perspective function of the XR device will not be turned on, so as to avoid user's misoperation.

[0084] For example, Figure 5 As shown, when the XR device captures the user's pinch gesture 501 and the user is still looking at the first gaze position A, the local perspective function can be turned on.

[0085] After the local perspective function of the XR device is turned on, the XR device can display the movement trajectory corresponding to the second gesture on the display screen according to the captured second gesture. The display process can refer to the above embodiment.

[0086] In the embodiment of the present application, the local perspective function of XR is turned on only when the user's first gesture is captured and the user has been looking at the same position during the process of capturing the first gesture. After the local perspective function is turned on, the corresponding movement trajectory is displayed based on the captured second gesture, which can effectively avoid user's misoperation and meet the user's needs for interaction with the real world.

[0087] Taking the initial region parameters including at least one of the following: the shape and area of ​​the region as an example, the above S320 may include the following steps:

[0088] When the third gesture of the user is captured, determining the target area parameters of the first perspective area according to the initial area parameters and a reference mapping relationship, where the reference mapping relationship is used to characterize the mapping relationship between the initial area parameters and the target area parameters;

[0089] According to the target area parameters, a first perspective area is displayed on the display screen with the first gaze position as the center.

[0090] The third gesture is used to confirm the hand-drawn area. That is, when the user confirms that the hand-drawn area is drawn, the third gesture can be executed. The embodiment of the present application does not specifically limit the third gesture. For example, the third gesture can be a pinch gesture similar to "OK".

[0091] Exemplarily, when the XR device captures the third gesture, the first gaze position may be cancelled.

[0092] The mapping relationship between the initial area parameter and the target area parameter can be set according to actual needs. For example, the mapping relationship can be a mapping from an irregular shape to a regular shape, that is, the initial area parameter is an irregular shape, and the target area parameter can be a regular shape. The shape types of the two can be the same or different. For example, the initial area parameter is an irregular quadrilateral, and the target area parameter can be a regular circle, or the initial area parameter is an irregular rectangle, and the target area parameter is a regular rectangle, and so on.

[0093] Exemplarily, the mapping relationship may also be a multiple relationship of area, that is, the area corresponding to the target area parameter may be enlarged or reduced relative to the area corresponding to the initial area parameter, and the multiple of enlargement or reduction may be set according to actual needs.

[0094] by Figure 4 Taking the hand-drawn area shown in FIG. 1 as an example, assuming that the mapping relationship includes irregular shapes to regular shapes, and the area is enlarged by 1.5 times, then according to the mapping relationship, it can be displayed Figure 6 The first perspective area 601 shown has a regular shape and an area enlarged by 1.5 times.

[0095] For example, Figure 7 As shown, when the XR captures the confirmation gesture 701, the first gaze position A can be cancelled, and the user can interact with the real world through the first perspective area 601. For example, through the first perspective area 601, the user can look for objects such as a water cup and a mobile phone.

[0096] When capturing the user's confirmation gesture, the embodiment of the present application can obtain the target area parameters of the first perspective area based on the area parameters of the hand-drawn area in combination with the mapping relationship, and then, based on the target area parameters, obtain the first perspective area with a regular shape and appropriate area with the first gaze position as the center, thereby meeting the user's local perspective needs.

[0097] In some embodiments, the display method may further include the following steps:

[0098] Obtain the user's second eye movement data;

[0099] Determine, according to the second eye movement data, a second gaze position corresponding to the gaze point of the user on the display screen, where the second gaze position is different from the first gaze position;

[0100] When the fourth gesture of the user is not captured, the second perspective area is displayed in an area associated with the second gaze position in the display screen according to the target area parameter of the first perspective area.

[0101] The second eye movement data may be the eye movement data of the user looking at another area. That is, the user can adjust the gaze area as needed while wearing the XR device, thereby dynamically adjusting the position of the gaze area without the user having to operate physical buttons, thereby reducing user fatigue.

[0102] The second gaze position is the position of the user's gaze point on the display screen determined according to the second eye movement data. The determination process of the second gaze position can refer to the first gaze position, and for the sake of brevity, it is not repeated here. For example, Figure 8 As shown, the second gaze position B and the first gaze position A are different positions on the display screen.

[0103] The fourth gesture may be a gesture matching the second gaze position B, and used to determine a new local perspective area. Exemplarily, the fourth gesture may include a tap gesture, a slide gesture, a knuckle circle gesture, and the like.

[0104] For example, if it is detected that the user is gazing at the second gaze position B, but the fourth gesture is not captured, the second perspective area can be displayed in the area associated with the second gaze position B according to the area parameters of the local perspective area displayed last time. For example, if the local perspective area displayed last time is the first perspective area, the second perspective area can be displayed with the second gaze position B as the center according to the target area parameters of the first perspective area. Figure 8 As shown, the shape and area of ​​the second perspective area 801 are the same as those of the first perspective area 601 .

[0105] When the user's gaze point changes but a gesture matching the new gaze point is not captured, the embodiment of the present application can automatically display a new perspective area in the area associated with the new gaze point based on the area parameters of the local perspective area displayed last time. That is, the embodiment of the present application can dynamically update the local perspective area according to the change of the user's intention (change of the gaze point), thereby reducing the user's operations and improving the flexibility of the local perspective area.

[0106] In some embodiments, after “determining, according to the second eye movement data, a second gaze position corresponding to the gaze point of the user on the display screen”, the display method may further include the following steps:

[0107] When a fourth gesture of the user is captured and the fourth gesture is different from the target gesture, a third perspective area corresponding to the fourth gesture is displayed in an area associated with the second gaze position in the display screen.

[0108] The fourth gesture is different from the target gesture, which means that the hand-drawn area corresponding to the fourth gesture and the hand-drawn area corresponding to the target gesture have different area parameters. Taking the area parameters including shape and area as an example, the hand-drawn area corresponding to the fourth gesture and the hand-drawn area corresponding to the target gesture have at least one difference in shape and area.

[0109] Exemplarily, when the user's gaze point changes and the user's fourth gesture is captured, and the fourth gesture is different from the target gesture, the XR device can update the local perspective area according to the new gaze point and the new gesture to obtain a third perspective area.

[0110] For example, the third perspective area may be displayed with the second gaze position as the center according to the area parameters of the area corresponding to the fourth gesture.

[0111] Exemplarily, if the fourth gesture is the same as the target gesture, the fourth perspective area may be displayed with the second gaze position as the center according to the target area parameters of the first perspective area.

[0112] The embodiments of the present application can dynamically update the local perspective area according to the changes in the gaze point and the gesture, thereby improving the flexibility of the local perspective area and meeting the personalized needs of the user.

[0113] Combine the following Fig. 9 The display method of the embodiment of the present application is described.

[0114] S1. Turn on the eye tracking and gesture tracking functions of the XR device, that is, turn on the eye tracking sensor and camera of the XR device.

[0115] S2. Obtain the user's eye movement data through an eye tracking sensor.

[0116] S3. Determine whether the user has gaze behavior based on the eye movement data.

[0117] S4. Convert the gaze point from the spatial coordinate system to the screen coordinate system to obtain the gaze position of the gaze point on the display screen.

[0118] S5. Capture the user's gesture and recognize the captured gesture.

[0119] S6. When a pinch gesture is detected, the local perspective function of the XR device is enabled.

[0120] S7: When it is recognized that the user performs a poking gesture, determine the area parameters of the hand-painted area.

[0121] S8. Display the local perspective area based on the area parameters and taking the gaze position in S4 as the center.

[0122] S9: The user gazes at another area to obtain a new gaze position.

[0123] S10: Whether the user's gesture corresponding to the new gaze position is captured, if so, execute S11, otherwise, execute S12.

[0124] S11. With the new gaze position as the center, generate and display a local perspective area with the same regional parameters as the previous local perspective area.

[0125] S12. With the new gaze position as the center, generate and display a local perspective area corresponding to the area parameters corresponding to the new gesture.

[0126] The embodiment of the present application displays a local perspective area based on eye gaze and gestures, and realizes the interaction between the user and the real world without affecting the user's immersion and fatigue. The local perspective area can be flexibly updated according to the eye gaze position and / or gestures, meeting the personalized needs of the user.

[0127] It should be noted that the display method provided in the embodiment of the present application can be executed by a display device or a processing module in the display device for executing the display method. In the embodiment of the present application, the display device provided in the embodiment of the present application is described by taking the display method executed by the display device as an example.

[0128] Fig.10 A schematic diagram of the structure of a display device provided in an embodiment of the present application.

[0129] like Fig.10 As shown, the display device 1000 may include:

[0130] An acquisition module 1001 is used to acquire the user's first eye movement data;

[0131] A determination module 1002 is used to determine a first gaze position corresponding to a gaze point of a user on a display screen of an extended reality device according to the first eye movement data;

[0132] The display module 1003 is used to display a first perspective area corresponding to the target gesture in an area associated with the first gaze position in the display screen when the target gesture of the user is captured, and the first perspective area is used to display the real scene.

[0133] In an embodiment of the present application, the user's first eye movement data is obtained; based on the first eye movement data, the first gaze position corresponding to the user's gaze point on the display screen of the extended reality device is determined; when the user's target gesture is captured, the first perspective area corresponding to the target gesture is displayed in the area associated with the first gaze position in the display screen, and the first perspective area is used to display the real scene. That is, the embodiment of the present application can display a local perspective area according to the user's gaze position and target gesture, and display the real scene through the local perspective area, and display the virtual scene through other areas, which will not affect the user's sense of immersion, and there is no need for the user to operate the physical buttons of the extended reality device, so that the user will not feel fatigued. In addition, since the user's eye movement data will change, the gaze position determined based on the eye movement data will also change accordingly, that is, the local perspective area can change dynamically, so that the local perspective area can be flexibly updated.

[0134] In some possible implementations of the embodiments of the present application, the determination module 1002 is specifically configured to:

[0135] determining an eye movement behavior of the user according to the first eye movement data;

[0136] When the eye movement behavior includes gaze behavior, the gaze point is converted from the spatial coordinate system to the screen coordinate system according to the spatial position information of the gaze point and the conversion relationship between the spatial coordinate system and the screen coordinate system to obtain the first gaze position of the gaze point on the display screen. The screen coordinate system is the coordinate system where the display screen is located.

[0137] In some possible implementations of the embodiments of the present application, the display module 1003 is specifically used for:

[0138] In response to the target gesture, displaying the movement trajectory of the hand on the display screen;

[0139] A first perspective area centered at the first gaze position is displayed on the display screen according to initial area parameters of the area formed by the movement trajectory.

[0140] In some possible implementations of the embodiments of the present application, the target gesture includes a first gesture and a second gesture;

[0141] The display module 1003 is specifically used for:

[0142] In response to the first gesture, when the first gaze position remains unchanged for a preset time period, enabling a local perspective function of the extended reality device;

[0143] In response to the second gesture, a movement track corresponding to the second gesture is displayed on the display screen.

[0144] In some possible implementations of the embodiments of the present application, the initial region parameters include at least one of the following: the shape and area of ​​the region;

[0145] The determination module 1002 is further configured to determine, when capturing a third gesture of the user, target area parameters of the first perspective area according to the initial area parameters and a reference mapping relationship, where the reference mapping relationship is used to characterize a mapping relationship between the initial area parameters and the target area parameters;

[0146] The display module 1003 is specifically used for:

[0147] According to the target area parameters, a first perspective area is displayed on the display screen with the first gaze position as the center.

[0148] In some possible implementations of the embodiments of the present application, the acquisition module 1001 is further used to acquire the second eye movement data of the user;

[0149] The determination module 1002 is further used to determine a second gaze position corresponding to the gaze point of the user on the display screen according to the second eye movement data, where the second gaze position is different from the first gaze position;

[0150] The display module 1003 is further configured to display a second perspective area in an area associated with the second gaze position in the display screen according to the target area parameters of the first local perspective area when the fourth gesture of the user is not captured.

[0151] In some possible implementations of the embodiments of the present application, the display module 1003 is also used to display a third perspective area corresponding to the fourth gesture in the area associated with the second gaze position on the display screen after the determination module 1002 determines the second gaze position corresponding to the user's gaze point on the display screen based on the second eye movement data, and when a fourth gesture of the user is captured and the fourth gesture is different from the target gesture.

[0152] The embodiment of the present application displays a local perspective area based on eye gaze and gestures, and realizes the interaction between the user and the real world without affecting the user's immersion and fatigue. The local perspective area can be flexibly updated according to the eye gaze position and / or gestures, meeting the personalized needs of the user.

[0153] The display device in the embodiment of the present application can be a device, or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a user computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0154] The electronic device in the embodiment of the present application may be a terminal with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0155] The display device provided in the embodiment of the present application can achieve Figures 1 to 9 The various processes in the display method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0156] like Fig.11 As shown, an embodiment of the present application further provides an electronic device 1100, including a processor 1101 and a memory 1102, wherein the memory 1102 stores programs or instructions that can be executed on the processor 1101, and when the program or instructions are executed by the processor 1101, the various steps of the above-mentioned display method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0157] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0158] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0159] The electronic device 1200 includes but is not limited to: a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, and a processor 1210 and other components.

[0160] Those skilled in the art will appreciate that the electronic device 1200 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1210 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The structure of the electronic device 1200 shown in the figure does not constitute a limitation of the electronic device 1200. The electronic device 1200 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0161] The processor 1210 is configured to obtain first eye movement data of the user; and determine a first gaze position corresponding to a gaze point of the user on the display screen of the extended reality device according to the first eye movement data;

[0162] The display unit 1206 is used to display a first perspective area corresponding to the target gesture in an area associated with the first gaze position in the display screen when the target gesture of the user is captured, and the first perspective area is used to display the real scene.

[0163] In an embodiment of the present application, the user's first eye movement data is obtained; based on the first eye movement data, the first gaze position corresponding to the user's gaze point on the display screen of the extended reality device is determined; when the user's target gesture is captured, the first perspective area corresponding to the target gesture is displayed in the area associated with the first gaze position in the display screen, and the first perspective area is used to display the real scene. That is, the embodiment of the present application can display a local perspective area according to the user's gaze position and target gesture, and display the real scene through the local perspective area, and display the virtual scene through other areas, which will not affect the user's sense of immersion, and there is no need for the user to operate the physical buttons of the extended reality device, so that the user will not feel fatigued. In addition, since the user's eye movement data will change, the gaze position determined based on the eye movement data will also change accordingly, that is, the local perspective area can change dynamically, so that the local perspective area can be flexibly updated.

[0164] In some possible implementations of the embodiments of the present application, the processor 1210 is specifically configured to:

[0165] determining an eye movement behavior of the user according to the first eye movement data;

[0166] When the eye movement behavior includes gaze behavior, the gaze point is converted from the spatial coordinate system to the screen coordinate system according to the spatial position information of the gaze point and the conversion relationship between the spatial coordinate system and the screen coordinate system to obtain the first gaze position of the gaze point on the display screen. The screen coordinate system is the coordinate system where the display screen is located.

[0167] In some possible implementations of the embodiments of the present application, the display unit 1206 is specifically used to:

[0168] In response to the target gesture, displaying the movement trajectory of the hand on the display screen;

[0169] A first perspective area centered at the first gaze position is displayed on the display screen according to initial area parameters of the area formed by the movement trajectory.

[0170] In some possible implementations of the embodiments of the present application, the target gesture includes a first gesture and a second gesture;

[0171] The display unit 1206 is specifically used for:

[0172] In response to the first gesture, when the first gaze position remains unchanged for a preset time period, enabling a local perspective function of the extended reality device;

[0173] In response to the second gesture, a movement track corresponding to the second gesture is displayed on the display screen.

[0174] In some possible implementations of the embodiments of the present application, the initial region parameters include at least one of the following: the shape and area of ​​the region;

[0175] The processor 1210 is specifically configured to:

[0176] When the third gesture of the user is captured, determining the target area parameters of the first perspective area according to the initial area parameters and a reference mapping relationship, where the reference mapping relationship is used to characterize the mapping relationship between the initial area parameters and the target area parameters;

[0177] The display unit 1206 is specifically used for:

[0178] According to the target area parameters, a first perspective area is displayed on the display screen with the first gaze position as the center.

[0179] In some possible implementations of the embodiments of the present application, the processor 1210 is specifically configured to:

[0180] Obtain the user's second eye movement data;

[0181] Determine, according to the second eye movement data, a second gaze position corresponding to the gaze point of the user on the display screen, where the second gaze position is different from the first gaze position;

[0182] The display unit 1206 is specifically used for:

[0183] When the fourth gesture of the user is not captured, the second perspective area is displayed in an area associated with the second gaze position in the display screen according to the target area parameter of the first perspective area.

[0184] In some possible implementations of the embodiments of the present application, the display unit 1206 is also used to display a third perspective area corresponding to the fourth gesture in an area associated with the second gaze position on the display screen after the processor 1210 determines, based on the second eye movement data, a second gaze position corresponding to the user's gaze point on the display screen, and when a fourth gesture of the user is captured and the fourth gesture is different from the target gesture.

[0185] The embodiment of the present application displays a local perspective area based on eye gaze and gestures, and realizes the interaction between the user and the real world without affecting the user's immersion and fatigue. The local perspective area can be flexibly updated according to the eye gaze position and / or gestures, meeting the personalized needs of the user.

[0186] It should be understood that in the embodiment of the present application, the input unit 1204 may include a graphics processor (Graphics Processing Unit, GPU) 12041 and a microphone 12042, and the graphics processor 12041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1206 may include a display panel 12061, and the display panel 12061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1207 includes a touch panel 12071 and at least one of other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 may include two parts: a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0187] The memory 1209 can be used to store software programs and various data. The memory 1209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1209 may include a volatile memory or a non-volatile memory, or the memory 1209 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1209 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0188] The processor 1210 may include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1210.

[0189] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned display method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0190] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0191] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned display method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0192] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0193] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned display method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0194] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A display method, characterized in that: Applied to an extended reality device, the method comprises: Get the user's first glance data; Determine, according to the first eye movement data, a first gaze position corresponding to a gaze point of the user on a display screen of the extended reality device; When the user's target gesture is captured, a first perspective area corresponding to the target gesture is displayed in an area associated with the first gaze position in the display screen, where the first perspective area is used to display a real scene.

2. The method according to claim 1, characterized in that: The determining, according to the first eye movement data, a first gaze position corresponding to a gaze point of the user on the display screen of the extended reality device includes: determining the eye movement behavior of the user according to the first eye movement data; In the case where the eye movement behavior includes gaze behavior, the gaze point is converted from the spatial coordinate system to the screen coordinate system according to the spatial position information of the gaze point and the conversion relationship between the spatial coordinate system and the screen coordinate system to obtain the first gaze position of the gaze point on the display screen, and the screen coordinate system is the coordinate system where the display screen is located.

3. The method according to claim 1, characterized in that The method of, in a case where a target gesture of the user is captured, displaying a first perspective area corresponding to the target gesture in an area associated with the first gaze position in the display screen, comprises: In response to the target gesture, displaying a movement trajectory of the hand on the display screen; A first perspective area centered at the first gaze position is displayed on the display screen according to initial area parameters of the area formed by the movement trajectory.

4. The method according to claim 3, characterized in that: The target gesture includes a first gesture and a second gesture; The step of displaying the movement track of the hand on the display screen in response to the target gesture comprises: In response to the first gesture, and when the first gaze position remains unchanged for a preset time period, enabling a local perspective function of the extended reality device; In response to the second gesture, a movement track corresponding to the second gesture is displayed on the display screen.

5. The method according to claim 3, characterized in that: The initial region parameters include at least one of the following: the shape and area of ​​the region; The initial area parameters of the area formed by the movement trajectory, displaying a first perspective area centered on the gaze position on the display screen, comprises: In the case of capturing the third gesture of the user, determining the target area parameters of the first perspective area according to the initial area parameters and a reference mapping relationship, wherein the reference mapping relationship is used to characterize the mapping relationship between the initial area parameters and the target area parameters; The first perspective area is displayed on the display screen with the first gaze position as the center according to the target area parameters.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquiring second eye movement data of the user; determining, according to the second eye movement data, a second gaze position corresponding to the gaze point of the user on the display screen, wherein the second gaze position is different from the first gaze position; In a case where the fourth gesture of the user is not captured, a second perspective area is displayed in an area associated with the second gaze position in the display screen according to the target area parameter of the first perspective area.

7. The method according to claim 6, characterized in that After determining, according to the second eye movement data, a second gaze position corresponding to the gaze point of the user on the display screen, the method further includes: When a fourth gesture of the user is captured and the fourth gesture is different from the target gesture, a third perspective area corresponding to the fourth gesture is displayed in an area associated with the second gaze position in the display screen.

8. A display device, characterized in that: Applied to an extended reality device, the device comprises: An acquisition module, used to acquire the user's first gaze movement data; a determination module, configured to determine, based on the first eye movement data, a first gaze position corresponding to a gaze point of the user on the display screen of the extended reality device; The display module is used to display a first perspective area corresponding to the target gesture in an area associated with the first gaze position in the display screen when a target gesture of the user is captured, wherein the first perspective area is used to display a real scene.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronically readable storage medium, characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method according to any one of claims 1 to 7.

Images