Wearable display device, control method, apparatus, electronic device, and storage medium

Abstract

The application discloses a wearable display device, a control method and device, an electronic device and a storage medium, and belongs to the technical field of electronic devices. The wearable display device comprises a body, a camera assembly, a display screen and a controller. The body comprises a wearing part. The camera assembly is arranged on the body and comprises an image sensor and a driving part. The image sensor is used for shooting an environmental picture, and the driving part is used for driving the image sensor to rotate relative to the body. The display screen is arranged in the wearing part and is used for displaying the environmental picture. The eyeball sensor is arranged on the wearing part and is used for acquiring the pupil position of the wearer of the wearable display device. The controller is electrically connected with the camera assembly and the eyeball sensor, is used for determining a target focal point according to the pupil position, controlling the driving part to drive the image sensor to rotate according to the target focal point, determining a first display area and a second display area on the display screen according to the target intersection, and controlling the display resolution and / or display brightness of the second display area to be reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of electronic devices, and particularly relates to a wearable display device, a control method and device, an electronic device and a storage medium. BACKGROUND

[0002] In related technologies, wearable devices such as extended reality (MR) glasses capture images of the external environment through a camera arranged in front of the glasses, display the images of the external environment through a display screen inside the glasses, and display virtual objects in the images of the external environment through display control to create an environment that combines virtuality and reality and is interactive with humans.

[0003] The visual angle of the picture displayed by the above-mentioned extended reality device is generally about 100°. The central part of the picture captured by the camera is relatively clear, while the picture at the edge of the picture is distorted due to the weak ability of the camera to process the focal point outside the quadrant and the lens distortion of the camera itself.

[0004] When the user's head rotates, the wearable device and the camera move together with the user's head, so the focal point of the captured image also changes, and the user's visual focal point is consistent with the focal point of the camera. When the user's head does not move, but the eyeball rotates, the focal point of the camera does not change, but the focal point position of the human eye changes. When the focal point of the human eye falls on the edge area of the picture, the display effect is affected due to the distortion of the picture. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a wearable display device, a control method and device, an electronic device and a storage medium, which can reduce the influence of lens distortion of the camera on the display effect of the extended reality.

[0006] In a first aspect, the embodiments of the present application provide a wearable display device, comprising:

[0007] a body, the body comprising a wearing part;

[0008] a camera assembly arranged in the body, comprising an image sensor and a driving part, the image sensor being configured to capture an environment picture, and the driving part being configured to drive the image sensor to rotate relative to the body;

[0009] a display screen arranged in the wearing part, the display screen being configured to display the environment picture;

[0010] an eyeball sensor arranged in the wearing part, the eyeball sensor being configured to obtain a pupil position of a wearer of the wearable display device;

[0011] The controller is electrically connected with the camera assembly and the eyeball sensor, is used for determining a target focus point according to the pupil position, and controlling the driving part to drive the image sensor to rotate according to the target focus point, wherein the shooting focus point of the rotated image sensor matches the target focus point; determining a first display area and a second display area on the display screen according to the target intersection point, wherein the distance value between the pixel point in the first display area and the target focus point is less than a preset threshold value, and the distance value between the pixel point in the second display area and the target focus point is greater than the preset threshold value; and controlling to reduce the display resolution and / or display brightness of the second display area.

[0012] In a second aspect, the embodiments of the present application provide a control method applied to the wearable display device of the first aspect, and the control method comprises:

[0013] In the case of displaying the environmental picture, the pupil position of the wearer of the wearable display device is acquired;

[0014] According to the pupil position, a target focus point is determined;

[0015] According to the target focus point, the driving part of the wearable display device is controlled to drive the image sensor to rotate, wherein the shooting focus point of the rotated image sensor matches the target focus point;

[0016] According to the target intersection point, a first display area and a second display area on the display screen are determined, wherein the distance value between the pixel point in the first display area and the target focus point is less than a preset threshold value, and the distance value between the pixel point in the second display area and the target focus point is greater than the preset threshold value;

[0017] The display resolution and / or display brightness of the second display area are controlled to be reduced.

[0018] In a third aspect, the embodiments of the present application provide a control device applied to the wearable display device of the first aspect, and the control device comprises:

[0019] The acquisition module is used for acquiring the pupil position of the wearer of the wearable display device in the case of displaying the environmental picture;

[0020] The determination module is used for determining a target focus point according to the pupil position;

[0021] The control module is used for controlling the driving part of the wearable display device to drive the image sensor to rotate according to the target focus point, wherein the shooting focus point of the rotated image sensor matches the target focus point;

[0022] The determining module is further configured to determine a first display area and a second display area on the display screen according to the target intersection point, wherein a distance value between a pixel point in the first display area and the target focus point is less than a preset threshold, and a distance value between a pixel point in the second display area and the target focus point is greater than the preset threshold.

[0023] The control module is further configured to control the display resolution and / or display brightness of the second display area to be reduced.

[0024] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory storing programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method in the second aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a readable storage medium, the readable storage medium storing programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the method in the second aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to run programs or instructions to implement the steps of the method in the second aspect.

[0027] In a seventh aspect, an embodiment of the present application provides a computer program product, the program product being stored in a storage medium, and the program product being executed by at least one processor to implement the method in the second aspect.

[0028] In the embodiment of the present application, the wearable display device includes a camera assembly, an eyeball sensor and a display screen, and the environment picture captured by the camera assembly is displayed through the display screen, wherein the eyeball sensor can monitor the pupil position of the wearing user, so as to determine the line-of-sight focus position of the current user, i.e., the target focus point, according to the pupil position.

[0029] The camera assembly comprises an image sensor and a driving part. When the user's visual focus point is consistent with the shooting focus point of the image sensor, the driving part keeps the image sensor still. When the user's eyeball rotates and the visual focus point moves, the driving part drives the image sensor to rotate according to the position of the target focus point collected, specifically, drives the image sensor to change the shooting direction, so that the shooting direction is consistent with the user's visual direction, so that the focus point of the image sensor shooting the environmental picture can always be consistent with the user's visual focus point, that is, the target focus point, so that the user's visual focus point can always be in the center position of the environmental picture, thereby avoiding the user's visual focus point falling on the edge area of the environmental picture shot by the camera assembly, that is, the area where the picture may be distorted or distorted, so that the user can always see the clearest picture part, effectively reducing the influence of the lens distortion problem of the camera on the display effect of the extended reality, and dynamically adjusting the picture clarity according to the user's visual focus point, on the basis of ensuring that the picture seen by the user is always a clear picture, reducing screen energy consumption. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 A structural schematic diagram of a wearable display device according to an embodiment of the application is shown;

[0031] Figure 2 A flowchart of a control method according to an embodiment of the application is shown;

[0032] Figure 3 One of the schematic diagrams of the visual angle of the human eye is shown;

[0033] Figure 4 The second one of the schematic diagrams of the visual angle of the human eye is shown;

[0034] Figure 5 The schematic diagram of the visual cone of the human eye at the first time is shown;

[0035] Figure 6 The schematic diagram of the visual cone of the human eye at the second time is shown;

[0036] Figure 7 The conversion schematic diagram of the visual cone to the regular observation body is shown;

[0037] Figure 8 The schematic diagram of the vertex in the camera space in the right-hand coordinate system is shown;

[0038] Figure 9 The schematic diagram of the wearable device displaying static content according to an embodiment of the application is shown;

[0039] Figure 10 The schematic diagram of the wearable device displaying dynamic content according to an embodiment of the application is shown;

[0040] Figure 11 A structural block diagram of a control device according to an embodiment of the application is shown;

[0041] Figure 12 A structural block diagram of an electronic device according to an embodiment of the application is shown;

[0042] Figure 13 A hardware structural schematic diagram of an electronic device according to an embodiment of the application.

[0043] Reference signs:

[0044] 100 wearable display device, 102 body, 1022 wearing part, 104 camera assembly, 1042 image sensor, 1044 driving part, 106 display screen, 108 eyeball sensor, 110 controller. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the application will be clearly described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some, but not all of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the application.

[0046] The terms "first", "second", and the like in the specification and claims of the application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0047] The wearable display device, control method, device, electronic device, and storage medium provided by the embodiments of the application will be described in detail below with reference to the drawings and specific embodiments and application scenarios.

[0048] In some embodiments of the application, a wearable display device is provided, Figure 1 A structural schematic diagram of a wearable display device according to an embodiment of the application is shown, as shown in Figure 1 The wearable display device 100 includes:

[0049] The body 102 includes the wearing part 1022;

[0050] The camera assembly 104 is arranged on the body 102 and includes an image sensor 1042 and a driving part 1044, the image sensor 1042 is configured to capture an environmental image, and the driving part 1044 is configured to drive the image sensor 1042 to rotate relative to the body 102.

[0051] The display screen 106 is arranged in the wearing part 1022, and the display screen 106 is configured to display the environmental image.

[0052] The eye sensor 108 is arranged on the wearing part 1022, and the eye sensor 108 is configured to obtain a pupil position of a wearer of the wearable display device 100.

[0053] The controller 110 is electrically connected with the camera assembly 104 and the eye sensor 108, and is configured to determine a target focal point according to the pupil position, and control the driving part 1044 to drive the image sensor 1042 to rotate according to the target focal point, wherein a shooting focal point of the image sensor 1042 after rotation matches the target focal point; determine a first display area and a second display area on the display screen according to the target intersection point, wherein a distance value between a pixel point in the first display area and the target focal point is less than a preset threshold value, and a distance value between a pixel point in the second display area and the target focal point is greater than the preset threshold value; and control to reduce a display resolution and / or a display brightness of the second display area.

[0054] In the embodiments of the present application, the wearable display device 100 includes an extended reality display device, a virtual reality (VR) display device, an augmented reality (AR) display device, a mixed reality (XR) display device, etc.

[0055] The wearable display device 100 includes a body 102, which can be a glasses body 102, a helmet body 102 or an eyecup body 102, and the body 102 includes a wearing part 1022, such as a glasses frame, a helmet face shield, etc. After the user wears the body 102 of the wearable display device 100 to the head position, at least part of the wearing part 1022 covers the user's eye.

[0056] The wearable device further includes a camera assembly 104 and a display screen 106, the camera assembly 104 is specifically arranged on the body 102 and is arranged towards the line of sight direction of the user, and the number of the camera assembly 104 can be one or more, such as Figure 1 As shown, the camera assembly 104 can be arranged on both sides of the glasses. The camera assembly 104 can capture an environmental image in the current line of sight direction of the user, and the display screen 106 can display the environmental image captured by the camera assembly in real time, so that the user can master the real environmental information through the environmental image displayed on the display screen 106, thereby simulating the real scene of the human eye observing the world.

[0057] When the user wears the wearable device, if the user turns the head, the wearable device and the camera assembly 104 thereon move synchronously with the user's head movement, when the user turns the head to the left, the orientation of the camera assembly 104 is also synchronously turned to the left, and thus the shooting focal point of the camera assembly 104 is synchronous with the user's visual focal point at this time.

[0058] When the user does not turn the head but turns the eyeball, the orientation of the camera assembly 104 does not change, and the shooting focal point does not change, but the user's visual focal point moves, which causes the user's visual focal point, i.e., the focal point of the gaze, to be mismatched with the shooting focal point of the camera assembly 104. The user's visual focal point, i.e., the focal point of the gaze, can deviate from the optimal display area of the environmental image captured by the camera assembly. When the user's visual focal point falls on the edge area of the image, due to the weak ability of the camera to process the focal point outside the quadrant and the lens distortion of the camera itself, the user can see a distorted image, which affects the display effect.

[0059] In view of the above situation, the wearable device 1022 is provided with an eyeball sensor 108, which can collect the position of the user's pupil in real time. The controller 110 can determine the user's visual focal point, i.e., the target focal point, according to the position of the user's pupil collected by the eyeball sensor 108.

[0060] Meanwhile, the camera assembly 104 is provided with an image sensor 1042 and a driving part 1044. The image sensor 1042 can include a sensor body 102 and a lens, and the driving part 1044 can be a motor driving part 1044. The motor driving part 1044 drives the image sensor 1042 to rotate relative to the body 102 of the wearable display device 100, so as to change the orientation of the lens and change the shooting focal point of the image sensor 1042.

[0061] Specifically, when the user's visual focal point is consistent with the shooting focal point of the image sensor 1042, the controller 110 controls the driving part 1044 to be inactive, and the orientation of the image sensor 1042 does not change. When the user's eyeball turns and the visual focal point moves, the controller 110 determines the target focal point according to the collected pupil position, and controls the driving part 1044 to drive the image sensor 1042 to rotate according to the target focal point, specifically to drive the image sensor 1042 to change the shooting direction, so as to make the shooting direction consistent with the visual direction of the user, so that the focal point of the image sensor 1042 for shooting the environmental image can always be consistent with the user's visual focal point, i.e., the target focal point, so that the user's visual focal point can always be in the center position of the environmental image.

[0062] After determining the visual focus of the user, a first display area and a second display area are further determined on the display screen, wherein the first display area is a central area of the display area of the display screen 106, i.e., the area where the visual focus of the user is located, and the second display area is an edge area of the display area of the display screen 106, i.e., the area away from the visual focus of the user.

[0063] Specifically, since the application sets the driving part 1044 and the eyeball sensor 108, the camera assembly 104 can rotate following the pupil position of the user, so that the shooting focus of the camera assembly 104 can always match the visual focus of the user, i.e., the target focus. Since the shooting focus of the camera assembly 104 matches the central area of the environment picture obtained by shooting, the shooting focus and the target focus are always in the central area of the environment picture, i.e., the central area of the display area of the display screen 106, i.e., the first display area.

[0064] Therefore, it can be considered that the visual attention of the user is concentrated in the first display area, and the picture in the second display area is in the "glancing" area of the user.

[0065] For the first display area, i.e., the shooting focus area of the camera assembly 104, which is the area of interest of the user, it is displayed with relatively higher resolution and / or higher brightness, which can ensure the display effect and enable the user to always see a high-definition picture.

[0066] And for the second display area, which is not the area of interest of the user, it is displayed with relatively lower resolution and / or lower brightness, which can effectively reduce the display energy consumption of the display screen 106, thereby improving the endurance level of the wearable display device 100.

[0067] The application embodiment sets the eyeball sensor 108 and the driving part 1044, so that the shooting focus of the camera assembly 104 can follow the visual focus of the user, the visual focus of the user can be kept in the central position of the environment picture shot by the camera assembly 104, and the user can see the clearest picture part shot by the camera assembly 104, effectively reducing the influence of the lens distortion problem of the camera on the display effect of the extended reality, and dynamically adjusting the picture clarity according to the visual focus of the user, on the basis of ensuring that the picture seen by the user is always clear, and reducing the screen energy consumption.

[0068] In the application embodiment, the controller 110 is electrically connected with the display screen 106, and the controller 110 is further configured to determine a display coordinate of a target object, the display content of the display screen 106 includes an environment picture and the target object, and a display position of the target object matches the display coordinate.

[0069] In the embodiment of the present application, the wearable display device 100 can display a virtual target object on the display screen 106 according to the selected program. Specifically, the controller 110 determines the display coordinates of the target object, i.e., the coordinates of the target object on the display screen 106, according to the selected program. The display screen 106 displays the target object while displaying the environmental picture, i.e., combines the virtual target object with the real environmental picture, thereby realizing the effect of the fusion of the "real world" and the "virtual world".

[0070] In the embodiment of the present application, the display area of the display screen 106 includes a first display area and a second display area. The first display area is the central area of the display area of the display screen 106, and the second display area is the edge area of the display area of the display screen 106.

[0071] Specifically, since the present application sets the driving part 1044 and the eyeball sensor 108 to enable the camera assembly 104 to rotate following the pupil position of the user, so that the shooting focus of the camera assembly 104 can always match the line-of-sight focus of the user, i.e., the target focus, and the shooting focus of the camera assembly 104 matches the central area of the environmental picture obtained by shooting, the shooting focus and the target focus are always in the central area of the environmental picture, i.e., the central area of the display area of the display screen 106, i.e., the first display area.

[0072] Therefore, it can be considered that the visual attention of the user is concentrated in the first display area, and the picture in the second display area is in the "peripheral vision" area of the user.

[0073] For the first display area, i.e., the shooting focus area of the camera assembly 104, which is the area of interest of the user, the picture is displayed with relatively higher resolution and / or higher brightness, which can ensure the display effect and enable the user to always see a high-definition picture.

[0074] For the second display area, which is not the area of interest of the user, the picture is displayed with relatively lower resolution and / or lower brightness, which can effectively reduce the display energy consumption of the display screen 106, thereby improving the endurance level of the wearable display device 100.

[0075] In some embodiments of the present application, a control method is provided, which is applied to the wearable display device provided in any of the technical solutions above, Figure 2 A flowchart of the control method according to the embodiment of the present application is shown in FIG. 2. Figure 2 As shown in FIG. 2, the method includes the following steps.

[0076] Step 202: obtaining the pupil position of the wearer of the wearable display device in the case of displaying the environmental picture;

[0077] Step 204, determining a target focus point according to the pupil position;

[0078] Step 206, controlling the driving part of the wearable display device to drive the image sensor to rotate according to the target focus point;

[0079] wherein the shooting focus point of the rotated image sensor matches the target focus point;

[0080] Step 208, determining a first display area and a second display area on the display screen according to the target intersection point;

[0081] wherein the distance value between the pixel points in the first display area and the target focus point is less than a preset threshold value, and the distance value between the pixel points in the second display area and the target focus point is greater than the preset threshold value;

[0082] Step 210, controlling to reduce the display resolution and / or display brightness of the second display area.

[0083] In the embodiments of the present application, the wearable device further comprises a camera assembly and a display screen, the camera assembly can shoot the environment image in the current user's line of sight direction, and the display screen can display the environment image shot by the camera assembly in real time. The user can master the real environment information through the environment image displayed on the display screen, so as to simulate the real scene of human eye observing the world.

[0084] The eyeball sensor is arranged on the wearable part, and the eyeball sensor can collect the position of the user's pupil in real time when the environment picture is displayed. The controller can determine the user's visual focus point, i.e. the target focus point, according to the pupil position collected by the eyeball sensor.

[0085] The camera assembly comprises an image sensor and a driving part, and the driving part drives the image sensor to rotate relative to the body of the wearable display device, so as to change the lens direction, and the line of sight changes the shooting focus point of the image sensor. When the user's eyeball rotates and the line of sight focus point moves, the target focus point is determined according to the collected pupil position, and the driving part is driven to rotate according to the target focus point. Specifically, the image sensor changes the shooting direction, so that the shooting direction is consistent with the user's line of sight direction, so that the focus point of the image sensor shooting the environment picture can always be consistent with the user's line of sight focus point, i.e. the target focus point, so that the user's line of sight focus point can always be in the center position of the environment picture.

[0086] As shown in FIG. 1, Figure 1 At this time, the visual focus point of the pupil of the human eye is in the visual cone area 1. If the camera focus point is perpendicular to the device angle, the camera reverse error is caused by the camera focusing error, the edge lens distortion and other reasons for the part of the area in the visual cone area 1, especially the edge position.

[0087] Specifically,Figure 3 One of the schematic diagrams of the visual angle of the human eye is shown, Figure 4 One of the schematic diagrams of the visual angle of the human eye is shown, Figure 3 And Figure 4 As shown, the maximum visual angle of the human eye is about 220°, and the closer to the 0° area of the standard line of sight, the higher the concentration of the human eye.

[0088] Figure 5 A schematic diagram of the visual cone of the human eye at the first moment is shown, Figure 5 As shown, at the first moment, the visual cone in front of the human eye rendered and displayed by the wearable display device is a, and the rendering angle of the environment picture taken by the camera assembly is a'.

[0089] Figure 6 A schematic diagram of the visual cone of the human eye at the second moment is shown, Figure 6 As shown, when the pupil position of the user moves, the visual cone in front of the human eye at this time is β, and the rendering angle of the environment picture taken by the camera assembly is β'.

[0090] The sequence of real-time rendering of the XR display device in the related art is camera reverse sphere, repeated calculation and 3D implementation modeling. When the pupil position of the user moves, the visual cone angle captured by the pupil visual angle deviates from the rendering angle of the camera. When the deviation is small, the scene rendered by the device does not differ much, but when the deviation angle is too large, the reverse perspective algorithm caused by lens distortion of the camera produces errors, and the 3D object rendered cannot well fit the actual environment.

[0091] In order to solve such problems, by tracking the pupil position of the user's eye, the target focus is determined, and the shooting focus of the camera assembly is synchronously transformed, so that the center of the visual cone always matches the shooting focus of the camera, and therefore the 3D object rendered by the device can well fit the actual environment picture.

[0092] Eye tracking is performed by an eye sensor, which captures the gaze point of the human eye, i.e. the target focus moves to the visual cone region 1, and the pupil position information, including the pupil angle and the pupil movement rate, is transmitted to the controller. The camera assembly is controlled to follow the pupil and rotate synchronously by the driving part, so that the target focus of the pupil of the human eye and the shooting focus of the camera vision are always at the same position, so that the picture (XR virtual object) reversed by the camera is always matched with the visual cone region (real scene) of the human eye.

[0093] Similarly, when the visual focus of the pupil of the human eye moves from the visual cone area 1 to the visual cone area 2, the camera is controlled to move synchronously with the pupil according to the collected pupil position information, so that the target focus of the pupil of the human eye and the shooting focus of the camera are both in the visual cone area 2, which can ensure the rendering accuracy of the camera reverse solution, so that the XR device can always present a refined virtual scene that fits the actual environment and reduces the dizziness of the device on the human body.

[0094] After the visual line focus of the user is determined, a first display area and a second display area are further determined on the display screen, where the first display area is a central area of the display area of the display screen, that is, an area where the visual focus of the user is located, and the second display area is an edge area of the display area of the display screen, that is, an area away from the visual focus of the user.

[0095] Specifically, since the driving part and the eyeball sensor are arranged, the camera assembly can rotate following the pupil position of the user, so that the shooting focus of the camera assembly can always match the visual line focus of the user, that is, the target focus. Since the shooting focus of the camera assembly matches the central area of the environment picture obtained by shooting, the shooting focus and the target focus are always in the central area of the environment picture, that is, the central area of the display area of the display screen, that is, the first display area.

[0096] Therefore, it can be considered that the visual attention of the user is concentrated in the first display area, and the picture in the second display area is in the "peripheral vision" area of the user.

[0097] For the first display area, that is, the shooting focus area of the camera assembly, which is the area of interest of the user, a relatively higher resolution and / or higher brightness are used for display, which can ensure the display effect and enable the user to always see a high-definition picture.

[0098] For the second display area, which is not the area of interest of the user, a relatively lower resolution and / or lower brightness are used for display, which can effectively reduce the display energy consumption of the display screen, thereby improving the endurance level of the wearable display device.

[0099] By collecting the pupil position of the user and controlling the camera assembly to rotate according to the pupil position, the shooting focus of the camera assembly can follow the visual line focus of the user, the visual line focus of the user can be kept in the central position of the environment picture captured by the camera assembly, and the visual line focus of the user can be reduced or avoided from falling into the edge area of the environment picture captured by the camera assembly, which may have picture distortion and picture distortion, so that the user always sees the clearest picture part captured by the camera assembly, effectively reducing the influence of the lens distortion problem of the camera on the display effect of the extended reality, and dynamically adjusting the picture clarity according to the visual line focus of the user, so as to ensure that the picture seen by the user is always clear and reduce the screen energy consumption.

[0100] In some embodiments of the present application, before determining the target focus according to the pupil position, the control method further comprises:

[0101] determining display coordinates of the target object;

[0102] displaying the target object according to the display coordinates, wherein the display content of the wearable display device comprises the environmental picture and the target object.

[0103] In embodiments of the present application, the wearable display device can display a virtual target object on the display screen according to the selected program. Specifically, the controller determines the display coordinates of the target object according to the selected program, which are the coordinates of the target object on the display screen. The display screen displays the target object while displaying the environmental picture, that is, the virtual target object is combined with the real environmental picture, thereby realizing the effect of the fusion of the "real world" and the "virtual world".

[0104] In some embodiments of the present application, determining the display coordinates of the target object comprises:

[0105] obtaining a distance value between the wearer's eyeball and the display screen;

[0106] determining a first transformation matrix according to the distance value and the eye cone coordinates of the target object in the wearer's eye cone;

[0107] determining first projection coordinates of the target object in the plane where the display screen is located according to the first transformation matrix, and determining the display coordinates according to the first projection coordinates.

[0108] In embodiments of the present application, the process of determining the display coordinates, that is, the process of converting a point P on the target object from the user's eye cone to the regular observation volume and obtaining the point P' in the projection plane, wherein the plane where the display screen of the wearable display device is located is the projection plane.

[0109] Specifically, Figure 7 A conversion diagram from the eye cone to the regular observation volume is shown as follows: Figure 7 As shown, the display coordinates of the target object are determined, specifically, the point in the camera space is converted from the view frustum to the canonical view volume (CVV), and the perspective division is performed after the clipping is completed, which specifically includes the perspective matrix multiplication and the perspective division in the algorithm.

[0110] Wherein, the perspective projection transformation is calculated using homogeneous coordinate transformation, including the following steps:

[0111] 1. Use the perspective transformation matrix to transform the vertex from the view frustum to the CVV in the clipping space;

[0112] 2. After the CVV clipping, perspective division is performed.

[0113] The projection relationship can be considered in a fixed direction, Figure 8 A diagram of a vertex in a right-hand coordinate system in camera space is shown as Figure 8 As shown, let P(x, z) be a point after camera transformation, the frustum includes eye position eye, near clipping plane np and far clipping plane fp, let the distance from eye eye to near clipping plane np (i.e. display screen) be N, and the distance from eye eye to far clipping plane be F.

[0114] The projection plane can be any plane parallel to the near clipping plane, for convenience of illustration, the near clipping plane is taken as the projection plane for example. Let P'(x', z') be the point after projection, then z' = -N. By the similar triangle property, the following relationship is obtained:

[0115]

[0116] From the above relationship (1), the P' point of the P point after projection on the projection plane is obtained:

[0117]

[0118] where (x, z) is the coordinate of the point P after camera transformation, (x', z') is the coordinate of the point P' which is the projection of the point P on the projection plane, N is the distance from eye eye to near clipping plane np.

[0119] From the above relationship (2), it can be obtained that z' in the projection result is always equal to -N, therefore z' for the projected P' on the projection plane has no meaning, this information point is useless, but for the 3D graphics pipeline, in order to facilitate subsequent operations, the CVV is combined to operate, which is easier to process, therefore the following relationship is obtained:

[0120]

[0121] where (x, y) is the coordinate of the point P after camera transformation, (x', z') is the coordinate of the point P' which is the projection of the point P on the projection plane, N is the distance from eye eye to near clipping plane np, and a and b are constants.

[0122] According to the above relationship (3), the projection transformation is expressed by matrix and homogeneous coordinate transformation theory, which can be obtained as follows:

[0123]

[0124] Wherein, the initial coordinates of the point P after the camera transformation and the coordinates of the projected point P' are respectively:

[0125]

[0126] The process of obtaining the relationship (5) is called perspective division in the perspective projection process, and the original z value is discarded after this process, and the vertex is considered to be projected. The CVV clipping process is between them, and the CVV is a regular body with the range of (x, y, z) being [-1, 1], which is convenient for polygon clipping. The coefficients a and b can be appropriately selected here, so that the relationship (3) and the relationship (5) are:

[0127]

[0128] The part of is -1 when z = -N, and is 1 when z = -F, thereby constructing the CVV in the z direction, and at this time a and b can be solved:

[0129]

[0130] Wherein, N is the distance from the eye eye to the near clipping plane np, F is the distance from the eye eye to the far clipping plane, and a and b are the coefficients to be solved.

[0131] At this point, after obtaining the values of a and b, the projection matrix can be obtained, which can construct the CVV from the z direction, and in order to be able to change the fixed point from the Frustum case to the CVV case in the x direction and the y direction, define the effective range of -Nx / z as the left boundary value and the right boundary value of the projection plane, respectively recorded as left and right, then -Nx / z belongs to [left, right], and by analogy, -Ny / z belongs to [bottom, top].

[0132] Further, -Nx / z belongs to [left, right] is mapped to x belongs to [-1, 1], and -Ny / z belongs to [bottom, top] is mapped to y belongs to [-1, 1], to obtain the final first transformation matrix:

[0133]

[0134] Wherein, P' is the first projection coordinates, N is the distance from the eye eye to the near clipping plane np, F is the distance from the eye eye to the far clipping plane, z = -N or z = -F, a and b are coefficients, and a and b are constants, left is the left boundary value of the projection plane, right is the right boundary value of the projection plane, top is the upper boundary value of the projection plane, and bottom is the lower boundary value of the projection plane.

[0135] Through the first transformation matrix, the target object can be projected from the human eye visual cone to the CVV plane. If the target object is in the visual cone, the transformed target object is in the CVV. If the target object is outside the visual cone, the transformed target object is outside the CVV.

[0136] The present application determines the first projection coordinates of the target object on the projection plane of the display screen based on the distance value between the eyeball and the display screen and the determined first transformation matrix, and determines the display coordinates of the target object, so as to accurately obtain the display coordinates of the target object, thereby combining the virtual target object into the real environment picture and ensuring the display effect of the wearable display device.

[0137] In some embodiments of the present application, the target object is a moving object. After determining the display coordinates of the target object, the control method further comprises:

[0138] determining whether the target focal point matches the display position of the target object;

[0139] in the case that the target focal point matches the display position, obtaining the motion information of the target object, wherein the motion information includes the motion direction, the motion trajectory and the motion speed;

[0140] determining the predicted focal point according to the motion information and the display coordinates;

[0141] after controlling the driving part of the wearable display device to drive the image sensor to rotate according to the target focal point, the control method further comprises:

[0142] controlling the driving part of the wearable display device to drive the image sensor to rotate according to the predicted focal point, wherein the shooting focal point of the rotated image sensor matches the predicted focal point.

[0143] In the embodiments of the present application, the content displayed by the display screen of the wearable display device includes static content or dynamic content, wherein the static content is the case that the display position of the target object is fixed and unchanged, Figure 9 a schematic diagram of the wearable device displaying static content according to the embodiments of the present application is shown, as shown in Figure 9 the target object is a butterfly 902, and the display position of the butterfly 902 is unchanged, so the image sensor can be normally controlled to rotate according to the pupil position of the user, thereby ensuring that the visual focal point of the user, i.e. the target focal point, coincides with the shooting focal point.

[0144] the dynamic content is the case that the display position of the target object changes according to the programmed motion mode, Figure 10 a schematic diagram of the wearable device displaying dynamic content according to the embodiments of the present application is shown, as shown in Figure 10As shown, the target object is a butterfly 1002, and the butterfly will move from a first position 1004 to a second position 1006 within a set time according to a set movement mode.

[0145] When the user watches the combined picture of the virtual world and the real world through the wearable display device, the user's attention is likely to be on the virtual world, that is, the user can stare at the virtual target object for a long time. When the target object moves, the user's line of sight focus will also move with the target object. When the moving speed of the user's line of sight focus exceeds the upper limit of the moving speed of the camera assembly, the shooting focus may not be able to catch up with the target focus.

[0146] For this case, since the movement mode of the target object is set by the program, the wearable display device can actually "predict" the position of the target object after a period of time. When the user stares at the target object, the position of the target object predicted by the wearable display device after a period of time is also the position of the user's line of sight focus, that is, the position of the target focus after a period of time.

[0147] Specifically, the wearable electronic device determines whether the user's line of sight focus, that is, the target focus, matches the display position of the target object. If the determination result is not matched, it means that the user's line of sight does not follow the movement of the target object, and there is no need to predict the change of the user's line of sight focus at this time.

[0148] If the determination result is matched, it means that the user is staring at the target object. At this time, according to the set program, the movement information of the target object, such as the movement direction, movement trajectory and movement speed of the target object, is obtained. Combined with the movement information of the target object and the current display coordinates of the target object, the display position of the target object after a period of time can be predicted. The predicted display position is also the position of the user's line of sight focus after a period of time, which is recorded as a predicted focus.

[0149] After obtaining the predicted focus, the predicted focus is taken as a target to control the driving part to drive the image sensor to rotate in advance, so as to move the shooting focus of the camera assembly to the predicted focus in advance, thereby ensuring that the user's line of sight focus, that is, the target focus, coincides with the shooting focus of the camera assembly to the greatest extent.

[0150] The embodiment of the present application predicts the position of the user's line of sight focus in advance according to the movement information of the target object, and controls the camera assembly to rotate in advance, so as to reduce the situation that the shooting focus cannot catch up with the user's line of sight focus due to the moving speed of the user's line of sight focus exceeding the moving speed of the camera assembly. Therefore, the user's line of sight focus can be reduced to fall on the edge of the picture, and the influence of lens distortion on the display effect can be reduced.

[0151] In some embodiments of the present application, the predicted focus point is determined according to the motion information and the display coordinates, comprising:

[0152] obtaining a distance value between the wearer's eyeball and the display screen;

[0153] determining a second transformation matrix according to the distance value, the target object's stereomic coordinate in the wearer's stereomic, the motion information and a preset time length;

[0154] determining the target object's second projection coordinate on the plane where the display screen is located after the preset time length according to the second transformation matrix;

[0155] determining the predicted focus point according to the second projection coordinate.

[0156] In the embodiments of the present application, the predicted focus point is specifically predicted by the wearable display device according to the motion information of the target object, and is the display position of the target object after the preset time length. When the user gazes at the target object, the line-of-sight focus point of the user after the preset time length will also change along with the change of the display position of the target object. Therefore, the display position of the target object after the preset time length is also the line-of-sight focus point position of the user after the preset time length, that is, the predicted focus point.

[0157] Specifically, as shown in Figure 10 , it is assumed that at the current time t0, the coordinates of the butterfly in the user's frustum are P(t0), and the coordinates, that is, the display coordinates of the butterfly in the projection area are P'(t0). It is assumed that the preset time length is △t, and the motion speed of the butterfly is v. Then, after the preset time length △t, the coordinates of the butterfly in the user's frustum are P(t0+△t), and the display coordinates are P'(t0+△t).

[0158] According to the first transformation matrix of the above embodiments, it can be known that the relationship of P'(t0) at the current time t0 is:

[0159]

[0160] wherein P'(t0) is the projection coordinate of the target object at t0, N is the distance from the user's eye eye to the near clipping plane np, F is the distance from the eye eye to the far clipping plane, z=-N or z=-F, a and b are coefficients, and a and b are constants, left is the left boundary value of the projection plane, right is the right boundary value of the projection plane, top is the upper boundary value of the projection plane, and bottom is the lower boundary value of the projection plane.

[0161] According to the motion information of the target object, the line-of-sight focus point of the human eye moves along with the motion of the target object, and the predicted focus point at the predicted time point t1 after the preset time length △t is predicted in advance according to the motion path of the target object, wherein t1=t0+△t.

[0162] The second transformation matrix can be determined as follows when the predicted focus point after a preset time length △t is P'(t1):

[0163]

[0164] wherein P'(t1) is the projection coordinate of the target object at t1, △t is the preset time length t1=t0+△t, v is the motion speed of the target object, N is the distance from the eye of the user to the near clipping plane np, F is the distance from the eye of the user to the far clipping plane, z=-N or z=-F, a and b are coefficients, and a and b are constants, left is the left boundary value of the projection plane, right is the right boundary value of the projection plane, top is the upper boundary value of the projection plane, and bottom is the lower boundary value of the projection plane.

[0165] The position of the target object, i.e., the position of the predicted focus point, after the preset time length can be predicted through the second transformation matrix (relationship (9)).

[0166] The embodiments of the present application predict the position of the line-of-sight focus point of the user through the second transformation matrix, thereby controlling the rotation of the image capturing assembly in advance, so as to reduce the situation that the moving speed of the line-of-sight focus point of the user exceeds the motion speed of the image capturing assembly, resulting in that the shooting focus point cannot catch up with the line-of-sight focus point of the user, and thus the line-of-sight focus point of the user can be reduced to fall on the edge of the picture, and the influence of lens distortion on the display effect is reduced.

[0167] The control method provided by the embodiments of the present application can be executed by the control device. The control method executed by the control device is taken as an example to describe the control device provided by the embodiments of the present application.

[0168] In some embodiments of the present application, a control device is provided, Figure 11 The structural block diagram of the control device according to the embodiments of the present application is shown in FIG. 11. Figure 11 As shown in FIG. 11, the control device 1100 includes:

[0169] The acquisition module 1102 is configured to acquire the pupil position of the wearer of the wearable display device in the case of displaying the environmental picture.

[0170] The determination module 1104 is configured to determine the target focus point according to the pupil position.

[0171] The control module 1106 is configured to control the driving part of the wearable display device to drive the image sensor to rotate according to the target focus point, wherein the shooting focus point of the rotated image sensor matches the target focus point.

[0172] The determining module is further configured to determine, according to the target intersection point, a first display region and a second display region on the display screen, wherein a distance value between a pixel point in the first display region and the target focal point is less than a preset threshold, and a distance value between a pixel point in the second display region and the target focal point is greater than the preset threshold.

[0173] The control module is further configured to control the display resolution and / or the display brightness of the second display region to be reduced.

[0174] The embodiments of the present application can collect the pupil position of the user, control the camera assembly to rotate according to the pupil position, make the shooting focal point of the camera assembly follow the visual focal point of the user, keep the visual focal point of the user at the center position of the environment picture shot by the camera assembly, reduce or avoid the visual focal point of the user from falling into the edge region of the environment picture shot by the camera assembly, which may exist picture distortion and picture distortion, make the user always see the clearest picture part shot by the camera assembly, effectively reduce the influence of the lens distortion problem of the camera on the display effect of the extended reality, and dynamically adjust the picture definition according to the visual focal point of the user, ensure that the picture seen by the user is always clear, and reduce the screen energy consumption.

[0175] In some embodiments of the present application, the determining module is configured to determine a display coordinate of the target object;

[0176] The control device further comprises:

[0177] The display module is configured to display the target object according to the display coordinate, wherein the display content of the wearable display device comprises an environment picture and the target object.

[0178] In the embodiments of the present application, the wearable display device can display a virtual target object on the display screen according to the selected program. Specifically, the controller determines the display coordinate of the target object according to the selected program, which is the coordinate of the target object on the display screen. The display screen displays the target object while displaying the environment picture, that is, combines the virtual target object with the real environment picture, thereby realizing the effect of the fusion of the “real world” and the “virtual world”.

[0179] In some embodiments of the present application, the acquisition module is further configured to acquire a distance value between the eyeball of the wearer and the display screen;

[0180] The determining module is further configured to:

[0181] determine a first transformation matrix according to the distance value and the visual cone coordinate of the target object in the visual cone of the wearer;

[0182] determine a first projection coordinate of the target object in the plane where the display screen is located according to the first transformation matrix, and determine the display coordinate according to the first projection coordinate.

[0183] The application determines the first projection coordinate of the target object on the projection plane of the display screen according to the distance value between the eyeball and the display screen and the determined first transformation matrix, and determines the display coordinate of the target object, so that the display coordinate of the target object can be accurately obtained, the virtual target object can be combined into the real environment picture, and the display effect of the wearable display device is ensured.

[0184] In some embodiments of the application, the target object is a moving object, and the control device further comprises:

[0185] The judging module is configured to judge whether the target focal point matches the display position of the target object.

[0186] The obtaining module is further configured to obtain motion information of the target object when the target focal point matches the display position, wherein the motion information comprises a motion direction, a motion trajectory and a motion speed.

[0187] The determining module is further configured to determine a predicted focal point according to the motion information and the display coordinate.

[0188] The control module is further configured to control the driving part of the wearable display device to drive the image sensor to rotate, so that the shooting focal point of the rotated image sensor matches the predicted focal point.

[0189] The embodiments of the application can predict the position of the line-of-sight focal point of the user in advance according to the motion information of the target object, so as to control the camera assembly to rotate in advance, thereby reducing the situation that the moving speed of the line-of-sight focal point of the user exceeds the motion speed of the camera assembly, and the shooting focal point cannot catch up with the line-of-sight focal point of the user, and thus the line-of-sight focal point of the user can be reduced to fall on the edge of the picture, and the influence of lens distortion on the display effect is reduced.

[0190] In some embodiments of the application, the obtaining module is further configured to obtain the distance value between the eyeball of the wearer and the display screen.

[0191] The determining module is further configured to:

[0192] determine a second transformation matrix according to the distance value, the ocular-vertex coordinate of the target object in the ocular vertex of the wearer, the motion information and a preset time length.

[0193] determine the second projection coordinate of the target object on the plane where the display screen is located after the preset time length according to the second transformation matrix.

[0194] determine the predicted focal point according to the second projection coordinate.

[0195] The embodiment of the present application predicts the position of the user's visual focus point through the second transformation matrix, thereby controlling the rotation of the camera assembly in advance, so as to reduce the situation that the user's visual focus point cannot be captured due to the moving speed of the user's visual focus point exceeding the moving speed of the camera assembly, and thus the user's visual focus point falls on the edge of the picture and the influence of lens distortion on the display effect is reduced.

[0196] The control device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than the terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like, and can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiment of the present application is not limited in this regard.

[0197] The control device in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, and the embodiment of the present application is not limited in this regard.

[0198] The control device provided in the embodiment of the present application can implement each process implemented by the method embodiments, and thus details are not described herein again.

[0199] Optionally, the embodiment of the present application further provides an electronic device, Figure 12 A structural block diagram of an electronic device according to the embodiment of the present application is shown in FIG. 12. Figure 12 As shown in FIG. 12, the electronic device 1200 includes a processor 1202, a memory 1204, and a program or instruction stored in the memory 1204 and executable on the processor 1202. The program or instruction is executed by the processor 1202 to implement each process of the above method embodiments and achieve the same technical effects, and thus details are not described herein again.

[0200] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0201] Figure 13 A hardware structure schematic diagram of an electronic device according to an embodiment of the present application.

[0202] The electronic device 1300 includes, but is not limited to, a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, and a processor 1310, etc.

[0203] Those skilled in the art can understand that the electronic device 1300 can also include a power supply (such as a battery) for powering each component, and the power supply can be logically connected to the processor 1310 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. Figure 13 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than shown, or combine certain components, or different component arrangements, which are not described here.

[0204] The processor 1310 is configured to, in a case where a user displays an environmental picture, acquire a pupil position of a wearer of a wearable display device; determine a target focal point according to the pupil position; control a driving part of the wearable display device to drive an image sensor to rotate according to the target focal point, wherein a shooting focal point of the rotated image sensor matches the target focal point; determine a first display area and a second display area on a display screen according to the target intersection, wherein a distance value between a pixel point in the first display area and the target focal point is less than a preset threshold value, and a distance value between a pixel point in the second display area and the target focal point is greater than the preset threshold value; and control to reduce a display resolution and / or a display brightness of the second display area.

[0205] The embodiments of the present application can collect the pupil position of the user, control the camera assembly to rotate according to the pupil position, so that the shooting focal point of the camera assembly can follow the visual focal point of the user, can keep the visual focal point of the user at the center position of the environmental picture shot by the camera assembly, reduce or avoid the visual focal point of the user falling into the edge area of the environmental picture shot by the camera assembly, which mayexist picture distortion, picture distortion, so that the user always sees the clearest picture part shot by the camera assembly, effectively reduces the influence of the lens distortion problem of the camera on the display effect of the extended reality, and can dynamically adjust the picture definition according to the visual focal point of the user, on the basis of ensuring that the picture seen by the user is always clear, reduces the screen energy consumption.

[0206] Optionally, the processor 1310 is further configured to determine display coordinates of the target object; and the display unit 1306 is further configured to display the target object according to the display coordinates, wherein the display content of the wearable display device comprises the environmental picture and the target object.

[0207] In the embodiment of the present application, the wearable display device can display a virtual target object on the display screen according to the selected program. Specifically, the controller determines the display coordinates of the target object according to the selected program, which are the coordinates of the target object on the display screen. The display screen displays the target object while displaying the environmental picture, that is, the virtual target object is combined with the real environmental picture, thereby realizing the effect of the fusion of the "real world" and the "virtual world".

[0208] Optionally, the processor 1310 is further configured to obtain a distance value between the eyeball of the wearer and the display screen; determine a first transformation matrix according to the distance value and the ocular-vertex coordinates of the target object in the ocular-vertex of the wearer; determine the first projection coordinates of the target object on the plane where the display screen is located according to the first transformation matrix, and determine the display coordinates according to the first projection coordinates.

[0209] In the embodiment of the present application, the first projection coordinates of the target object on the projection plane on the display screen are determined based on the distance value between the eyeball and the display screen and the determined first transformation matrix, and the display coordinates of the target object are determined, thereby accurately obtaining the display coordinates of the target object, combining the virtual target object into the real environmental picture, and ensuring the display effect of the wearable display device.

[0210] Optionally, the processor 1310 is further configured to determine whether the target focal point matches the display position of the target object; and in the case that the target focal point matches the display position, obtain motion information of the target object, wherein the motion information comprises a motion direction, a motion trajectory and a motion speed; determine a predicted focal point according to the motion information and the display coordinates; and control the driving unit of the wearable display device to drive the image sensor to rotate according to the predicted focal point, wherein the shooting focal point of the rotated image sensor matches the predicted focal point.

[0211] In the embodiment of the present application, the position of the visual focal point of the user is predicted in advance according to the motion information of the target object, thereby controlling the camera assembly to rotate in advance, so as to reduce the case that the moving speed of the visual focal point of the user exceeds the motion speed of the camera assembly, and the shooting focal point cannot catch up with the visual focal point of the user, thereby reducing the case that the visual focal point of the user falls on the edge of the picture, and reducing the influence of lens distortion on the display effect.

[0212] Optionally, the processor 1310 is further configured to acquire a distance value between the eyeball of the wearer and the display screen; determine a second transformation matrix according to the distance value, the eye-cone coordinate of the target object in the eye-cone of the wearer, the motion information and the preset time length; determine a second projection coordinate of the target object on the plane where the display screen is located after the preset time length according to the second transformation matrix; and determine the predicted focal point according to the second projection coordinate.

[0213] The embodiments of the present application predict the position of the user's visual line focal point through the second transformation matrix, thereby controlling the rotation of the camera assembly in advance, so as to reduce the situation that the moving speed of the user's visual line focal point exceeds the motion speed of the camera assembly, resulting in that the shooting focal point cannot catch up with the user's visual line focal point, and thus the user's visual line focal point can be reduced to fall on the edge of the picture, and the influence of lens distortion on the display effect is reduced.

[0214] It should be understood that in the embodiments of the present application, the input unit 1304 can include a graphics processing unit (GPU) 13041 and a microphone 13042. The graphics processing unit 13041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1306 can include a display panel 13061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1307 includes at least one of a touch panel 13071 and other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 can include two parts of a touch detection device and a touch controller. The other input devices 13072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, an operating rod, and the like, which will not be described here.

[0215] The memory 1309 can be used to store software programs and various data. The memory 1309 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 1309 can include a volatile memory or a non-volatile memory, or the memory 1309 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 1309 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0216] The processor 1310 can include one or more processing units; optionally, the processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1310.

[0217] The embodiments of the present application also provide a readable storage medium, and the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize various processes of the above-mentioned method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0218] The processor is a processor in the electronic device in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disc or an optical disc, and the like.

[0219] The embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is used to run programs or instructions, to realize various processes of the above method embodiments, and to achieve the same technical effects. To avoid repetition, details are not described herein.

[0220] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system or a system-on-chip chip, and the like.

[0221] The embodiment of the present application provides a computer program product. The program product is stored in a storage medium. The program product is executed by at least one processor to realize various processes of the above method embodiments, and to achieve the same technical effects. To avoid repetition, details are not described herein.

[0222] It should be noted that, in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of performing the functions shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0223] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disc, an optical disc), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present application.

[0224] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A wearable display device, comprising: The wearable display device comprises: a body comprising a wearing part; a camera assembly arranged on the body, comprising an image sensor and a driving part, the image sensor being configured to capture an environment picture, and the driving part being configured to drive the image sensor to rotate relative to the body; a display screen arranged in the wearing part, the display screen being configured to display the environment picture; an eyeball sensor arranged on the wearing part, the eyeball sensor being configured to obtain a pupil position of a wearer of the wearable display device; a controller electrically connected with the camera assembly and the eyeball sensor, configured to determine a target focal point according to the pupil position, and control the driving part to drive the image sensor to rotate according to the target focal point, wherein a shooting focal point of the image sensor after rotation matches the target focal point; and determine a first display area and a second display area on the display screen according to the target focal point, wherein a distance value between a pixel point in the first display area and the target focal point is less than a preset threshold value, and a distance value between a pixel point in the second display area and the target focal point is greater than the preset threshold value; and control to reduce display resolution and / or display brightness of the second display area. The controller is electrically connected with the display screen, and the controller is further configured to determine a display coordinate of a target object, the display coordinate being a coordinate of the target object on the display screen, and display content of the display screen comprising the environment picture and the target object, and a display position of the target object matching the display coordinate.

2. A control method applied to the wearable display device according to claim 1, characterized in that, The control method comprises: in a case of displaying an environment picture, obtaining a pupil position of a wearer of the wearable display device; determining a target focal point according to the pupil position; controlling a driving part of the wearable display device to drive an image sensor to rotate according to the target focal point, wherein a shooting focal point of the image sensor after rotation matches the target focal point; determining a first display area and a second display area on the display screen according to the target focal point, wherein a distance value between a pixel point in the first display area and the target focal point is less than a preset threshold value, and a distance value between a pixel point in the second display area and the target focal point is greater than the preset threshold value; and controlling to reduce display resolution and / or display brightness of the second display area.

3. The control method according to claim 2, characterized by, Before the step of determining a target focal point according to the pupil position, the control method further comprises: determining a display coordinate of a target object; and displaying the target object according to the display coordinate, wherein display content of the wearable display device comprises the environment picture and the target object.

4. The control method according to claim 3, characterized by The step of determining a display coordinate of a target object comprises: obtaining a distance value between an eyeball of the wearer and the display screen; determining a first transformation matrix according to the distance value and a visual cone coordinate of the target object in a visual cone of the wearer; and determining a first projection coordinate of the target object in a plane where the display screen is arranged according to the first transformation matrix, and determining the display coordinate according to the first projection coordinate.

5. The control method according to claim 3, characterized by, The target object is a moving object, and after the display coordinates of the target object are determined, the control method further includes: determining whether the target focus point matches the display position of the target object; in the case that the target focus point matches the display position, obtaining motion information of the target object, wherein the motion information includes a motion direction, a motion trajectory, and a motion speed; determining a predicted focus point according to the motion information and the display coordinates; after the driving part of the wearable display device is controlled to drive the image sensor to rotate according to the target focus point, the control method further includes: controlling the driving part of the wearable display device to drive the image sensor to rotate according to the predicted focus point, wherein the shooting focus point of the rotated image sensor matches the predicted focus point.

6. The control method according to claim 5, characterized by The determination of the predicted focus point according to the motion information and the display coordinates includes: obtaining a distance value between the wearer's eyeball and the display screen; determining a second transformation matrix according to the distance value, the eye cone coordinates of the target object in the wearer's eye cone, the motion information, and a preset time length; determining a second projection coordinate of the target object on the plane where the display screen is located after the preset time length according to the second transformation matrix; determining the predicted focus point according to the second projection coordinate.

7. A control device applied to the wearable display device according to claim 1, characterized in that, The control device includes: an obtaining module configured to obtain a pupil position of a wearer of the wearable display device in a case that an environment picture is displayed; a determining module configured to determine a target focus point according to the pupil position; a control module configured to control a driving part of the wearable display device to drive an image sensor to rotate according to the target focus point, wherein a shooting focus point of the rotated image sensor matches the target focus point; The determining module is further configured to determine a first display area and a second display area on the display screen according to the target focus point, wherein a distance value between a pixel point in the first display area and the target focus point is less than a preset threshold value, and a distance value between a pixel point in the second display area and the target focus point is greater than the preset threshold value. The control module is further configured to control to reduce a display resolution and / or a display brightness of the second display area.

8. The control device according to claim 7, wherein the determining module is configured to determine display coordinates of a target object; the control device further includes: a display module configured to display the target object according to the display coordinates, wherein display content of the wearable display device includes the environment picture and the target object.

9. The control device according to claim 8, wherein the obtaining module is further configured to obtain a distance value between an eyeball of the wearer and the display screen; the determining module is further configured to: determine a first transformation matrix according to the distance value and eye cone coordinates of the target object in the wearer's eye cone; determine a first projection coordinate of the target object on the plane where the display screen is located according to the first transformation matrix, and determine the display coordinates according to the first projection coordinate.

10. The control device of claim 8, wherein The target object is a moving object, and the control device further comprises: a judging module configured to judge whether the target focus matches a display position of the target object; the obtaining module is further configured to, in a case where the target focus matches the display position, obtain motion information of the target object, wherein the motion information comprises a motion direction, a motion trajectory, and a motion speed; the determining module is further configured to determine a predicted focus according to the motion information and the display coordinates; the control module is further configured to control a driving unit of the wearable display device to drive an image sensor to rotate, so that a shooting focus of the rotated image sensor matches the predicted focus.

11. The control device of claim 10, wherein: the obtaining module is further configured to obtain a distance value between the wearer's eyeball and the display screen; the determining module is further configured to: determine a second transformation matrix according to the distance value, a visual cone coordinate of the target object in a visual cone of the wearer, the motion information, and a preset time length; determine a second projection coordinate of the target object on a plane where the display screen is located after the preset time length according to the second transformation matrix; determine the predicted focus according to the second projection coordinate.

12. An electronic device, comprising: a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the control method of any one of claims 2 to 6.

13. A readable storage medium, characterized by, programs or instructions are stored on the readable storage medium, and the programs or instructions are executed by the processor to implement the steps of the control method of any one of claims 2 to 6.

Images