Head-mounted device and head-mounted device control method

Abstract

The embodiment of the invention discloses a head-mounted device and a head-mounted device control method, and the method comprises the steps: obtaining the sight line information of a user, determining a relative position parameter and a shooting distance according to the sight line information, and enabling the relative position parameter to be used for representing the relative position of a sight line center and a camera; the shooting distance is the distance between the vertical plane where the fixation area is located and the sight line center, then determining the target angle of the camera according to the shooting distance and the relative position parameter, and further controlling the camera driving device to adjust the shooting angle of the camera according to the target angle, so that the view field of the camera corresponds to the fixation area. Therefore, the shooting angle of the camera can be automatically adjusted based on the sight line of the user so as to ensure that the image shot by the camera is consistent with the picture watched by the user as far as possible.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent wearable devices, and more particularly to a head-mounted device and a head-mounted device control method. Background Art

[0002] With the development of science and technology, more and more wearable smart devices have entered people's daily lives. Head-mounted devices are a common type of wearable smart devices. Head-mounted devices can be, for example, virtual reality (VR) glasses, augmented reality (AR) glasses, or mixed reality (MR) glasses. The camera of this type of head-mounted device is usually set at the edge of the frame or at the connection between the frame and the temples. When the human eye changes its line of sight to observe objects in different positions, the image range captured by the camera of the existing head-mounted device is fixed. There is a situation where the center of the image captured by the camera deviates greatly from the center of the human eye's line of sight, resulting in a decline in user experience. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a head-mounted device and a head-mounted device control method, which are conducive to improving the accuracy of image acquisition of the head-mounted device.

[0004] In a first aspect, an embodiment of the present invention provides a head-mounted device, comprising a camera, a camera driving device and a control circuit; the camera driving device is connected to the camera; the control circuit is electrically connected to the camera and the camera driving device, and the control circuit is configured to: obtain the user's line of sight information, the line of sight information including the user's line of sight center and gaze area; determine a relative position parameter based on the line of sight information, the relative position parameter being used to characterize the relative position of the line of sight center and the camera; determine a shooting distance based on the line of sight information, the shooting distance being the distance between the vertical plane where the gaze area is located and the line of sight center; determine a target angle of the camera based on the shooting distance and the relative position parameter; and control the camera driving device to adjust the shooting angle of the camera based on the target angle so that the field of view of the camera corresponds to the gaze area.

[0005] Furthermore, the head-mounted device also includes an eye tracking component configured to detect the user's line of sight information; wherein the control circuit is configured to obtain the user's line of sight information from the eye tracking component.

[0006] Furthermore, the relative position parameters include a first vertical distance and a first horizontal distance, the first vertical distance being the vertical distance between the center of the line of sight and the camera, and the first horizontal distance being the horizontal distance between the center of the line of sight and the camera; the target angle includes a horizontal target angle and a vertical target angle; determining the target angle of the camera based on the shooting distance and the relative position parameters includes: determining the vertical target angle based on the shooting distance and the first vertical distance; and determining the horizontal target angle based on the shooting distance and the first horizontal distance.

[0007] Furthermore, the determining of the vertical target angle according to the shooting distance and the first vertical distance includes: determining a reference vertical adjustment angle according to the shooting distance and the first vertical distance; when the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera, determining the vertical target angle as a first angle value, the vertical angle adjustment range of the camera is a closed interval, and the first angle value is an endpoint value with the smallest difference between the two endpoint values ​​of the vertical angle adjustment range of the camera and the reference vertical adjustment angle; and when the reference vertical adjustment angle is within the vertical angle adjustment range of the camera, determining the vertical target angle as the reference vertical adjustment angle. Angle; determining the horizontal target angle according to the shooting distance and the first horizontal distance includes: determining a reference horizontal adjustment angle according to the shooting distance and the first horizontal distance; when the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, determining the horizontal target angle as a second angle value, the camera horizontal angle adjustment range is a closed interval, and the second angle value is an endpoint value with the smallest difference from the reference horizontal adjustment angle among the two endpoint values ​​of the camera horizontal angle adjustment range; and, when the reference horizontal adjustment angle is within the camera horizontal angle adjustment range, determining the horizontal target angle as the reference horizontal adjustment angle.

[0008] Furthermore, the control circuit is also configured to: determine the vertical cropping area of ​​the image when the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera; determine the horizontal cropping area of ​​the image when the reference horizontal adjustment angle exceeds the horizontal angle adjustment range of the camera; and crop the image captured by the camera according to the vertical cropping area of ​​the image and / or the horizontal cropping area of ​​the image to obtain a target image so that the center of the target image corresponds to the gaze area.

[0009] In a second aspect, an embodiment of the present invention further provides a head-mounted device control method, wherein the head-mounted device includes a camera and a camera driving device connected to the camera, the method including: obtaining the user's line of sight information, the line of sight information including the user's line of sight center and gaze area; determining a relative position parameter based on the line of sight information, the relative position parameter being used to characterize the relative position of the line of sight center and the camera; determining a shooting distance based on the line of sight information, the shooting distance being the distance between the vertical plane where the gaze area is located and the line of sight center; determining a target angle of the camera based on the shooting distance and the relative position parameter; and controlling the camera driving device to adjust the shooting angle of the camera based on the target angle so that the field of view of the camera corresponds to the gaze area.

[0010] Furthermore, the relative position parameters include a first vertical distance and a first horizontal distance, the first vertical distance being the vertical distance between the center of the line of sight and the camera, and the first horizontal distance being the horizontal distance between the center of the line of sight and the camera; the target angle includes a horizontal target angle and a vertical target angle; determining the target angle of the camera based on the shooting distance and the relative position parameters includes: determining the vertical target angle based on the shooting distance and the first vertical distance; and determining the horizontal target angle based on the shooting distance and the first horizontal distance.

[0011] Furthermore, the determining of the vertical target angle according to the shooting distance and the first vertical distance includes: determining a reference vertical adjustment angle according to the shooting distance and the first vertical distance; when the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera, determining the vertical target angle as a first angle value, the vertical angle adjustment range of the camera is a closed interval, and the first angle value is an endpoint value with the smallest difference between the two endpoint values ​​of the vertical angle adjustment range of the camera and the reference vertical adjustment angle; and when the reference vertical adjustment angle is within the vertical angle adjustment range of the camera, determining the vertical target angle as the reference vertical adjustment angle. Angle; determining the horizontal target angle according to the shooting distance and the first horizontal distance includes: determining a reference horizontal adjustment angle according to the shooting distance and the first horizontal distance; when the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, determining the horizontal target angle as a second angle value, the camera horizontal angle adjustment range is a closed interval, and the second angle value is an endpoint value with the smallest difference from the reference horizontal adjustment angle among the two endpoint values ​​of the camera horizontal angle adjustment range; and, when the reference horizontal adjustment angle is within the camera horizontal angle adjustment range, determining the horizontal target angle as the reference horizontal adjustment angle.

[0012] Furthermore, the method also includes: when the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera, determining the vertical cropping area of ​​the image; when the reference horizontal adjustment angle exceeds the horizontal angle adjustment range of the camera, determining the horizontal cropping area of ​​the image; and, according to the vertical cropping area of ​​the image and / or the horizontal cropping area of ​​the image, cropping the image captured by the camera to obtain a target image so that the center of the target image corresponds to the gaze area.

[0013] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer program instructions, wherein the computer program implements the method described in the second aspect when executed by a processor.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer program product, comprising instructions, which, when executed on a head-mounted device, causes the head-mounted device to execute the method described in the second aspect.

[0015] Embodiments of the present invention provide a head-mounted device and a method for controlling the head-mounted device. The method obtains a user's line of sight information and determines a relative position parameter and a shooting distance based on the line of sight information. The relative position parameter represents the relative position between the line of sight center and the camera, and the shooting distance is the distance between the vertical plane where the gaze area is located and the line of sight center. The method then determines the target angle of the camera based on the shooting distance and the relative position parameter. The method then controls a camera driver to adjust the camera's shooting angle based on the target angle so that the camera's field of view corresponds to the gaze area. Thus, the camera's shooting angle can be automatically adjusted based on the user's line of sight to ensure that the image recorded by the camera remains as consistent as possible with the image viewed by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0017] Figure 1 is a schematic structural diagram of a head-mounted device according to an embodiment of the present invention;

[0018] Figure 2 is a schematic block diagram of a head mounted device according to an embodiment of the present invention;

[0019] Figure 3 1 is a schematic diagram of adjusting the vertical shooting angle of a camera of a head mounted device according to an embodiment of the present invention;

[0020] Figure 4 1 is a schematic diagram of adjusting the horizontal shooting angle of a camera of a head mounted device according to an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of a head mounted device according to an embodiment of the present invention cropping an image captured by a camera to obtain a target image;

[0022] Figure 6 is a schematic diagram of a head mounted device control method according to an embodiment of the present invention;

[0023] Figure 7 is a schematic diagram of a head mounted device control method according to another embodiment of the present invention;

[0024] Figure 8 is a schematic diagram of a head mounted device control method according to another embodiment of the present invention;

[0025] Figure 9 2 is a schematic diagram of a head mounted device control method according to another embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present application is described below based on the following embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. To avoid obscuring the essence of the present application, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0027] Furthermore, persons of ordinary skill in the art will appreciate that the figures provided herein are for illustration purposes only and are not necessarily drawn to scale.

[0028] Unless the context clearly requires otherwise, words like “include”, “comprising” and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0029] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0030] One aspect of an embodiment of the present invention relates to a head mounted device 1, referring to Figure 1 and Figure 2 The head-mounted device 1 includes a camera 10 for capturing images. The head-mounted device 1 includes a device body 50, on which the camera 10 is disposed for capturing images of objects observed from the user's perspective. The head-mounted device 1 may or may not have a display component for displaying images to the user. If a display component is present, the display may be used to display the field of view of the camera 10 or for other utility purposes.

[0031] The head-mounted device 1 includes a control circuit 30, which is electrically connected to the camera 10 and can control the camera 10. The control circuit 30 may include a processor for controlling the overall operation of the smart glasses and may include one or more processing units. For example, the processor may include at least one of a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU), a video codec, a digital signal processor (DSP), a baseband processor, and a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may generate operation control signals based on instruction opcodes and timing signals to control the acquisition and execution of instructions. The processor may also include a memory for storing instructions and data. The processor executes the instructions stored in the memory to execute various functional applications and data processing of the smart glasses. In some embodiments, the memory is a cache memory.

[0032] In some application scenarios, the head-mounted device 1 may be smart glasses, such as virtual reality (VR) glasses, augmented reality (AR) glasses, or mixed reality (MR) glasses. The device body 50 of the head-mounted device 1 includes a display device. For example, the display device may include a waveguide and a projection mechanism. The waveguide is a flat optical element based on a transparent substrate (such as glass or polymer material) with a diffraction grating or geometric reflective structure integrated inside or on its surface. The control circuit 30 controls the projection mechanism to couple the emitted imaging light into the waveguide. The imaging light is replicated and expanded horizontally or vertically within the waveguide and then coupled out to the human eye to form a virtual image. In addition, the user can observe the environment and objects in front of the line of sight through the waveguide. The projection mechanism may include a display chip and a lens assembly. The display chip generates high-resolution images through pixel-level modulation. The projection lens assembly may be composed of multiple aspherical lenses or free-form surface lenses, which are used to relay the image plane of the display chip to the coupling area of ​​the waveguide. The display chip may include liquid crystal on silicon, digital micromirror or microdisplay.

[0033] In an embodiment of the present invention, the control circuit 30 obtains the user's line of sight information, wherein the line of sight information includes the user's line of sight center 3 and gaze area 4. In an embodiment of the present invention, the user's line of sight center 3 can be the center of the user's two eyes 2, and the gaze area 4 is the area where the user's line of sight is focused and concerned. In some embodiments, the head-mounted device 1 also includes an eye tracking component 40, which is provided in the device body 50 and electrically connected to the control circuit 30. The control circuit 30 can obtain the user's eye movement status, pupil position, etc. through the eye tracking component 40, and then determine the line of sight information. In an optional embodiment, the eye tracking component 40 may include an infrared light emitter and a sensing unit. The infrared light emitter emits detection infrared light, and the detection infrared light irradiates the surface of the eyeball and is reflected by the cornea and pupil to form reflected infrared light. The detection infrared light is invisible light with a wavelength between 850nm and 940nm, which can avoid interfering with the user's visual experience. The sensing unit may include an infrared camera 10, which can capture the displacement changes of the reflected infrared light C formed by the cornea and pupil reflection, so that the processor of the eye tracking component 40 can calculate the user's pupil position and size based on a specific gaze estimation algorithm, and then determine the user's line of sight center 3 and gaze area 4.

[0034] In this embodiment of the present invention, the head-mounted device 1 includes a camera driver 20 connected to a camera 10. A control circuit 30 is electrically connected to the camera driver 20. Under the control of the control circuit 30, the camera driver 20 can drive the camera 10 to deflect relative to the device body 50, thereby adjusting the field of view of the camera 10. The camera driver 20 may include a motor, a transmission mechanism disposed between the motor and the camera 10, or other necessary components. Its specific structure can be selected based on the specific structure of the head-mounted device 1 and the desired usage scenario.

[0035] The control circuit 30 determines a relative position parameter and a shooting distance based on the line of sight information. The relative position parameter represents the relative position between the line of sight center 3 and the camera 10, and the shooting distance is the distance between the vertical plane containing the gaze area 4 and the line of sight center 3. Based on the shooting distance and relative position parameter, the control circuit 30 determines the target angle of the camera 10 and then controls the camera driver 20 to adjust the shooting angle of the camera 10 based on the target angle so that the field of view of the camera 10 corresponds to the gaze area 4. For example, when the current angle of the camera 10 differs from the target angle, the control circuit 30 controls the camera driver 20 to adjust the angle of the camera 10 from the current angle to the target angle so that the central axis 5 of the field of view of the camera 10 intersects with or is as close as possible to the center of the gaze area 4, thereby making the imaging effect of the camera 10 more similar to the perspective seen by the user's eye 2. In this embodiment of the present invention, the field of view of the camera 10 corresponding to the gaze area 4 may mean that the central axis 5 of the field of view of the camera 10 intersects with or is as close as possible to the center of the gaze area 4.

[0036] In some embodiments, the camera driving device 20 can drive the camera 10 to adjust the angle in the horizontal direction and the vertical direction. The relative position parameter includes a first vertical distance and a first horizontal distance, wherein, with reference to Figure 3 and Figure 4 , the first vertical distance is the vertical distance between the sight center 3 and the camera 10, and the first horizontal distance is the horizontal distance between the sight center 3 and the camera 10. The target angle includes a horizontal target angle and a vertical target angle. The control circuit 30 determines the vertical target angle according to the shooting distance and the first vertical distance, and determines the horizontal target angle according to the shooting distance and the first horizontal distance. The control circuit 30 adjusts the horizontal angle of the camera 10 according to the horizontal target angle, and adjusts the horizontal angle of the camera 10 according to the horizontal target angle. For example, referring to Figure 3 When the vertical angle adjustment range of the camera 10 can meet the adjustment requirements, the vertical target angle α'=arctan(h / d), where h is the first vertical distance and d is the shooting distance. Figure 4 When the horizontal angle adjustment range of the camera 10 can meet the adjustment requirements, the horizontal target angle β'=arctan(s / d), where s is the first horizontal distance and d is the shooting distance.

[0037] Since the vertical angle (pitch angle) adjustment of the camera 10 is subject to certain angular range restrictions, and the horizontal angle adjustment is also subject to certain angular range restrictions, the vertical target angle and the horizontal target angle should be within the angular adjustment range of the camera 10. In one embodiment, a reference vertical adjustment angle can be determined based on the shooting distance and the first vertical distance. The reference vertical adjustment angle refers to the vertical angle of the camera 10 when the central axis 5 of the field of view of the camera 10 intersects the center of the gaze area 4, assuming that the vertical angle adjustment of the camera 10 is not subject to angular range restrictions. The vertical angle adjustment range of the camera can be a closed interval. When the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera, the vertical target angle is determined as the first angle value; when the reference vertical adjustment angle is within the vertical angle adjustment range of the camera, the vertical target angle is determined as the reference vertical adjustment angle. The first angle value is the endpoint value of the vertical angle adjustment range of the camera with the smallest difference from the reference vertical adjustment angle. For example, if the vertical angle adjustment range of the camera is [-50°, 50°] and the reference vertical adjustment angle is 53°, the vertical target angle can be determined as the endpoint value (first angle value) with the smallest difference between the two endpoints of the vertical angle adjustment range and the reference vertical adjustment angle, that is, the vertical target angle can be determined as 50°.

[0038] Similarly, a reference horizontal adjustment angle can be determined based on the shooting distance and the first horizontal distance. A determination can then be made as to whether the reference horizontal adjustment angle exceeds the camera's horizontal angle adjustment range, thereby determining the horizontal target angle. The reference horizontal adjustment angle refers to the horizontal angle of the camera 10 when the central axis 5 of the camera's field of view intersects the center of the gaze area 4, assuming that the camera's horizontal angle adjustment range is not restricted. The camera's horizontal angle adjustment range can be a closed interval. When the reference horizontal adjustment angle is within the camera's horizontal angle adjustment range, the horizontal target angle is determined as the reference horizontal adjustment angle. When the reference horizontal adjustment angle exceeds the camera's horizontal angle adjustment range, the horizontal target angle is determined as a second angle value. The second angle value is the endpoint value of the camera's horizontal angle adjustment range that has the smallest difference from the reference horizontal adjustment angle. For example, if the camera's horizontal angle adjustment range is [-35°, 35°] and the reference horizontal adjustment angle is 40°, the horizontal target angle can be determined as the endpoint value (the second angle value) of the horizontal angle adjustment range that has the smallest difference from the reference horizontal adjustment angle. That is, the horizontal target angle can be determined as 35°.

[0039] In this embodiment, referring to Figure 3 , the reference vertical adjustment angle can be calculated according to the following formula: α = arctan (h / d), where α is the reference vertical adjustment angle, h is the first vertical distance, and d is the shooting distance. Figure 4 The horizontal reference angle can be calculated using the following formula: β = arctan(s / d), where β is the reference horizontal adjustment angle, s is the first horizontal distance, and d is the shooting distance. For example, when the first vertical distance h = 0.8 cm, the shooting distance d = 200 cm, and the first horizontal distance s = 7 cm, the reference vertical adjustment angle α = arctan(h / d) = arctan(0.8 / 200) ≈ 0.229°, and the reference horizontal adjustment angle β = arctan(s / d) = arctan(7 / 200) ≈ 2.005°.

[0040] In some embodiments, the image captured by the camera 10 can be cropped to adjust the center of the image, so that the center of the obtained image is closer to the center of the gaze area 4, thereby providing the user with a more accurate visual effect. Optionally, if the angle adjustment range of the camera 10 is not sufficient to ensure that the central axis 5 of the camera's field of view intersects the center of the gaze area 4 by adjusting the angle of the camera 10, the control circuit 30 can adjust the image captured by the camera 10 by cropping it to ensure the imaging quality of the obtained image as much as possible.

[0041] In one embodiment, when the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera, the control circuit 30 determines the image vertical cropping area SY. When the reference horizontal adjustment angle exceeds the horizontal angle adjustment range of the camera, the control circuit 30 determines the image horizontal cropping area SX. Then, the image captured by the camera 10 is cropped according to the determined image vertical cropping area SY and / or image horizontal cropping area SX to obtain the target image so that the center of the target image corresponds to the gaze area 4. For example, referring to Figure 5 In the upper part, after determining the target angle of the camera 10, when the camera 10 is at the target angle, the center of the image A captured by the camera 10 is C1, and the center of the gaze area 4 is located at C2 in the image. C1 and C2 do not overlap. At this time, the vertical cropping area SY and the horizontal cropping area SX for cropping the image A can be determined based on the coordinates of C1 and C2. Figure 5 In the lower part of the image, the center of the cropped target image B coincides with the center of the gaze area 4 in the image. For example, a plane rectangular coordinate system xOy is established with the lower left corner of image A as the coordinate origin O. The coordinates of C1 are (20, 15) and the coordinates of C2 are (22, 12). Then, the horizontal cropping area SY of the image can be determined as a strip area with x coordinates from 0 to 2, and the vertical cropping area SX of the image can be determined as a strip area with y coordinates from 12 to 15.

[0042] In another aspect of the present invention, a method for controlling a head-mounted device is also provided. The head-mounted device 1 includes a camera 10 and a camera driving device 20 connected to the camera 10. The camera driving device 20 can drive the camera 10 to deflect. In some application scenarios, the head-mounted device control method can be applied to control the head-mounted device 1 in at least some of the above embodiments, and can be executed by the control circuit 30 of the head-mounted device 1. Figure 6 The method for adjusting the combined image distance includes the following steps S100 to S500:

[0043] Step S100: Acquire user's sight line information.

[0044] The line of sight information includes the user's line of sight center 3 and gaze area 4. The user's line of sight center 3 is the center of the user's eyes, and the gaze area 4 is the area where the user's line of sight is focused and attentive. In some embodiments, the head-mounted device 1 also includes an eye tracking component 40, which is disposed in the device body 50 and electrically connected to the control circuit 30. The control circuit 30 can obtain the user's eye movement status, pupil position, etc. through the data collected by the eye tracking component 40, and then determine the line of sight information.

[0045] Step S200: Determine relative position parameters according to line of sight information.

[0046] The relative position parameter is used to characterize the relative position between the sight center 3 and the camera 10. Optionally, the relative position parameter includes a first vertical distance and a first horizontal distance, wherein, with reference to Figure 3 and Figure 4 The first vertical distance is the vertical distance between the center of sight 3 and the camera 10 , and the first horizontal distance is the horizontal distance between the center of sight 3 and the camera 10 .

[0047] Step S300: Determine the shooting distance according to the line of sight information.

[0048] The shooting distance is the distance between the vertical plane where the gaze area 4 is located and the line of sight center 3, or in other words, the horizontal distance between the user and the gaze area 4. The user's line of sight focus position can be determined based on the line of sight information, thereby determining the shooting distance.

[0049] Step S400: Determine the target angle of the camera according to the shooting distance and relative position parameters.

[0050] In one embodiment, the camera driving device 20 can drive the camera 10 to adjust the angle in the vertical and horizontal directions. The target angle includes a horizontal target angle and a vertical target angle. The control circuit 30 determines the vertical target angle according to the shooting distance and the first vertical distance, and determines the horizontal target angle according to the shooting distance and the first horizontal distance. The relative position parameters include the first vertical distance and the first horizontal distance. The first vertical distance is the vertical distance between the line of sight center 3 and the camera 10. The first horizontal distance is the horizontal distance between the line of sight center 3 and the camera 10. The target angle includes the horizontal target angle and the vertical target angle. Figure 7 , step S400 may include the following steps S410 and S420:

[0051] Step S410: Determine a vertical target angle according to the shooting distance and the first vertical distance.

[0052] For example, refer to Figure 3 When the vertical angle adjustment range of the camera 10 can meet the adjustment requirements, the vertical target angle α′=arctan(h / d), where h is the first vertical distance and d is the shooting distance.

[0053] Step S420: Determine the horizontal target angle according to the shooting distance and the first horizontal distance.

[0054] For example, refer to Figure 4 When the horizontal angle adjustment range of the camera 10 can meet the adjustment requirements, the horizontal target angle β'=arctan(s / d), where s is the first horizontal distance and d is the shooting distance.

[0055] In some scenarios, since the vertical angle (pitch angle) adjustment of the camera 10 has a certain angle range limitation, and the horizontal angle adjustment also has a certain angle range limitation, the vertical target angle and the horizontal target angle should be within the angle adjustment range of the camera 10. In one embodiment, referring to Figure 8 Step S410 may include the following steps S411 to S414:

[0056] Step S411: Determine a reference vertical adjustment angle according to the shooting distance and the first vertical distance.

[0057] The reference vertical adjustment angle refers to the vertical angle of the camera 10 when the field of view center axis 5 of the camera 10 intersects the center of the gaze area 4, assuming that the vertical angle adjustment of the camera 10 has no angle range limit. Figure 3 The reference vertical adjustment angle can be calculated according to the following formula: α=arctan(h / d), where α is the reference vertical adjustment angle, h is the first vertical distance, and d is the shooting distance.

[0058] Step S412: Determine whether the reference vertical adjustment angle is within the camera vertical angle adjustment range.

[0059] The camera vertical angle adjustment range can be a closed interval.

[0060] Step S413: When the reference vertical adjustment angle exceeds the camera vertical angle adjustment range, the vertical target angle is determined as the first angle value.

[0061] The first angle value is the endpoint value of the vertical angle adjustment range of the camera with the smallest difference from the reference vertical adjustment angle. For example, with a horizontal plane parallel to the front of the head mounted device 1 as a reference, the vertical angle adjustment range of the camera is [-50°, 50°], and the reference vertical adjustment angle is 53°. Then, the vertical target angle can be determined as the endpoint value (the first angle value) of the vertical angle adjustment range with the smallest difference from the reference vertical adjustment angle, that is, the vertical target angle can be determined as 50°.

[0062] Step S414: When the reference vertical adjustment angle is within the camera vertical angle adjustment range, the vertical target angle is determined as the reference vertical adjustment angle.

[0063] When the reference vertical adjustment angle is within the camera vertical angle adjustment range, the vertical target angle α′ is equal to the reference vertical adjustment angle α.

[0064] In one embodiment, referring to Figure 8 Step S420 may include the following steps S421 to S424:

[0065] Step S421: Determine a reference horizontal adjustment angle according to the shooting distance and the first horizontal distance.

[0066] The reference horizontal adjustment angle refers to the horizontal angle of the camera 10 when the field of view center axis 5 of the camera 10 intersects the center of the gaze area 4, assuming that the horizontal angle adjustment of the camera 10 has no angle range limit. Figure 4 The horizontal reference angle can be calculated according to the following formula: β = arctan (s / d), where β is the reference horizontal adjustment angle, s is the first horizontal distance, and d is the shooting distance.

[0067] Step S422: Determine whether the reference vertical adjustment angle is within the camera vertical angle adjustment range.

[0068] The camera horizontal angle adjustment range can be a closed interval.

[0069] Step S423: When the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, the horizontal target angle is determined as a second angle value.

[0070] The second angle value is the endpoint value of the camera's horizontal angle adjustment range that has the smallest difference from the reference horizontal adjustment angle. For example, with a vertical plane parallel to the front of the head-mounted device 1 as a reference, the camera's horizontal angle adjustment range is [-35°, 35°], and the reference horizontal adjustment angle is 40°. The horizontal target angle can be determined as the endpoint value (the second angle value) of the horizontal angle adjustment range that has the smallest difference from the reference horizontal adjustment angle, that is, the horizontal target angle can be determined as 35°.

[0071] Step S424: When the reference horizontal adjustment angle is within the camera horizontal angle adjustment range, the horizontal target angle is determined as the reference horizontal adjustment angle.

[0072] When the reference vertical adjustment angle is within the camera vertical angle adjustment range, the horizontal target angle β' is equal to the reference horizontal adjustment angle β.

[0073] Step S500: Control the camera driving device to adjust the shooting angle of the camera according to the target angle so that the field of view of the camera corresponds to the gaze area.

[0074] Exemplarily, when the current angle of the camera 10 is different from the target angle, the control circuit 30 controls the camera driving device 20 to adjust the angle of the camera 10 from the current angle to the target angle, so that the field of view center axis 5 of the camera 10 intersects with or is as close as possible to the center of the gaze area 4, so that the gaze area 4 is located in the center of the image recorded by the camera 10, so that the imaging effect of the camera 10 is closer to the perspective seen by the user's eyes 2.

[0075] In one embodiment, the head-mounted device control method can further adjust the position of the image center by cropping the image captured by the camera 10, so that the center of the obtained image is closer to the center of the gaze area 4, thereby providing the user with a more accurate visual effect. Optionally, if the angle adjustment range of the camera 10 cannot meet the requirement of adjusting the angle of the camera 10 so that the central axis 5 of the field of view of the camera 10 intersects with the center of the gaze area 4, the control circuit 30 can adjust the image captured by the camera 10 by cropping it to ensure the imaging quality of the obtained image as much as possible. Figure 9is a flowchart of a head mounted device control method according to another embodiment of the present invention, wherein steps S100, S200, S300, S411, S412, S413, S414, S421, S422, S423, S424, and S500 are the same as those described above. Figure 8 The corresponding steps described can be the same. Figure 9 After step S413, the following steps S415 and S416 are also included:

[0076] Step S415: Determine the vertical cropping area of ​​the image.

[0077] When the reference vertical adjustment angle exceeds the camera vertical angle adjustment range, the control circuit 30 determines the image vertical cropping area SY. Figure 5 In the upper portion of FIG, after determining the target angle of camera 10, when camera 10 is at the target angle, the center of image A captured by camera 10 is C1, while the center of gaze area 4 is located at C2 in the image. C1 and C2 do not overlap. In this case, the vertical cropping area SY for cropping image A can be determined based on the coordinates of C1 and C2.

[0078] Step S416: crop the image captured by the camera according to the vertical cropping area of ​​the image.

[0079] After step S423, the following steps S425 and S426 are also included:

[0080] Step S425: Determine the horizontal cropping area of ​​the image.

[0081] When the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, the control circuit 30 determines the image horizontal cropping area SX. Figure 5 In the upper portion of the image, after determining the target angle of camera 10, when camera 10 is at the target angle, the center of image A captured by camera 10 is C1, while the center of gaze area 4 is located at C2 in the image. C1 and C2 do not overlap. In this case, the horizontal cropping area SX for cropping image A can be determined based on the coordinates of C1 and C2.

[0082] Step S426 : cropping the image captured by the camera according to the vertical cropping area and the horizontal cropping area to obtain a target image.

[0083] Reference Figure 5In the lower part of the image, after steps S416 and S426, the center of the cropped target image B coincides with the center of the gaze area 4 in the image. For example, a plane rectangular coordinate system xOy is established with the lower left corner of image A as the coordinate origin O. The coordinates of C1 are (20, 15) and the coordinates of C2 are (22, 12). Then, the horizontal cropping area SY of the image can be determined as a strip area with x coordinates from 0 to 2, and the vertical cropping area SX of the image can be determined as a strip area with y coordinates from 12 to 15.

[0084] The technical solution of the embodiment of the present invention obtains the user's line of sight information and determines a relative position parameter and a shooting distance based on the line of sight information. The relative position parameter is used to represent the relative position between the line of sight center 3 and the camera 10, and the shooting distance is the distance between the vertical plane where the gaze area 4 is located and the line of sight center 3. Then, based on the shooting distance and the relative position parameter, the target angle of the camera 10 is determined. Then, based on the target angle, the camera driving device 20 is controlled to adjust the shooting angle of the camera 10 so that the field of view of the camera 10 corresponds to the gaze area 4. In this way, the shooting angle of the camera 10 can be automatically adjusted based on the user's line of sight to ensure that the image recorded by the camera 10 is as consistent as possible with the image viewed by the user.

[0085] It will be understood by those skilled in the art that the embodiments of the present application may be provided as methods, devices (equipment), or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0087] These computer program instructions can also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0088] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, where the computer-readable program is used to enable a computer to execute part or all of the above method embodiments.

[0089] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be implemented by specifying the relevant hardware through a program, and the program is stored in a storage medium and includes a number of instructions for causing a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code. In one application scenario, the non-volatile storage medium storing the above-mentioned computer program product can be part of the control circuit 30 of the head-mounted device 1.

[0090] The foregoing is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application are intended to be within the scope of protection of the present application.

Claims

1. A head-mounted device, characterized in that: include: Camera; A camera driving device, connected to the camera; as well as A control circuit is electrically connected to the camera and the camera driving device, and the control circuit is configured to: Acquire user's sight line information, wherein the sight line information includes the user's sight line center and gaze area; Determining a relative position parameter based on the line of sight information, wherein the relative position parameter is used to represent the relative position of the line of sight center and the camera; Determine a shooting distance according to the sight line information, where the shooting distance is the distance between the vertical plane where the gaze area is located and the center of the sight line; Determining a target angle of the camera according to the shooting distance and the relative position parameter; as well as The camera driving device is controlled to adjust the shooting angle of the camera according to the target angle so that the field of view of the camera corresponds to the gaze area.

2. The head-mounted device according to claim 1, wherein: The head mounted device further comprises: an eye tracking component, configured to detect the user's gaze information; Wherein, the control circuit is configured to obtain the user's line of sight information from the eye tracking component.

3. The head-mounted device according to claim 1, wherein: The relative position parameters include a first vertical distance and a first horizontal distance, wherein the first vertical distance is the vertical distance between the center of the line of sight and the camera, and the first horizontal distance is the horizontal distance between the center of the line of sight and the camera; The target angle includes a horizontal target angle and a vertical target angle; Determining the target angle of the camera according to the shooting distance and the relative position parameter includes: determining the vertical target angle according to the shooting distance and the first vertical distance; as well as The horizontal target angle is determined according to the shooting distance and the first horizontal distance.

4. The head-mounted device according to claim 3, wherein: Determining the vertical target angle according to the shooting distance and the first vertical distance includes: determining a reference vertical adjustment angle according to the shooting distance and the first vertical distance; When the reference vertical adjustment angle exceeds the camera vertical angle adjustment range, determining the vertical target angle as a first angle value, the camera vertical angle adjustment range being a closed interval, and the first angle value being an endpoint value having the smallest difference with the reference vertical adjustment angle between two endpoint values ​​of the camera vertical angle adjustment range; and When the reference vertical adjustment angle is within the camera vertical angle adjustment range, determining the vertical target angle as the reference vertical adjustment angle; Determining the horizontal target angle according to the shooting distance and the first horizontal distance includes: determining a reference horizontal adjustment angle according to the shooting distance and the first horizontal distance; When the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, determining the horizontal target angle as a second angle value, the camera horizontal angle adjustment range being a closed interval, and the second angle value being the endpoint value with the smallest difference between the two endpoint values ​​of the camera horizontal angle adjustment range and the reference horizontal adjustment angle; and When the reference horizontal adjustment angle is within the camera horizontal angle adjustment range, the horizontal target angle is determined as the reference horizontal adjustment angle.

5. The head-mounted device according to claim 4, wherein: The control circuit is further configured to: When the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera, determining a vertical cropping area of ​​the image; When the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, determining the image horizontal cropping area; as well as The image captured by the camera is cropped according to the image vertical cropping area and / or the image horizontal cropping area to obtain a target image, so that the center of the target image corresponds to the gaze area.

6. A head-mounted device control method, characterized in that: The head mounted device includes a camera and a camera driving device connected to the camera, and the method includes: Acquire user's sight line information, wherein the sight line information includes the user's sight line center and gaze area; Determining a relative position parameter based on the line of sight information, wherein the relative position parameter is used to represent the relative position of the line of sight center and the camera; Determine a shooting distance according to the sight line information, where the shooting distance is the distance between the vertical plane where the gaze area is located and the center of the sight line; determining a target angle of the camera according to the shooting distance and the relative position parameter; and The camera driving device is controlled to adjust the shooting angle of the camera according to the target angle so that the field of view of the camera corresponds to the gaze area.

7. The method according to claim 6, characterized in that The relative position parameters include a first vertical distance and a first horizontal distance, wherein the first vertical distance is the vertical distance between the center of the line of sight and the camera, and the first horizontal distance is the horizontal distance between the center of the line of sight and the camera; The target angle includes a horizontal target angle and a vertical target angle; Determining the target angle of the camera according to the shooting distance and the relative position parameter includes: determining the vertical target angle according to the shooting distance and the first vertical distance; as well as The horizontal target angle is determined according to the shooting distance and the first horizontal distance.

8. The method according to claim 7, characterized in that Determining the vertical target angle according to the shooting distance and the first vertical distance includes: determining a reference vertical adjustment angle according to the shooting distance and the first vertical distance; When the reference vertical adjustment angle exceeds the camera vertical angle adjustment range, determining the vertical target angle as a first angle value, the camera vertical angle adjustment range being a closed interval, and the first angle value being an endpoint value having the smallest difference with the reference vertical adjustment angle between two endpoint values ​​of the camera vertical angle adjustment range; and When the reference vertical adjustment angle is within the camera vertical angle adjustment range, determining the vertical target angle as the reference vertical adjustment angle; Determining the horizontal target angle according to the shooting distance and the first horizontal distance includes: determining a reference horizontal adjustment angle according to the shooting distance and the first horizontal distance; When the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, determining the horizontal target angle as a second angle value, the camera horizontal angle adjustment range being a closed interval, and the second angle value being the endpoint value with the smallest difference between the two endpoint values ​​of the camera horizontal angle adjustment range and the reference horizontal adjustment angle; and When the reference horizontal adjustment angle is within the camera horizontal angle adjustment range, the horizontal target angle is determined as the reference horizontal adjustment angle.

9. The method according to claim 8, characterized in that The method further comprises: When the reference vertical adjustment angle exceeds the vertical angle adjustment range of the camera, determining a vertical cropping area of ​​the image; When the reference horizontal adjustment angle exceeds the camera horizontal angle adjustment range, determining an image horizontal cropping area; and The image captured by the camera is cropped according to the image vertical cropping area and / or the image horizontal cropping area to obtain a target image, so that the center of the target image corresponds to the gaze area.

10. A computer-readable storage medium storing computer program instructions, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 6 to 9 is implemented.

11. A computer program product comprising instructions, characterized in that When the instructions are executed on the head mounted device, the head mounted device is caused to execute the method according to any one of claims 6 to 9.

Images