Imaging method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an imaging method and device, equipment and a storage medium. The method comprises the following steps: determining a target object associated with the gaze of a user in a surrounding environment where the user is located; determining a target depth of the target object relative to a camera based on depth information of the surrounding environment relative to the camera for imaging the surrounding environment; and controlling focusing of the camera based on the target depth to image the target object. Therefore, focusing can be carried out according to the watching requirement of the user. Therefore, the depth of field in a larger range can be realized while high definition is met.

Description

Technical Field

[0001] Example embodiments of the present disclosure relate generally to the field of computers, and more particularly to methods, apparatuses, devices, and computer-readable storage media for imaging. Background Art

[0002] In recent years, technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) have made great progress and have been widely used in various fields. In some application scenarios of these technologies, it is necessary to use a camera to capture images of the real world and present the captured images to the user. The real-world images can be presented to the user together with the virtual images generated by the computer. For example, it can be applied to the field of extended reality (XR), including virtual reality, augmented reality, and mixed reality. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for imaging is provided. The method includes: determining a target object associated with a user's gaze in a surrounding environment where the user is located; determining a target depth of the target object relative to a camera that images the surrounding environment based on depth information of the surrounding environment relative to the camera; and controlling the focus of the camera based on the target depth to image the target object.

[0004] In a second aspect of the present disclosure, a wearable device is provided. The wearable device includes: a camera configured to image a surrounding environment in which a user is located; and a controller configured to: determine a target object in the surrounding environment associated with the user's gaze; determine a target depth of the target object relative to the camera based on depth information of the surrounding environment relative to the camera; and control the focus of the camera based on the target depth to image the target object.

[0005] In a third aspect of the present disclosure, a device for imaging is provided. The device includes: an object determination module configured to determine a target object associated with a user's gaze in a surrounding environment where the user is located; a depth determination module configured to determine a target depth of the target object relative to a camera that images the surrounding environment based on depth information of the surrounding environment relative to the camera; and a focus control module configured to control the focus of the camera based on the target depth to image the target object.

[0006] In a fourth aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1A and Figure 1B Schematic diagrams showing a number of example environments in which embodiments of the present disclosure can be implemented;

[0011] Figure 2A A schematic diagram showing the principle between clarity and depth of field;

[0012] Figure 2B A schematic diagram showing the relationship between spatial resolution and focal length;

[0013] Figure 3 A schematic diagram showing an architecture for performing an auto-focus function according to some embodiments of the present disclosure;

[0014] Figure 4 A flow chart showing a process for imaging according to some embodiments of the present disclosure is shown;

[0015] Figure 5A A schematic diagram showing a preset plane specified by discrete zoom of a camera according to some embodiments of the present disclosure;

[0016] Figure 5B A schematic diagram showing discrete zooming of a camera within a preset plane according to some embodiments of the present disclosure is shown;

[0017] Fig. 6A A schematic diagram showing a target plane for continuous zooming of a camera according to some embodiments of the present disclosure;

[0018] Figure 6B A schematic diagram showing continuous zooming of a camera within a target plane according to some embodiments of the present disclosure is shown;

[0019] Figure 7 A flowchart showing a process of starting a focus function according to some embodiments of the present disclosure is shown;

[0020] Fig. 8A A simulation diagram showing that the auto-focus function of a camera according to some embodiments of the present disclosure is not activated;

[0021] Figure 8B A simulation diagram showing a camera starting an auto-focus function according to some embodiments of the present disclosure is shown;

[0022] Fig. 9 A block diagram showing an apparatus for imaging according to some embodiments of the present disclosure; and

[0023] Fig.10 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0031] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0032] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0033] Example Environment

[0034] Figure 1A Schematic diagrams of several example environments 100 in which embodiments of the present disclosure can be implemented are shown. Figure 1A In this example environment 100, a user 150 wears an electronic device 120 having a camera 140 with a see-through function. The electronic device 120 can capture images of an eye 130-1 and an eye 130-2 (collectively referred to as eyes) of the user 150.

[0035] The controller 110 processes the image collected by the electronic device 120 and outputs the gaze point information. In some embodiments, the controller 110 may be a separate device capable of communicating with the electronic device 120, such as a server, a computing node, etc. for image or data processing.

[0036] In some embodiments, the controller 110 may be integrated with the electronic device 120 or at least coupled with the electronic device 120. For example, the controller 110 may be a module of the electronic device 120 or the controller 110 may be implemented by a processing unit in the electronic device 120.

[0037] refer to Figure 1B , the electronic device 120 may include multiple cameras 140, namely the cameras 140 with perspective function described above, which are configured to image the surrounding environment of the user 150. In some embodiments, multiple cameras 140 may be arranged on the electronic device 120 along the direction of the line connecting the two eyes. Multiple cameras 140 are usually arranged in front of the electronic device 120 so as to accurately capture the perspective seen by the user 150. This camera 140 is located in the center of the top of the device or biased to the side, aiming to simulate the position and line of sight of the human eye. Through this position, the camera 140 can record the scene that the user 150 is looking at and transmit it to the background processing unit for analysis and application. Multiple cameras 140 can also arrange the camera 140 at the top or back of the electronic device 120. The top layout can be used to capture information such as the user 150's own activities, gestures or eye movements. The rear layout can be used to record background or environmental changes, monitor surrounding conditions, etc. In some embodiments, multiple perspective cameras 140 can also be used and installed in different positions. By simultaneously acquiring images from multiple angles, more accurate and comprehensive visual perception can be achieved.

[0038] As an example, four cameras 140 are arranged in front of the left eye and the right eye, two at a time, so as to accurately capture the perspective seen by the user 150, aiming to simulate the position and sight direction of the human eye. Of course, this arrangement is not necessary. In some other embodiments, the paired cameras 140 can also be arranged along the direction of eye gaze.

[0039] It should be understood that the plurality of cameras 140 is not limited to Figure 1B Without departing from the principles of the present disclosure, any suitable modification or adjustment may be made to the positions and / or numbers of the plurality of cameras 140.

[0040] In some embodiments, the electronic device 120 may be, for example, a helmet, glasses, or other appropriate head-mounted display device (also referred to as a wearable or portable device) for VR, AR, MR, or XR.

[0041] It should be understood that the above reference Figure 1A and Figure 1B The electronic device 120 described is merely exemplary and is not intended to limit the scope of the present disclosure. Figure 2ATo describe the principle between clarity and depth of field, and reference Figure 2B To describe the principle between spatial resolution and focal length.

[0042] Figure 2A A schematic diagram showing the principle relationship between clarity and depth of field. Figure 2B A schematic diagram showing the relationship between spatial resolution and focal length. Figure 2A , the camera can provide real-time images of the real scene when using a head-mounted display device. Therefore, the perspective function of the head-mounted display device is the key to enabling users to seamlessly switch between virtual and real environments. However, there are multiple challenges and trade-offs in achieving optimal clarity in this perspective function. Current perspective technology has problems in pursuing the collaboration between high clarity, large depth of field, and low noise. First, the relationship between clarity and depth of field (DoF) and signal-to-noise ratio can be analyzed through the following three equations. Equation (1) can calculate the depth of field. Equation (2) can calculate the pixels per degree (PPD) used to characterize spatial resolution. Combining equations (1) and (2) to obtain equation (3), the relationship between depth of field and PPD can be obtained as shown below:

[0043]

[0044]

[0045]

[0046] Among them, Figure 2A As shown in the formula (1), DoF (depth of field) represents the depth of field. u represents the working distance, that is, the distance from the lens to the target object. N represents the F number. It can be seen that DoF and f 2 Inversely proportional (i.e. ). c is the circle of confusion. f is the focal length, also known as the effective focal length (EFL). Since the working distance u is significantly greater than the focal length f, u≈uf. According to equation (1) above, the depth of field is inversely proportional to the square of the focal length. Therefore, increasing the magnification of the camera's lens to improve clarity will reduce the depth of field. This means that while objects at a certain distance will remain sharp, objects at other distances will become blurred.

[0047] like Figure 2B As shown, the PPD in formula (2) represents the number of pixels corresponding to one degree in space. Pixel size represents the pixel size. According to formula (2) above, it can be seen that PPD is inversely proportional to the pixel size. According to formula (3) obtained from formulas (1) and (2) above, it can be seen that the depth of field is inversely proportional to the square of the PPD, and the depth of field is also inversely proportional to the pixel size.

[0048] As mentioned above, the perspective function of the head-mounted display device is the key to enabling users to seamlessly switch between virtual and real environments. The clarity in the perspective function is crucial to the perspective function of the head-mounted display device, which directly affects the user's perception and interaction with virtual and real elements. The PPD metric is usually used to quantify the level of perceptible detail. In order to achieve high clarity, a high-resolution image sensor with sufficient detail capture capability is required, such as a CMOS sensor. It should be understood that the larger the focal length of the camera of the current head-mounted display device, the higher the clarity (also known as accuracy), but the depth of field will inevitably be greatly reduced. This means that even a small change in distance from the lens will cause the object to be out of focus.

[0049] For example, traditional electronic devices significantly reduce the depth of field while pursuing high definition. Depth of field refers to the area inside the image that is considered to be within the acceptable sharp display range, and the depth of field is usually divided into shallow depth of field and deep depth of field. Head-mounted display devices find a balance between pursuing high definition while maintaining sufficient depth of field. A larger depth of field is critical for users to be able to perceive objects at close range (usually about 30 cm) and far away (more than 5 meters). It allows users to seamlessly switch between interacting with virtual objects at close range and perceiving real-world elements at a distance.

[0050] In addition, the camera with a see-through function needs to effectively capture the real-time environment in a short exposure time. In order to obtain a good signal-to-noise ratio and minimize the image noise of the camera, the image noise of the camera can be reduced by increasing the pixel size. However, increasing the pixel size will increase the focal length of the camera lens, which will further reduce the depth of field. Therefore, it has become an urgent problem for the camera with a see-through function on the traditional head-mounted display device to find a balance between the need for a high signal-to-noise ratio and sufficient depth of field.

[0051] In addition, the larger focal length of the camera of current head-mounted display devices makes the camera more susceptible to temperature changes, which can cause focus drift. Temperature changes can cause the lens material of the camera to expand or contract, which can cause the lens focus to change, resulting in focus drift. As the focal length of the camera lens increases to improve clarity and PPD, the depth of focus becomes shallower. At this time, due to the shallower depth of focus, the camera is more susceptible to focus fluctuations caused by temperature changes. Usually, HMDs are used in various temperature environments, which exacerbates the occurrence of focus fluctuations.

[0052] To this end, an embodiment of the present disclosure proposes a scheme for imaging. According to various embodiments of the present disclosure, a camera and a controller are installed on a wearable device with a perspective function. In use, a target object associated with the user's gaze in the surrounding environment in which the user is located (for example, the position and / or size of the target object that the pupil is looking at) is determined. Based on the depth information of the surrounding environment relative to the camera, the target depth of the target object relative to the camera is determined. Then, the focus of the camera is controlled based on the target depth to image the target object. The target object in the embodiment of the present disclosure can be any type of object in the surrounding environment. For example, the object that the user is currently looking at, or the object that the user is predicted to look at, or the object that the user is expected to look at (for example, according to the application scenario, the user is expected to look somewhere, or there is a change in the environment, and the object that the user is expected to notice). The target depth can be expressed as the relative distance between the target object and the camera (such as a lens).

[0053] In the embodiments of the present disclosure, the challenges faced by the perspective technology in wearable devices can be overcome by using the autofocus technology, and a better perspective experience can be provided to the user. It can maximize its potential in various application fields (such as games, education, medical treatment, etc.), and bring unprecedented immersion and comfort to users. In traditional mixed reality head-mounted display devices, there is a trade-off between clarity and depth of field. However, in the embodiments of the present disclosure, by introducing the autofocus technology, this problem can be alleviated, and a sharp perspective experience and a depth of field in a larger range (close and distant scenes) can be achieved. The autofocus technology allows the use of a larger pixel size to improve the signal-to-noise ratio and obtain better image quality. In addition, the perspective cameras used in the current mixed reality head-mounted display devices all use fixed aperture technology. However, the electronic device in the embodiments of the present disclosure can enable a dynamic aperture size adjustment function by introducing an autofocus function, thereby enhancing the photo and video capture capabilities. This allows users to select different depths of field according to the needs of different applications. In addition, in an environment with strong light, the autofocus technology can also better adapt to environmental conditions.

[0054] Continue to refer to Figures 3 to 10 Some example embodiments are described.

[0055] Example Architecture

[0056] Figure 3 FIG. 3 is a schematic diagram of an architecture 300 for performing an auto-focus function according to some embodiments of the present disclosure. The example architecture 300 may be implemented at the electronic device 120. Figure 1A and Figure 1B The example architecture 300 is described.

[0057] like Figure 3As shown, as described above, in order to solve the problem of coordination between clarity, depth of field and signal-to-noise ratio, as well as the focus drift problem existing in traditional mixed reality head-mounted display devices. In the embodiment of the present disclosure, the electronic device 120 adopts a camera 140 that can realize the XR autofocus perspective function / behavior to realize the problem of coordination between clarity, depth of field and signal-to-noise ratio mentioned above. In order to realize the XR autofocus perspective function / behavior, the architecture 300 may include application requirements 310, eye tracking information 320, depth information 330, lens autofocus algorithm 340 and lens autofocus hardware 350.

[0058] In some embodiments, according to application requirements 310, the electronic device 120 needs to obtain the gaze point of the user 150, or the place where the user 150 is expected to gaze, and eye tracking information 320 can be obtained. At the same time, depth information 330 of the gaze point can be obtained. At this time, the camera 140 responds to the depth information 330 to achieve autofocus through the lens autofocus hardware 350. The autofocus process of the lens autofocus hardware 350 is performed through the lens autofocus algorithm 340.

[0059] In some embodiments, the lens autofocus hardware 350 can use different autofocus technologies to enable autofocus. Autofocus can include but is not limited to voice coil motors (VCM), stepper motors, ultrasonic motors (USM), linear motors, piezoelectric motors, liquid lenses, electromagnetic coils, shape memory alloys (SMA), and hybrid technologies (e.g., poLight lenses that combine piezoelectric actuators, deformable membranes, and optical grade soft materials).

[0060] In some embodiments, the lens autofocus algorithm 340 also plays a key role. The algorithms in the disclosed embodiments include but are not limited to contrast detection autofocus (CDAF), phase detection autofocus (PDAF), laser (time of flight sensor) autofocus, and depth map based autofocus.

[0061] In some embodiments, the depth information 330 may be depth information or 3D mapping data of the environment acquired in real time by a 6DoF (6 Degrees Of Freedom) camera or a depth camera, etc. To ensure the robustness of the camera 140, a hybrid autofocus combining multiple algorithms may be required in some cases.

[0062] In addition, during the implementation process, the depth information 330 and the eye tracking information 320 can also be used to indicate that the camera 140 should focus according to the gaze point of the user 150, or predict the object that the user 150 may gaze at in the next few tens of milliseconds, or guide the user 150 to focus on an object.

[0063] Further, in some embodiments, the data of the eye tracking information 320 is obtained by an eye tracking sensor or other sensors (such as speech recognition, brain-computer interface sensors, neural detection sensors). These sensor data can be used to predict the gaze position, so that the camera 140 automatically moves the focus to the predicted object in advance to achieve lower latency.

[0064] In addition, in order to achieve low latency performance of the camera 140, multiple aspects need to be comprehensively optimized during the autofocus process. This includes but is not limited to optimization of the autofocus algorithm 340, improvement of the autofocus hardware 350 technology, optimization of the camera 140 image processing, display rendering, and eye tracking prediction and correction. By combining different autofocus technologies and methods and utilizing depth information and eye tracking signals, the autofocus perspective function can be implemented in the electronic device 120, and the low latency performance of the camera 140 can be ensured through the above-mentioned multi-faceted optimization.

[0065] Example Process

[0066] Figure 4 FIG. 4 is a flowchart of a process 400 for imaging according to some embodiments of the present disclosure. The process 400 may be implemented at the electronic device 120. Figure 1A and 1B The process 400 is described.

[0067] At block 410, the controller 110 determines a target object in the surrounding environment of the user 150 that is associated with the gaze of the user 150. The target object may be of different types. It is understood that the target object may be any type of object in the surrounding environment.

[0068] In some embodiments, the target object may be the object currently being gazed at by the user 150. The controller 110 determines the object currently being gazed at by the user 150 as the target object based on the tracking of the eyes of the user 150. For example, the controller 110 determines the target object based on the data of the eye tracking information 320.

[0069] Alternatively or additionally, in some embodiments, the target object may be an object that the controller 110 predicts the user 150 will look at. The controller 110 may predict the object that the user 150 is currently looking at as the target object based on sensor information associated with the user.

[0070] Alternatively or additionally, in some embodiments, the target object may be an object that the controller 110 guides the user 150 to look at. The controller 110 may determine an object for guiding the user 150 to look at from the surrounding environment as the target object. For example, according to the application scenario, the user 150 is expected to look at a certain place, or there is a change in the environment, and the object that the user 150 is expected to notice will be used as the target object.

[0071] continue Figure 4 In block 420, the controller 110 determines the target depth of the target object relative to the camera 140 based on the depth information of the surrounding environment relative to the camera 140 that images the surrounding environment. It is understood that the target depth can be the relative distance of the target object to the camera 140 (e.g., the lens). For example, the controller 110 determines the target depth of the target object relative to the camera 140 based on the data of the depth information 330.

[0072] At block 430, the controller 110 controls the focus of the camera 140 based on the target depth to image the target object. The electronic device 120 using the imaging method can achieve a greater depth of field while satisfying high definition by utilizing an autofocus technology.

[0073] In some embodiments, the controller 110 may determine a target plane in the surrounding environment that matches the target depth, and then the controller 110 may control the camera 140 to focus on the target plane to image the target object.

[0074] Figure 5A A schematic diagram of a preset plane specified by discrete zoom of a camera according to some embodiments of the present disclosure is shown. Figure 5B FIG. 2 shows a schematic diagram of discrete zooming of a camera within a preset plane according to some embodiments of the present disclosure. Figure 5A and 5B As shown, the target plane can be determined in a discrete manner. Specifically, there can be multiple preset depth ranges, which correspond to multiple preset planes respectively. The controller 110 can determine the preset depth range where the target depth is located from these preset depth ranges. Then, the controller 110 can determine the preset plane corresponding to the determined preset depth range as the target plane. Determine the target plane in the surrounding environment that matches the target depth in the direction of the target object. For example, refer to Figure 5AAn example is described, showing preset planes P1, P2, P3, P4 and P5. In the disclosed embodiment, each preset plane has a corresponding depth range. For example, the depth range 1 corresponding to the preset plane P1 is 1m to 2m, the depth range 2 corresponding to the preset plane P2 is 2m to 3m, the depth range 3 corresponding to the preset plane P3 is 3m to 4m, and the preset planes P4 and P5 are deduced by analogy, which are not specifically limited in the disclosed embodiment. As an example, if the target depth is in the range of 2m to 3m, then the preset plane P2 can be determined as the target plane. The number and depth range of the preset planes listed here are only exemplary.

[0075] Fig. 6A A schematic diagram of a target plane for continuous zooming of a camera according to some embodiments of the present disclosure is shown. Figure 6B FIG. 4 is a schematic diagram showing a camera continuously zooming in a target plane according to some embodiments of the present disclosure. Fig. 6A and Figure 6B As shown, the target plane can be determined by continuous focusing. Then, the controller 100 determines the target plane in the surrounding environment that matches the target depth in the direction of the target object. Figure 6B An example is described, showing some planes in the continuous depth plane, specifically planes P1, P2, P3, P4 ... and P8. In the embodiment of the present disclosure, these planes P1, P2, P3, P4 ... and P8 have corresponding continuous depth ranges. Therefore, a plane with a target depth can be selected as a target plane without approximation as in a discrete scheme.

[0076] In some embodiments, the controller 110 controls the camera 140 to focus on the target plane for imaging. Specifically, the controller 110 determines the configuration information of the lens of the camera 140 based on the depth of the target plane relative to the camera 140. The lens is adjusted based on the configuration information of the lens of the camera 140. It should be understood that the above-described determination of the configuration information of the lens of the camera 140 based on the depth of the target plane relative to the camera 140 is only an example of determining the configuration information, and the scope of the present disclosure is not limited thereto.

[0077] In some embodiments, the controller 110 may obtain the ambient temperature of the lens of the camera 140. The controller 110 may determine temperature compensation information for adjusting the lens based on the ambient temperature and the pre-calibrated focus characteristics of the lens that vary with temperature. The controller 110 may control the focus of the camera 140 based on the target depth. Further, the controller 110 controls the focus of the camera 140 based on the temperature compensation information and the target depth to image the target object. For example, the controller 110 senses the ambient temperature of the lens of the camera 140 based on the data of the temperature sensor.

[0078] The present disclosure plays a vital role in compensating for temperature drift using the autofocus function. The controller 110 continuously monitors the focus and uses a feedback mechanism so that the autofocus function of the camera 140 can adjust the position of the lens to maintain accurate focus under temperature fluctuations. The autofocus function of the camera 140 in the embodiment of the present disclosure can be used to offset the influence of temperature changes on the lens of the camera 140, ensuring that the electronic device 120 and other imaging systems operating in different temperature environments can focus stably and clearly. For example, the autofocus function of the camera 140 can be implemented by an autofocus material. Specifically, the lens of the camera 140 can use a poLight lens with an opposite thermal coefficient (such as refractive index and temperature) to passively compensate for the influence of temperature changes. The lens of the camera 140 can also actively compensate for temperature drift by connecting to a temperature sensor of the camera module. The autofocus function of the lens of the camera 140 can also be implemented by factory calibration or real-time image processing (such as contrast detection autofocus or phase detection autofocus), which is not specifically limited in the present disclosure.

[0079] In some embodiments, the controller 110 controls the focus of the camera 140 based on the target depth. Specifically, the controller 110 determines whether the camera 140 is enabled to provide the user 150 with an image of the surrounding environment.

[0080] Alternatively or nearby, the controller 110 provides an image of the surrounding environment to the user 150 in response to determining that the camera 140 is enabled, and determines whether the auto focus function of the camera 140 is enabled. Further, the controller 110 controls the focus of the camera 140 based on the target depth determined above in response to the auto focus function being enabled. This will be described in detail below.

[0081] Alternatively or nearby, the controller 110 provides an image of the surrounding environment to the user in response to determining that the camera 140 is not enabled. Further, the controller 110 controls the plane focused on by the camera 140 to change within the depth range of the surrounding environment to image or monitor the surrounding environment. This will be described in detail below.

[0082] Figure 7 FIG. 7 is a flow chart showing a process 700 of activating a focus function according to some embodiments of the present disclosure. Figure 7 As shown, in blocks 710 and 720, when the user 150 is using the electronic device 120, the electronic device 120 can disable or enable the perspective function of the camera 140 according to different application requirements (which can be the default HMDs operating environment). Therefore, in block 730, it is first necessary to determine whether to provide the user 150 with an image of the surrounding environment, thereby determining whether the camera 140 is enabled.

[0083] At block 740, when the perspective function of the camera 140 is disabled, it means that the user 150 cannot see the real world in real time (e.g., in 100% immersive virtual reality). Although the camera 140 does not acquire the real scene, the autofocus function of the camera 140 is still active for environmental monitoring and video recording. In this case, the algorithm for autofocus is different from MR. The camera 140 can scan the entire depth of the environment to obtain clear images / videos in the entire scene.

[0084] In block 750 , when the perspective function of the camera 140 is activated, that is, the electronic device 120 enables the MR function (requires the camera 140 to turn on the perspective function).

[0085] In block 760, the camera 140 may disable or enable the auto focus function, depending on the specific application requirements (e.g., only involving near-field objects, or only involving far-field objects) of the electronic device 120 and the maturity and stability of the auto focus. In other words, after the response determines that the camera 140 is enabled, in order to provide the user 150 with an image of the surrounding environment, it is necessary to determine whether the auto focus function of the camera 140 will be enabled.

[0086] At block 770, if the camera 140 disables the auto focus function, the camera 140 is fixed focus. In other words, when it is determined that the camera 140 is not enabled to provide the user 150 with an image of the surrounding environment, the plane focused on by the camera 140 is controlled to change within the depth range of the surrounding environment to image or monitor the surrounding environment.

[0087] At block 780, in the MR application scenario, when auto focus is enabled, it is implemented by hardware and software as well as integration with other sensor signals (including eye tracking, 6DoF camera, depth sensor, IMU, etc.). Therefore, by responding to the auto focus function being enabled, the focus of the camera 140 is controlled based on the target depth, thereby controlling the focus of the camera 140 by the target depth.

[0088] In block 790, there are three ways to implement autofocus of the camera 140. The first is the object that the user 150 is looking at, that is, an autofocus experience similar to "foveated rendering". No matter where the user 150 looks, the camera 140 will focus on the corresponding object with low latency (the user 150 cannot see the focus adjustment). The second is to guide the user 150 to look at another object, that is, the focus is different from the position where the user 150 is looking, so as to guide the user 150's eye focus area according to application requirements. The third is to predict the object that the user 150 will look at, which has been described in detail above and will not be repeated here.

[0089] Fig. 8AFIG801 shows a simulation diagram of a camera not starting the auto focus function according to some embodiments of the present disclosure. Fig. 8A As shown in FIG. 8 , when there is no auto focus function, the camera is in a fixed focus state. Curves 811, 812, and 813 show the change of PDD with the target distance. Fig. 8A As shown in the figure, the clarity gradually decreases as the target distance increases. Fig. 8A It can be seen that when the target distance is close to 1000mm, the pixel size (PPD) will drop rapidly, that is, the clarity will drop rapidly. The range in which clear imaging can be achieved is very narrow.

[0090] Figure 8B FIG802 shows a simulation diagram of a camera starting an auto-focus function according to some embodiments of the present disclosure. Figure 8B As shown in FIG. 8 , when the camera has an auto-focus function, the camera is in an auto-focus state. Curves 821, 822, and 823 show the change of PDD with the target distance. Figure 8B As shown in Figure 2, the clarity can remain consistent as the target distance increases. Figure 8B For example, when the target distance is 1000mm, 2000mm, 3000mm and 4000mm, the pixel size (PPD) is the same, that is, the clarity is consistent. Fig. 8A In comparison, the range in which clear imaging can be achieved is significantly increased.

[0091] As described above, in addition to solving the clarity and depth of field problems, the autofocus function of the camera 140 of the electronic device 120 in the embodiment of the present disclosure also provides other advantages. For example, it can provide active temperature drift compensation to improve the perspective experience and ensure image stability. In addition, this technology can also achieve more efficient display image processing, so that the user 150 can obtain a smoother and more realistic virtual experience.

[0092] Example devices and equipment

[0093] Fig. 9 A schematic structural block diagram of an apparatus 900 for imaging according to some embodiments of the present disclosure is shown. The apparatus 900 may be implemented as or included in the electronic device 120. Each module / component in the apparatus 900 may be implemented by hardware, software, firmware or any combination thereof.

[0094] As shown, the device 900 includes an object determination module 910, which is configured to determine a target object associated with the user's gaze in the surrounding environment where the user is located. The device 900 also includes a depth determination module 920, which is configured to determine a target depth of the target object relative to the camera based on the depth information of the surrounding environment relative to the camera that images the surrounding environment. The device 900 also includes a focus control module 930, which is configured to control the focus of the camera based on the target depth to image the target object.

[0095] In some embodiments, the focus control module 930 is configured to determine a target plane in the surrounding environment that matches the target depth, and control the camera to focus on the target plane to image the target object.

[0096] In some embodiments, the focus control module 930 is configured to determine a plane in the surrounding environment having a target depth relative to the camera as the target plane.

[0097] In some embodiments, the focus control module 930 is configured to determine a preset depth range where the target depth is located from a plurality of preset depth ranges. The plurality of preset depth ranges correspond to a plurality of preset planes respectively. The preset plane corresponding to the determined preset depth range is determined as the target plane.

[0098] In some embodiments, the focus control module 930 is configured to determine configuration information for a lens of the camera based on a depth of the target plane relative to the camera; and adjust the lens based on the configuration information.

[0099] In some embodiments, the object determination module 910 is configured to perform at least one of the following: based on tracking the user's eyes, determining the object that the user is currently looking at as the target object; based on sensor information associated with the user, predicting the object that the user will look at as the target object; or determining an object from the surrounding environment to guide the user's gaze as the target object.

[0100] In some embodiments, the apparatus 900 further includes an ambient temperature acquisition module configured to acquire the ambient temperature that the lens of the camera is in. Based on the ambient temperature and a pre-calibrated focus characteristic of the lens that varies with temperature, temperature compensation information for adjusting the lens is determined, and controlling the focus of the camera based on the target depth to image the target object includes controlling the focus of the camera based on the temperature compensation information and the target depth.

[0101] In some embodiments, the focus control module 930 is configured to determine whether the camera is enabled to provide an image of the surrounding environment to the user; in response to determining that the camera is enabled to provide an image of the surrounding environment to the user, determine whether an autofocus function of the camera is enabled; and in response to the autofocus function being enabled, control the focus of the camera based on the target depth.

[0102] In some embodiments, the device 900 also includes a change control module configured to control the plane focused by the camera to change within the depth range of the surrounding environment in response to determining that the camera is not enabled to provide an image of the surrounding environment to the user to image or monitor the surrounding environment.

[0103] Fig.10 1 shows a block diagram of an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Fig.10 The electronic device 1000 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Fig.10 The electronic device 1000 shown can be used to implement Figure 1A An electronic device 120.

[0104] like Fig.10 As shown, the electronic device 1000 is in the form of a general electronic device. The components of the electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, a memory 1020, a storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. The processing unit 1010 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 1020. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 1000.

[0105] The electronic device 1000 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1020 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1030 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 1000.

[0106] The electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.10As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1020 may include a computer program product 1025 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0107] The communication unit 1040 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 1000 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 1000 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0108] The input device 1050 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 1060 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 1000 may also communicate with one or more external devices (not shown) through the communication unit 1040 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 1000, or communicate with any device that allows the electronic device 1000 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0109] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0110] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0111] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0112] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0113] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0114] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for imaging, comprising: Determining a target object in the user's surroundings that is associated with the user's gaze; Determining a target depth of the target object relative to a camera that images the surrounding environment based on depth information of the surrounding environment relative to the camera; as well as The focus of the camera is controlled based on the target depth to image the target object.

2. The method of claim 1 , wherein controlling the focus of the camera to image the target object based on the target depth comprises: Determine a target plane in the surrounding environment that matches the target depth; as well as The camera is controlled to focus on the target plane to image the target object.

3. The method according to claim 2, wherein determining a target plane in the surrounding environment that matches the target depth comprises: A plane in the surrounding environment having the target depth relative to the camera is determined as the target plane.

4. The method according to claim 2, wherein determining a target plane in the surrounding environment that matches the target depth comprises: Determine a preset depth range in which the target depth is located from a plurality of preset depth ranges, wherein the plurality of preset depth ranges respectively correspond to a plurality of preset planes; as well as A preset plane corresponding to the determined preset depth range is determined as the target plane.

5. The method according to claim 2, wherein controlling the camera to focus on the target plane to image the target object comprises: Determining configuration information for a lens of the camera based on a depth of the target plane relative to the camera; as well as The lens is adjusted based on the configuration information.

6. The method according to claim 1, wherein determining a target object in the surrounding environment of the user that is associated with the user's gaze comprises at least one of the following: Based on the tracking of the user's eyes, determining the object currently being gazed at by the user as the target object, Based on sensor information associated with the user, predicting an object that the user will gaze at as the target object, or An object for guiding the user's gaze is determined from the surrounding environment as the target object.

7. The method according to claim 1, further comprising: Obtaining the ambient temperature of the camera lens; determining temperature compensation information for adjusting the lens based on the ambient temperature and a pre-calibrated temperature-dependent focusing characteristic of the lens, and Controlling the focus of the camera based on the target depth to image the target object includes: A focus of the camera is controlled based on the temperature compensation information and the target depth.

8. The method of claim 1 , wherein controlling the focus of the camera based on the target depth comprises: determining whether the camera is enabled to provide an image of the surrounding environment to the user; In response to determining that the camera is enabled to provide the user with an image of the surrounding environment, determining whether an autofocus function of the camera is enabled; as well as In response to the auto focus function being enabled, focusing of the camera is controlled based on the target depth.

9. The method according to claim 8, further comprising: In response to determining that the camera is not enabled to provide the user with an image of the surrounding environment, the plane on which the camera is focused is controlled to vary within a depth range of the surrounding environment to image or monitor the surrounding environment.

10. A wearable device comprising: A camera configured to image the user's surroundings; and The controller is configured as: Determining a target object in the surrounding environment associated with a gaze of the user; Determining a target depth of the target object relative to the camera based on depth information of the surrounding environment relative to the camera; as well as The focus of the camera is controlled based on the target depth to image the target object.

11. The wearable device according to claim 10, further comprising: a temperature sensor configured to sense the ambient temperature of the camera lens, and The controller is further configured to: determining temperature compensation information for adjusting the lens based on the ambient temperature and a pre-calibrated temperature-dependent focusing characteristic of the lens; and A focus of the camera is controlled based on the temperature compensation information and the target depth.

12. An apparatus for imaging, comprising: An object determination module is configured to determine a target object in the surrounding environment of the user that is associated with the user's gaze; a depth determination module configured to determine a target depth of the target object relative to a camera that images the surrounding environment based on depth information of the surrounding environment relative to the camera; as well as A focus control module is configured to control the focus of the camera based on the target depth to image the target object.

13. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 9.

Images