Method and apparatus for imaging, and device and storage medium

Abstract

According to the embodiments of the present disclosure, provided are a method and apparatus for imaging, and a device and a storage medium. The method comprises: determining a target object associated with a user's gaze in a surrounding environment in which the user is located; on the basis of depth information of the surrounding environment with respect to a camera for performing imaging on the surrounding environment, determining a target depth of the target object with respect to the camera; and on the basis of the target depth, controlling the focusing of the camera, so as to perform imaging on the target object. Thus, the focusing can be realized according to a gaze requirement of the user. This facilitates realizing a depth of field in a wider range while high definition is satisfied.

Description

Method, apparatus, device and storage medium for imaging

[0001] This application claims priority to the Chinese invention patent application entitled “Methods, devices, apparatus and storage media for imaging” and application number 2023112237799, filed on September 20, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Example embodiments of the present disclosure relate generally to the field of computers, and more particularly to methods, apparatuses, devices, and computer-readable storage media for imaging. Background Art

[0003] In recent years, technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) have made significant progress and have been widely used in various fields. In some application scenarios of these technologies, it is necessary to use a camera to capture images of the real world and present the captured images to the user. The real-world images can be presented to the user together with computer-generated virtual images. For example, this can be applied to the field of extended reality (XR), including virtual reality, augmented reality, and mixed reality.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a method for imaging is provided. The method includes: determining a target object in a user's surrounding environment associated with a user's gaze; determining a target depth of the target object relative to a camera imaging the surrounding environment based on depth information of the surrounding environment relative to the camera; and controlling the focus of the camera based on the target depth to image the target object.

[0006] In a second aspect of the present disclosure, a wearable device is provided. The wearable device includes: a camera configured to image a user's surrounding environment; and a controller configured to: determine a target object in the surrounding environment associated with the user's gaze; determine a target depth of the target object relative to the camera based on depth information of the surrounding environment relative to the camera; and control the focus of the camera based on the target depth to image the target object.

[0007] In a third aspect of the present disclosure, a device for imaging is provided. The device includes: an object determination module configured to determine a target object in a user's surrounding environment that is associated with the user's gaze; a depth determination module configured to determine a target depth of the target object relative to a camera imaging the surrounding environment based on depth information of the surrounding environment relative to the camera; and a focus control module configured to control the focus of the camera based on the target depth to image the target object.

[0008] In a fourth aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method of the first aspect.

[0009] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0010] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] 1A and 1B illustrate schematic diagrams of several example environments in which embodiments of the present disclosure can be implemented;

[0013] FIG2A shows a schematic diagram showing the principle of the relationship between clarity and depth of field;

[0014] FIG2B shows a schematic diagram showing the principle relationship between spatial resolution and focal length;

[0015] FIG3 is a schematic diagram illustrating an architecture for performing an auto-focus function according to some embodiments of the present disclosure;

[0016] FIG4 shows a flow chart of a process for imaging according to some embodiments of the present disclosure;

[0017] FIG5A is a schematic diagram showing a preset plane specified by discrete zoom of a camera according to some embodiments of the present disclosure;

[0018] FIG5B is a schematic diagram showing discrete zooming of a camera within a preset plane according to some embodiments of the present disclosure;

[0019] FIG6A is a schematic diagram showing a target plane for continuous zooming of a camera according to some embodiments of the present disclosure;

[0020] FIG6B shows a schematic diagram of a camera continuously zooming within a target plane according to some embodiments of the present disclosure;

[0021] FIG7 shows a flowchart of a process of starting a focus function according to some embodiments of the present disclosure;

[0022] FIG8A shows a simulation diagram of a camera with the autofocus function disabled according to some embodiments of the present disclosure;

[0023] FIG8B shows a simulation diagram of a camera starting an auto-focus function according to some embodiments of the present disclosure;

[0024] FIG9 shows a block diagram of an apparatus for imaging according to some embodiments of the present disclosure; and

[0025] FIG10 shows a block diagram of a device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0027] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0028] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0031] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0032] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0033] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0034] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0035] Sample Environment

[0036] FIG1A illustrates a schematic diagram of multiple example environments 100 in which embodiments of the present disclosure can be implemented. Referring to FIG1A , in this example environment 100, a user 150 wears an electronic device 120 having a see-through camera 140. The electronic device 120 can capture images of the user's 150 eyes 130 - 1 and 130 - 2 (collectively, eyes).

[0037] The controller 110 processes the image captured by the electronic device 120 and outputs the gaze point information. In some embodiments, the controller 110 may be a separate device capable of communicating with the electronic device 120, such as a server or computing node for image or data processing.

[0038] In some embodiments, the controller 110 may be integrated with the electronic device 120 or at least coupled to the electronic device 120. For example, the controller 110 may be a module of the electronic device 120 or implemented by a processing unit in the electronic device 120.

[0039] Referring to Figure 1B , electronic device 120 may include multiple cameras 140, namely the aforementioned cameras 140 with perspective functionality, configured to image the surrounding environment of user 150. In some embodiments, multiple cameras 140 may be arranged on electronic device 120 along a line connecting the two eyes. These cameras 140 are typically positioned on the front of electronic device 120 to accurately capture the perspective of user 150. These cameras 140 are located at the top center or to the side of the device, mimicking the position and direction of the human eye. From this position, cameras 140 can record the scene observed by user 150 and transmit it to a backend processing unit for analysis and application. Multiple cameras 140 can also be positioned on the top or rear of electronic device 120. A top-mounted arrangement can be used to capture information such as user 150's movements, gestures, or eye movements. A rear-mounted arrangement can be used to record background or environmental changes, monitor surrounding conditions, and more. In some embodiments, multiple perspective cameras 140 can be used and mounted in different locations to simultaneously capture images from multiple angles, enabling more accurate and comprehensive visual perception.

[0040] As an example, four cameras 140 are arranged in pairs, two in front of each other, in front of the left and right eyes. This allows for accurate capture of the user's 150 perspective, simulating the position and direction of the human eye. Of course, this arrangement is not required. In other embodiments, pairs of cameras 140 can also be placed in the direction of eye gaze.

[0041] It should be understood that the plurality of cameras 140 are not limited to the arrangement shown in the embodiment of Figure 1B , and any suitable modification or adjustment may be made to the positions and / or number of the plurality of cameras 140 without departing from the principles of the present disclosure.

[0042] In some embodiments, the electronic device 120 may be, for example, a helmet, glasses, or other appropriate head-mounted display device (also referred to as a wearable or portable device) for VR, AR, MR, or XR.

[0043] It should be understood that the electronic device 120 described above with reference to Figures 1A and 1B is merely exemplary and is not intended to limit the scope of the present disclosure. The following describes the principle between clarity and depth of field with reference to Figure 2A, and the principle between spatial resolution and focal length with reference to Figure 2B.

[0044] FIG2A shows a schematic diagram of the principle between clarity and depth of field. FIG2B shows a schematic diagram of the principle between spatial resolution and focal length. Referring to FIG2A , when using a head-mounted display device, the camera can provide a real-time image of the real scene. Therefore, the perspective function of the head-mounted display device is the key to enabling users to seamlessly switch between virtual and real environments. However, achieving optimal clarity in this perspective function faces multiple challenges and trade-offs. Current perspective technology has problems in pursuing the collaboration between high clarity, large depth of field and low noise. First, the relationship between clarity and depth of field (DoF) and signal-to-noise ratio can be analyzed through the following three equations. Equation (1) can calculate the depth of field. Equation (2) can calculate the pixels per degree (PPD) used to characterize spatial resolution. Combining Equations (1) and (2) to obtain Equation (3), the relationship between depth of field and PPD can be obtained as shown below:

[0045] As shown in Figure 2A, the DoF (depth of field) in formula (1) represents the depth of field. u represents the working distance, that is, the distance from the lens to the target object. N represents the F number. It can be seen that DoF and f 2 Inversely proportional (i.e. ). c represents the circle of confusion. f represents the focal length, also known as the effective focal length (EFL). Since the working distance u is significantly greater than the focal length f, u ≈ uf. According to equation (1) above, the depth of field is inversely proportional to the square of the focal length. Therefore, increasing the magnification of a camera lens to improve clarity will reduce the depth of field. This means that while objects at a certain distance will remain sharp, objects at other distances will become blurred.

[0046] As shown in FIG2B , the PPD in equation (2) represents the number of pixels corresponding to one degree in space. Pixel size represents the pixel size. According to equation (2) above, the PPD is inversely proportional to the pixel size. From equations (1) and (2) above, equation (3) is obtained, showing that the depth of field is inversely proportional to the square of the PPD, and also inversely proportional to the pixel size.

[0047] As mentioned above, the perspective function of head-mounted display devices is key to enabling users to seamlessly switch between virtual and real environments. The clarity of the perspective function is crucial to the perspective function of head-mounted display devices, which directly affects the user's perception and interaction with virtual and real elements. The PPD indicator is usually used to quantify the level of perceptible detail. In order to achieve high clarity, a high-resolution image sensor with sufficient detail capture capability is required, such as a CMOS sensor. It should be understood that the greater the focal length of the camera of current head-mounted display devices, the higher the clarity (also known as accuracy), but the depth of field will inevitably be greatly reduced. This means that even a small change in distance from the lens will cause the object to be out of focus.

[0048] For example, traditional electronic devices significantly reduce the depth of field while pursuing high definition. Depth of field refers to the area inside the image that is considered to be within the acceptable sharp display range, and the depth of field is usually divided into shallow depth of field and deep depth of field. Head-mounted display devices find a balance between pursuing high definition while maintaining sufficient depth of field. A larger depth of field is critical for users to be able to perceive objects at close range (usually about 30 cm) and far away (more than 5 meters). It allows users to seamlessly switch between interacting with virtual objects at close range and perceiving real-world elements at a distance.

[0049] Furthermore, see-through cameras must effectively capture the real-time environment with short exposure times. To achieve a good signal-to-noise ratio and minimize image noise, increasing the pixel size can reduce image noise. However, increasing the pixel size increases the focal length of the camera lens, further reducing the depth of field. Therefore, finding a balance between a high signal-to-noise ratio and sufficient depth of field is a pressing issue for see-through cameras in traditional head-mounted displays.

[0050] Furthermore, the larger focal length of cameras in current head-mounted display devices makes them more susceptible to temperature fluctuations, which can lead to focus drift. Temperature fluctuations can cause the camera lens material to expand or contract, resulting in a shift in lens focus and causing focus drift. As the focal length of camera lenses increases to improve clarity and PPD, the depth of focus becomes shallower. This shallower depth of focus makes the camera more susceptible to focus fluctuations caused by temperature changes. Focus fluctuations are often exacerbated by the fact that HMDs are used in a variety of temperature environments.

[0051] To this end, an embodiment of the present disclosure proposes a scheme for imaging. According to various embodiments of the present disclosure, a camera and a controller are installed on a wearable device with a perspective function. In use, a target object associated with the user's gaze in the surrounding environment in which the user is located (for example, the position and / or size of the target object that the pupil is looking at) is determined. Based on the depth information of the surrounding environment relative to the camera, the target depth of the target object relative to the camera is determined. Furthermore, the focus of the camera is controlled based on the target depth to image the target object. The target object in the embodiment of the present disclosure can be any type of object in the surrounding environment. For example, the object that the user is currently looking at, or the object that the user is predicted to look at, or the object that the user is expected to look at (for example, according to the application scenario, the user is expected to look somewhere, or there is a change in the environment, and the object that the user is expected to notice). The target depth can be expressed as the relative distance between the target object and the camera (for example, a lens).

[0052] In the embodiments of the present disclosure, autofocus technology is utilized to overcome the challenges faced by see-through technology in wearable devices, providing users with a superior see-through experience. This technology can be used to maximize its potential in various application areas (such as gaming, education, and healthcare), bringing users unprecedented immersion and comfort. In traditional mixed reality head-mounted display devices, there is a trade-off between clarity and depth of field. However, by introducing autofocus technology in the embodiments of the present disclosure, this problem can be alleviated, achieving a sharp see-through experience and a wider depth of field (for both near and far scenes). This autofocus technology allows the use of larger pixel sizes to improve the signal-to-noise ratio and achieve better image quality. Furthermore, the see-through cameras used in current mixed reality head-mounted display devices all use fixed aperture technology. However, by introducing autofocus functionality, the electronic devices in the embodiments of the present disclosure can enable dynamic aperture adjustment, thereby enhancing photo and video capture capabilities. This allows users to select different depths of field based on the needs of different applications. Furthermore, in brightly lit environments, autofocus technology can better adapt to environmental conditions.

[0053] Some example embodiments are described below with continued reference to FIG. 3 to FIG. 10 .

[0054] Example Architecture

[0055] FIG3 shows a schematic diagram of an architecture 300 for performing an auto-focus function according to some embodiments of the present disclosure. The example architecture 300 may be implemented at the electronic device 120. The example architecture 300 is described below with reference to FIG1A and FIG1B.

[0056] As shown in FIG3 , as described above, in order to solve the problem of coordination between clarity, depth of field and signal-to-noise ratio, as well as the focus drift problem existing in traditional mixed reality head-mounted display devices. In the embodiment of the present disclosure, the electronic device 120 adopts a camera 140 that can realize the XR autofocus perspective function / behavior to achieve the problem of coordination between clarity, depth of field and signal-to-noise ratio mentioned above. In order to realize the XR autofocus perspective function / behavior, the architecture 300 may include application requirements 310, eye tracking information 320, depth information 330, lens autofocus algorithm 340 and lens autofocus hardware 350.

[0057] In some embodiments, based on application requirements 310, electronic device 120 may need to obtain the gaze point of user 150, or the desired location of user 150's gaze, and may obtain eye tracking information 320. Simultaneously, depth information 330 of the gaze point may be obtained. In response to depth information 330, camera 140 implements autofocus via lens autofocus hardware 350. The autofocus process of lens autofocus hardware 350 is performed via lens autofocus algorithm 340.

[0058] In some embodiments, the lens autofocus hardware 350 can utilize various autofocus technologies to enable autofocus, including but not limited to voice coil motors (VCMs), stepper motors, ultrasonic motors (USMs), linear motors, piezoelectric motors, liquid lenses, electromagnetic coils, shape memory alloys (SMAs), and hybrid technologies (e.g., poLight lenses that combine piezoelectric actuators, deformable membranes, and optical-grade soft materials).

[0059] In some embodiments, the lens autofocus algorithm 340 also plays a key role. Algorithms in the disclosed embodiments include, but are not limited to, contrast detection autofocus (CDAF), phase detection autofocus (PDAF), laser (time-of-flight sensor) autofocus, and depth map-based autofocus.

[0060] In some embodiments, the depth information 330 may be depth information or 3D mapping data of the environment acquired in real time by a 6DoF (6 Degrees Of Freedom) camera or a depth camera. To ensure the robustness of the camera 140, a hybrid autofocus method combining multiple algorithms may be required in some cases.

[0061] In addition, during the implementation process, the depth information 330 and the eye tracking information 320 can also be used to instruct the camera 140 to focus according to the gaze point of the user 150, or to predict the object that the user 150 may look at in the next few tens of milliseconds, or to guide the user 150 to look at an object for focus.

[0062] Furthermore, in some embodiments, the data for obtaining the eye tracking information 320 is obtained using an eye tracking sensor or other sensors (e.g., speech recognition, brain-computer interface sensors, neural detection sensors). These sensor data can be used to predict the gaze position, allowing the camera 140 to automatically focus on the predicted object in advance to achieve lower latency.

[0063] Furthermore, to achieve low-latency performance for camera 140, multiple aspects of the autofocus process require comprehensive optimization. This includes, but is not limited to, optimization of autofocus algorithm 340, improvements to autofocus hardware 350 technology, optimization of camera 140 image processing, display rendering, and eye-tracking prediction and correction. By combining different autofocus technologies and methods and leveraging depth information and eye-tracking signals, an autofocus perspective function can be implemented in electronic device 120, ensuring low-latency performance for camera 140 through the aforementioned multi-faceted optimizations.

[0064] Example Process

[0065] 4 shows a flow chart of a process 400 for imaging according to some embodiments of the present disclosure. The process 400 may be implemented at the electronic device 120. The process 400 is described below with reference to FIG1A and FIG1B.

[0066] At block 410, the controller 110 determines a target object in the surrounding environment of the user 150 that is associated with the gaze of the user 150. The target object may be of different types. It is understood that the target object may be any type of object in the surrounding environment.

[0067] In some embodiments, the target object may be the object currently being gazed at by the user 150. The controller 110 determines the object currently being gazed at by the user 150 as the target object based on eye tracking of the user 150. For example, the controller 110 determines the target object based on data from the eye tracking information 320.

[0068] Alternatively or additionally, in some embodiments, the target object may be an object that the controller 110 predicts the user 150 will look at. The controller 110 may predict the object that the user 150 is currently looking at as the target object based on sensory information associated with the user.

[0069] Alternatively or additionally, in some embodiments, the target object may be an object that the controller 110 directs the user 150 to look at. The controller 110 may determine an object from the surrounding environment to direct the user 150's attention as the target object. For example, depending on the application scenario, the target object may be an object that the user 150 is expected to look at, or an object that the user 150 is expected to notice due to a change in the environment.

[0070] Continuing with the process shown in FIG. 4 , at block 420 , controller 110 determines a target depth of the target object relative to camera 140 based on depth information of the surrounding environment relative to camera 140 imaging the surrounding environment. It will be appreciated that the target depth may be the relative distance of the target object from camera 140 (e.g., a lens). For example, controller 110 determines the target depth of the target object relative to camera 140 based on data from depth information 330 .

[0071] At block 430 , the controller 110 controls the focus of the camera 140 based on the target depth to image the target object. The electronic device 120 using the imaging method can achieve a greater depth of field while meeting high definition by utilizing autofocus technology.

[0072] In some embodiments, the controller 110 may determine a target plane in the surrounding environment that matches the target depth, and then the controller 110 may control the camera 140 to focus on the target plane to image the target object.

[0073] Figure 5A illustrates a schematic diagram of a preset plane designated for discrete zooming of a camera according to some embodiments of the present disclosure. Figure 5B illustrates a schematic diagram of discrete zooming of a camera within a preset plane according to some embodiments of the present disclosure. As shown in Figures 5A and 5B , a target plane can be determined in a discrete manner. Specifically, there may be multiple preset depth ranges, each corresponding to a plurality of preset planes. Controller 110 can determine the preset depth range within which the target depth lies from these preset depth ranges. Controller 110 can then determine the preset plane corresponding to the determined preset depth range as the target plane. A target plane matching the target depth in the surrounding environment is determined in the direction of the target object. For example, an example is described with reference to Figure 5A , which illustrates preset planes P1, P2, P3, P4, and P5. In the present embodiment, each preset plane has a corresponding depth range. For example, depth range 1 for preset plane P1 is 1m to 2m, depth range 2 for preset plane P2 is 2m to 3m, depth range 3 for preset plane P3 is 3m to 4m, and so on for preset planes P4 and P5. This is not specifically limited in the present embodiment. As an example, if the target depth is within the range of 2m to 3m, the preset plane P2 may be determined as the target plane. The number and depth range of the preset planes listed here are merely exemplary.

[0074] Figure 6A shows a schematic diagram of a target plane for continuous zooming of a camera according to some embodiments of the present disclosure. Figure 6B shows a schematic diagram of continuous zooming of a camera within a target plane according to some embodiments of the present disclosure. As shown in Figures 6A and 6B, the target plane can be determined in a continuous focusing manner. Then, the controller 100 determines a target plane in the surrounding environment that matches the target depth in the direction of the target object. For example, an example is described with reference to Figure 6B, showing some planes in the continuous depth plane, specifically planes P1, P2, P3, P4... and P8. In an embodiment of the present disclosure, these planes P1, P2, P3, P4... and P8 have corresponding continuous depth ranges. Therefore, a plane with a target depth can be selected as the target plane without approximation as in a discrete scheme.

[0075] In some embodiments, controller 110 controls camera 140 to focus on a target plane for imaging. Specifically, controller 110 determines configuration information for the lens of camera 140 based on the depth of the target plane relative to camera 140. The lens is adjusted based on the configuration information of the lens of camera 140. It should be understood that determining the configuration information of the lens of camera 140 based on the depth of the target plane relative to camera 140 described above is merely an example of determining configuration information, and the scope of this disclosure is not limited thereto.

[0076] In some embodiments, controller 110 may obtain the ambient temperature of the lens of camera 140. Based on the ambient temperature and the lens's pre-calibrated focus characteristics that vary with temperature, controller 110 may determine temperature compensation information for adjusting the lens. Controller 110 may control the focus of camera 140 based on the target depth. Furthermore, controller 110 controls the focus of camera 140 based on the temperature compensation information and the target depth to image the target object. For example, controller 110 may sense the ambient temperature of the lens of camera 140 based on data from a temperature sensor.

[0077] The present disclosure utilizes an autofocus function that plays a vital role in compensating for temperature drift. By continuously monitoring the focus and utilizing a feedback mechanism, the controller 110 enables the autofocus function of the camera 140 to adjust the position of the lens to maintain accurate focus despite temperature fluctuations. In the embodiments of the present disclosure, the autofocus function of the camera 140 can be used to offset the effects of temperature changes on the lens of the camera 140, ensuring that the electronic device 120 and other imaging systems operating in different temperature environments can focus stably and clearly. For example, the autofocus function of the camera 140 can be implemented using autofocus materials. Specifically, the lens of the camera 140 can use a poLight lens with an opposite thermal coefficient (such as refractive index and temperature) to passively compensate for the effects of temperature changes. The lens of the camera 140 can also actively compensate for temperature drift using a temperature sensor connected to the camera module. The autofocus function of the lens of the camera 140 can also be implemented through factory calibration or real-time image processing (such as contrast detection autofocus or phase detection autofocus), which is not specifically limited in the present disclosure.

[0078] In some embodiments, the controller 110 controls the focus of the camera 140 based on the target depth. Specifically, the controller 110 determines whether the camera 140 is enabled to provide the user 150 with an image of the surrounding environment.

[0079] Alternatively or in a similar manner, in response to determining that the camera 140 is enabled, the controller 110 provides an image of the surrounding environment to the user 150 and determines whether the autofocus function of the camera 140 is enabled. Further, in response to the autofocus function being enabled, the controller 110 controls the focus of the camera 140 based on the target depth determined above. This will be described in detail below.

[0080] Alternatively or in a related embodiment, in response to determining that camera 140 is not enabled, controller 110 provides an image of the surrounding environment to the user. Further, controller 110 controls the plane on which camera 140 focuses to vary within the depth range of the surrounding environment to image or monitor the surrounding environment. This will be described in detail below.

[0081] FIG7 illustrates a flow chart of a process 700 for enabling a focus function according to some embodiments of the present disclosure. As shown in FIG7 , at blocks 710 and 720, when user 150 is using electronic device 120, electronic device 120 may disable or enable the see-through function of camera 140 based on different application requirements (which may be a default HMD operating environment). Therefore, at block 730, it is first necessary to determine whether to provide an image of the surrounding environment to user 150, thereby determining whether camera 140 is enabled.

[0082] At block 740 , when the see-through function of camera 140 is disabled, this means that user 150 cannot see the real world in real time (e.g., in 100% immersive virtual reality). Although camera 140 does not capture the real scene, the autofocus function of camera 140 is still active for environmental monitoring and video recording. In this case, the autofocus algorithm is different from that of MR. Camera 140 can scan the entire depth of the environment to obtain clear images / video of the entire scene.

[0083] In block 750 , when the perspective function of the camera 140 is activated, that is, the electronic device 120 enables the MR function (requires the camera 140 to turn on the perspective function).

[0084] At block 760, the camera 140 may disable or enable the autofocus function, depending on the specific application requirements of the electronic device 120 (e.g., only involving near-field objects or only involving far-field objects) and the maturity and stability of the autofocus. In other words, after the response determines that the camera 140 is enabled, in order to provide the user 150 with an image of the surrounding environment, it is necessary to determine whether the autofocus function of the camera 140 will be enabled.

[0085] At block 770, if the camera 140 has its autofocus function disabled, the camera 140 is fixed in focus. In other words, upon determining that the camera 140 is not enabled to provide the user 150 with an image of the surrounding environment, the plane on which the camera 140 is focused is controlled to vary within the depth range of the surrounding environment to image or monitor the surrounding environment.

[0086] At block 780 , in an MR application scenario, when autofocus is enabled, it is implemented through hardware and software as well as integration with other sensor signals (including eye tracking, 6DoF camera, depth sensor, IMU, etc.). Thus, in response to the autofocus function being enabled, the focus of the camera 140 is controlled based on the target depth, thereby controlling the focus of the camera 140 based on the target depth.

[0087] At block 790, there are three ways to implement autofocus for camera 140. The first is to focus on the object that user 150 is looking at, which is an autofocus experience similar to "foveated rendering." No matter where user 150 is looking, camera 140 will focus on the corresponding object with low latency (the user 150 cannot see the focus adjustment). The second is to guide user 150 to look at another object, that is, to change the focus position from the location where user 150 is looking, so as to guide the user 150's eye focus area according to application needs. The third is to predict the object that user 150 will look at. This has been described in detail above and will not be repeated here.

[0088] Figure 8A shows a simulation diagram 801 of a camera without the autofocus function activated according to some embodiments of the present disclosure. As shown in Figure 8A, when the autofocus function is not enabled, the camera is in a fixed focus state. Curves 811, 812, and 813 show the change of PDD with target distance. As shown in Figure 8A, as the target distance increases, the clarity gradually decreases. Referring to Figure 8A, it can be seen that when the target distance approaches 1000mm, the pixel size (PPD) drops rapidly, that is, the clarity drops rapidly. The range in which clear imaging can be achieved is very narrow.

[0089] FIG8B shows a simulation diagram 802 of a camera activating the autofocus function according to some embodiments of the present disclosure. As shown in FIG8B , when the autofocus function is enabled, the camera is in an autofocus state. Curves 821, 822, and 823 show the change in PDD with target distance. As shown in FIG8B , as the target distance increases, the clarity can remain consistent. Referring to FIG8B , for example, when the target distance is 1000 mm, 2000 mm, 3000 mm, and 4000 mm, the pixel size (PPD) is the same, that is, the clarity is consistent. Compared with FIG8A , the range in which clear imaging can be achieved is significantly increased.

[0090] As described above, in addition to addressing clarity and depth of field issues, the autofocus functionality of camera 140 in electronic device 120 in the disclosed embodiments offers other advantages. For example, it can provide active temperature drift compensation to improve the perspective experience and ensure image stability. Furthermore, this technology enables more efficient display image processing, resulting in a smoother and more realistic virtual experience for user 150.

[0091] Example devices and equipment

[0092] 9 shows a schematic structural block diagram of an imaging apparatus 900 according to certain embodiments of the present disclosure. The apparatus 900 may be implemented as or included in the electronic device 120. Each module / component in the apparatus 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0093] As shown, apparatus 900 includes an object determination module 910 configured to determine a target object in a user's surrounding environment that is associated with the user's gaze. Apparatus 900 also includes a depth determination module 920 configured to determine a target depth of the target object relative to a camera imaging the surrounding environment based on depth information of the surrounding environment relative to the camera. Apparatus 900 also includes a focus control module 930 configured to control the focus of the camera based on the target depth to image the target object.

[0094] In some embodiments, the focus control module 930 is configured to determine a target plane in the surrounding environment that matches the target depth, and control the camera to focus on the target plane to image the target object.

[0095] In some embodiments, the focus control module 930 is configured to determine a plane in the surrounding environment having a target depth relative to the camera as the target plane.

[0096] In some embodiments, the focus control module 930 is configured to determine a preset depth range in which the target depth is located from a plurality of preset depth ranges. The plurality of preset depth ranges respectively correspond to a plurality of preset planes. The preset plane corresponding to the determined preset depth range is determined as the target plane.

[0097] In some embodiments, the focus control module 930 is configured to determine configuration information for a lens of the camera based on a depth of the target plane relative to the camera; and adjust the lens based on the configuration information.

[0098] In some embodiments, the object determination module 910 is configured to perform at least one of the following: determining the object that the user is currently looking at as the target object based on tracking the user's eyes, predicting the object that the user will look at as the target object based on sensory information associated with the user, or determining an object from the surrounding environment to guide the user's gaze as the target object.

[0099] In some embodiments, the apparatus 900 further includes an ambient temperature acquisition module configured to acquire the ambient temperature of the environment in which the camera lens is located. Based on the ambient temperature and a pre-calibrated focus characteristic of the lens that varies with temperature, temperature compensation information for adjusting the lens is determined, and controlling the focus of the camera based on the target depth to image the target object includes controlling the focus of the camera based on the temperature compensation information and the target depth.

[0100] In some embodiments, the focus control module 930 is configured to determine whether the camera is enabled to provide an image of the surrounding environment to the user; in response to determining that the camera is enabled to provide an image of the surrounding environment to the user, determine whether the autofocus function of the camera is enabled; and in response to the autofocus function being enabled, control the focus of the camera based on the target depth.

[0101] In some embodiments, the device 900 further includes a change control module configured to control the plane on which the camera is focused to change within a depth range of the surrounding environment in response to determining that the camera is not enabled to provide an image of the surrounding environment to the user, so as to image or monitor the surrounding environment.

[0102] FIG10 is a block diagram illustrating an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 1000 shown in FIG10 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 1000 shown in FIG10 may be used to implement the electronic device 120 of FIG1A .

[0103] As shown in FIG10 , electronic device 1000 is in the form of a general electronic device. Components of electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. Processing unit 1010 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 1020. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 1000.

[0104] The electronic device 1000 typically includes a plurality of computer storage media. Such media can be any accessible media that is accessible to the electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1020 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1030 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 1000.

[0105] The electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 10 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 1020 may include a computer program product 1025 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0106] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 1000 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0107] Input device 1050 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 1060 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 1000 may also communicate with one or more external devices (not shown) via communication unit 1040 as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with electronic device 1000, or with any device that allows electronic device 1000 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0108] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0109] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that, when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams.

[0111] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0112] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0113] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Images