Automatic exposure control method and device, equipment, storage medium and system

Abstract

The invention discloses an automatic exposure control method and device, equipment, a storage medium and a system, and the method comprises the steps: obtaining a first image collected by a camera under a first exposure parameter, and determining the gradient of one or more image blocks in the first image; determining an automatic exposure metric according to the gradient of the image block and a target function for calculating the automatic exposure metric; optimizing the automatic exposure measurement based on an optimization algorithm, and determining a target exposure parameter of the image block corresponding to the optimal automatic exposure measurement; determining a target exposure parameter of the camera according to the target exposure parameter of the one or more image blocks; and controlling outflow of the camera according to the target exposure parameter. Therefore, the AE measurement based on the image gradient is adopted to perform automatic exposure control, so that the camera can acquire the image with enough abundant texture details, the information amount provided by the visual front-end system is maximized, and the perception capability and the performance of the visual rear-end system are improved.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is filed based on and claims the priority of U.S. patent application No. 63 / 509,314, filed on June 21, 2023. The entire contents of the patent application are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of imaging technology, and in particular to an automatic exposure control method and apparatus, device, storage medium, and system. Background Art

[0004] Visual information (e.g., image frames) from a camera is essential for a Simultaneous Localization and Mapping (SLAM) system to understand the surrounding environment and estimate its position and posture relative to the environment. However, the dynamic range of a camera is limited, and the camera usually loses information due to oversaturation caused by strong light, or weak signals and noise caused by low light. In order for a SLAM system to work under a wide range of lighting conditions, automatic exposure control is essential for estimating the optimal exposure time, analog gain, and digital gain, and adjusting the SLAM system to the optimal exposure time, analog gain, and digital gain. Figure 1 A schematic diagram of the framework of a typical SLAM system is shown, which highlights the roles of auto exposure (AE) and exposure compensation (EC).

[0005] Traditional AE control is designed for photography or video recording, so the AE metrics and AE control mechanisms used are intended to be visually appealing to the human eye. For example, AE control optimized for photography can be set to target faces with visually pleasing brightness. However, for computer vision systems such as SLAM, visual or aesthetic appeal is not directly related to the design purpose of these systems. The desired goal of AE control should be the amount of information provided by the camera system to downstream computer vision programs. Therefore, the cameras used in SLAM systems need to optimize the AE control mechanism to extract more information and achieve accurate and high-performance SLAM systems. Summary of the invention

[0006] The embodiments of the present application are intended to provide an automatic exposure control method and apparatus, device, storage medium, and system.

[0007] The technical solution of this application is implemented as follows:

[0008] In a first aspect, an automatic exposure control method is provided, comprising:

[0009] Acquire a first image captured by a camera under a first exposure parameter;

[0010] determining a gradient of one or more image blocks in the first image according to a pixel value of a reference pixel in the first image;

[0011] Determining an automatic exposure metric according to the gradient of the image block and an objective function for calculating the automatic exposure metric, wherein the automatic exposure metric is used to measure the quality of the exposure state of the image block;

[0012] Optimizing the automatic exposure metric based on an optimization algorithm, and determining a target exposure parameter of the image block corresponding to the optimal automatic exposure metric;

[0013] determining a target exposure parameter of the camera according to the target exposure parameters of the one or more image blocks;

[0014] The camera outflow is controlled according to a target exposure parameter of the camera.

[0015] In a second aspect, an automatic exposure control device is provided, comprising:

[0016] An acquisition unit is configured to acquire a first image captured by a camera under a first exposure parameter;

[0017] a processing unit configured to determine a gradient of one or more image blocks in the first image based on a pixel value of a reference pixel in the first image;

[0018] The processing unit is further configured to determine an automatic exposure metric according to the gradient of the image block and an objective function for calculating the automatic exposure metric, wherein the automatic exposure metric is used to measure the quality of the exposure state of the image block; optimize the automatic exposure metric based on an optimization algorithm, and determine a target exposure parameter of the image block corresponding to the optimal automatic exposure metric;

[0019] The processing unit is further configured to determine a target exposure parameter of the camera according to the target exposure parameters of the one or more image blocks;

[0020] A control unit is configured to control the camera outflow according to a target exposure parameter of the camera.

[0021] In a third aspect, an automatic exposure control device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0022] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0023] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0024] In a fifth aspect, a simultaneous positioning and map building system includes: a camera, an automatic exposure control device and a simultaneous positioning and map building device; wherein,

[0025] The camera is configured to capture images and output images to the automatic exposure control device;

[0026] The automatic exposure control device is configured to perform the steps of the aforementioned method to output an image to the simultaneous positioning and mapping device; and

[0027] The simultaneous positioning and mapping device is configured to construct an environment map based on the received images and estimate its own position and posture relative to the environment.

[0028] The present application provides an automatic exposure control method and apparatus, device, storage medium, and system, the method comprising: obtaining a first image captured by a camera under a first exposure parameter, determining the gradient of one or more image blocks in the first image; determining the automatic exposure metric according to the gradient of the image block and the objective function of calculating the automatic exposure metric; optimizing the automatic exposure metric based on an optimization algorithm, determining the target exposure parameter of the image block corresponding to the optimal automatic exposure metric; determining the target exposure parameter of the camera according to the target exposure parameter of one or more image blocks; and controlling the camera outflow according to the target exposure parameter. In this way, the automatic exposure control is performed using the AE metric based on the image gradient, so that the camera can capture images with sufficiently rich texture details, maximize the amount of information provided by the visual front-end system, and thus improve the perception ability and performance of the visual back-end system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the framework of a typical SLAM system is shown;

[0030] Figure 2 This is a schematic diagram of a first process flow of the automatic exposure control method in an embodiment of the present application;

[0031] Figure 3 Schematic diagram of the spatial distribution decision method and the time averaging method in the embodiment of the present application;

[0032] Figure 4 This is a second flow chart of the automatic exposure control method in an embodiment of the present application;

[0033] Figure 5 This is a schematic diagram of the structure of the automatic exposure control device in the embodiment of the present application;

[0034] Figure 6 This is a schematic diagram of the structure of the automatic exposure control device in the embodiment of the present application;

[0035] Figure 7 Schematic diagram of the structure of the simultaneous positioning and map building system in the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0037] Figure 2 FIG. 1 is a schematic diagram of a first process flow of the automatic exposure control method in an embodiment of the present application. Figure 2 As shown, the method may specifically include:

[0038] Step 201: Acquire a first image captured by a camera under a first exposure parameter;

[0039] It should be noted that an automatic exposure control method provided in an embodiment of the present application can specifically be an automatic exposure control method for a simultaneous positioning and mapping system, which uses an image gradient-based AE metric for automatic exposure control, so that the camera can capture images with sufficiently rich texture details, providing more visual information for the visual back-end system of the simultaneous positioning and mapping system, thereby improving the perception ability and performance of the visual back-end system.

[0040] The first exposure parameter may be understood as an exposure parameter used when controlling the camera to capture the first image. The first exposure parameter may include one or more exposure parameters of the camera.

[0041] In some embodiments, the first exposure parameter includes at least exposure time and / or gain (including analog gain and digital gain), wherein exposure time refers to the length of time the camera sensor is exposed to incident light, and gain is also called ISO or sensitivity, which is an indicator used to measure the sensitivity of the camera to light. Both exposure time and gain control the brightness of the image frame. One difference between them is that gain usually increases the noise level but does not increase motion blur.

[0042] The goal of the automatic exposure control method in the embodiment of the present application is to estimate the optimal settings of exposure time and / or gain, and adjust the camera to the optimal settings of exposure time and / or gain. Here, the optimal settings can be designed to change according to the task to be performed by the camera.

[0043] In some embodiments, the method further includes: determining whether to trigger AE adjustment based on the AE statistical value of the first image; if AE adjustment is triggered, executing steps 202 to 206 of the automatic exposure control method based on AE measurement of image gradient; if AE adjustment is not triggered, continuing to use the first exposure parameter to control the camera outflow.

[0044] Exemplarily, the AE statistic value may be the average brightness of the first image. When the average brightness is less than a first brightness threshold, AE adjustment is triggered to increase the exposure time; when the average brightness is greater than a second brightness threshold, AE adjustment is triggered to reduce the exposure time.

[0045] Step 202: determining the gradient of one or more image blocks in the first image according to the pixel value of the reference pixel in the first image;

[0046] Specifically, the first image is divided into one or more image blocks, and the gradient of each image block is determined using the pixel value of the reference pixel in each image block, wherein the reference pixel may include all pixels in the image block or certain specific pixels in the image block.

[0047] In some embodiments, if the first image includes one image block, that is, the first image is not divided into image blocks, the gradient of the image block is the gradient of the first image, and the corresponding target exposure parameter is calculated according to the gradient of the first image. If the first image is divided into two or more image blocks, each image block independently calculates the corresponding target exposure parameter, and the calculation method is consistent, and the target exposure parameter for controlling the camera outflow is further determined according to the target exposure parameters of all image blocks.

[0048] Step 203: determining an automatic exposure metric according to the gradient of the image block and the objective function of calculating the automatic exposure metric, where the automatic exposure metric is used to measure the quality of the exposure state of the image block;

[0049] The objective function provides an AE measurement method based on image gradients. The AE measurement quantifies the quality of the current image exposure state. The higher the AE measurement value, the higher the quality of the exposure state and the more information the image contains. In the embodiment of the present application, the objective function is a mathematical function that aims to maximize the AE measurement. Specifically, it can be any mathematically definable and computable function. The objective function can be expressed as M AE =f(x), where x represents the image gradient.

[0050] Most traditional AE control methods, usually designed for digital photography or video recording, use metrics based on image brightness or other brightness-based image statistics. The design principle is based on the assumption that the brightness of the region of interest (ROI) image should be moderate, neither saturated (greater than the maximum value allowed by the dynamic range) nor too low (overwhelmed by noise). Improperly exposed images will have unpleasant visual appeal and may be unacceptable to users. Although this is usually a valid assumption for photography or video recording, it is not the best solution for cameras in visual backend systems (such as SLAM) that require information (rather than visual appeal). Therefore, an AE metric optimized for the visual backend system is needed to maximize the amount of information in the visual front end. More specifically, the ideal AE metric for the SLAM system should meet the following conditions: for non-direct SLAM methods, the AE metric helps the SLAM system to collect as many tracking feature points as possible; while for direct SLAM methods, the AE metric helps the SLAM system to increase the signal-to-noise ratio of the collected images as much as possible.

[0051] In some embodiments, determining the automatic exposure metric based on the gradient of the image block and the objective function for calculating the automatic exposure metric includes: using the horizontal component and the vertical component of the gradient of the image block as variables of the objective function for calculating the automatic exposure metric to determine the automatic exposure metric.

[0052] In some embodiments, the automatic exposure metric is determined based on the gradient of the image block and the objective function for calculating the automatic exposure metric, including: determining the sum of square gradients of the image block based on the horizontal component and vertical component of the gradient of the image block; and determining the automatic exposure metric using the sum of square gradients of the image block as a variable of the objective function for calculating the automatic exposure metric.

[0053] Exemplarily, using the squared gradient of the image block as a variable of an objective function for calculating the automatic exposure metric, and determining the automatic exposure metric, includes: using the squared gradient of the image block as a variable of one or more transformation functions in the objective function, and determining one or more gradient transformation parameters of the image block; determining the automatic exposure metric based on the one or more gradient transformation parameters;

[0054] The one or more gradient transformation parameters include at least one of the following: a weighted average of the square of the gradient and local statistics, a weighted average of the square of the gradient and global statistics, a regularization term, and a frequency domain feature of a frequency domain range of interest. In some embodiments, the transformation parameters also include other transformation parameters of the gradient.

[0055] In some embodiments, an automatic exposure metric is determined based on one or more gradient transformation parameters, one transformation parameter is used as the automatic exposure metric, or multiple gradient transformation parameters are fused to obtain the automatic exposure metric.

[0056] Different transformation functions aim to produce AE metrics with different emphases.

[0057] If the system noise is relatively large, the signal can be enhanced by adding an additional image regularization operation to the objective function of the AE metric. By introducing the regularization term, the complexity of the objective function can be controlled, overfitting can be avoided, and generalization ability can be improved. The regularization term can be an L1 regularization term or an L2 regularization term.

[0058] If the signal of interest of the system is a narrow frequency band, additional frequency domain filtering operations can be added to the objective function of the AE metric to enhance the effect. These operations usually involve Fourier transform or similar transformation methods to convert the image from the spatial domain to the frequency domain, and then process specific frequency components in the frequency domain, including filtering, enhancement, spectrum analysis, etc.

[0059] The gradient of an image describes the spatial variation of pixel intensity and can be expressed as Where I(x,y) is the pixel value at the (x,y) position, is the horizontal component of the gradient, is the vertical component of the gradient. The gradient of an image measures the level of detail of the image texture. Compared with brightness, it can better estimate the amount of information about trackable features in the image, which is beneficial for information extraction by the SLAM system. The image gradient characterizes the magnitude of image change. Specifically, the rate of change in the X and Y directions of the image can be calculated. The change in the X direction refers to the pixel value to the right of the current pixel (X plus 1) minus the pixel value to the left of the current pixel (X minus 1). The change in the Y direction is the pixel value below the current pixel (Y plus 1) minus the pixel value above the current pixel (Y minus 1).

[0060] The gradient-based AE metric can be expressed as in is the sum of the squares of the gradients, and f(·) represents some transformation of the original gradient magnitude at each pixel. Some examples of such transformations might include weighted averages based on global or local statistics of the image, or some image regularization operations that enhance the image, or some frequency domain operations that are of interest to a particular spectral range.

[0061] Step 204: optimizing the automatic exposure metric based on the optimization algorithm, and determining a target exposure parameter of the image block corresponding to the optimal automatic exposure metric;

[0062] The optimization algorithm is used to find a set of parameter values ​​that achieve the optimal solution of the objective function. In the embodiment of the present application, the optimal solution of the objective function is the maximum value of the automatic exposure metric. Exemplarily, the optimization algorithm can be used to solve the first-order derivatives of the exposure parameters during the optimization process, and the values ​​of these first-order derivatives are the fine-tuning parameters of the parameters in the optimization function.

[0063] Based on the given objective function for calculating the AE metric, the AE control module is designed to adjust the exposure parameters according to the exposure parameters of the higher AE metric until a stable state is reached. In order to move the exposure parameter settings toward the optimal value, the embodiment of the present application uses an online optimization procedure to estimate the direction and step size of the change of the exposure parameter. In some embodiments, the derivative of the estimated AE metric is first calculated during the online optimization. As an optimized function of exposure time and / or gain, one can then use Estimate the desired change in exposure time and / or gain, where M AE is the AE metric and x is the variable to be optimized (exposure time and / or gain).

[0064] In some embodiments, the online optimization procedure simulates images with slightly different exposure parameter settings and calculates their corresponding AE metrics, and then selects the exposure parameter setting with the best AE metric as the updated target exposure parameter.

[0065] In some embodiments, based on the optimization algorithm, the exposure parameter variables of the image block corresponding to the optimal automatic exposure metric of the objective function are determined; the target exposure parameter is determined according to the first exposure parameter and the exposure parameter variable, and the exposure parameter variable includes an adjustment step size and an adjustment direction.

[0066] Step 205: determining a target exposure parameter of the camera according to the target exposure parameters of one or more image blocks;

[0067] In some embodiments, if the first image includes an image block, that is, the first image is not divided into image blocks, the target exposure parameter of the image block is used as the second exposure parameter; the second exposure parameter is smoothed using a time filter to determine the target exposure parameter of the camera.

[0068] In some embodiments, if the first image is divided into more than two image blocks, the method may specifically include: deciding on the target exposure parameters of multiple image blocks based on a consensus mechanism between the image blocks to determine the second exposure parameter; using a time filter to smooth the second exposure parameter to determine the target exposure parameter of the camera.

[0069] In order to avoid the AE metric of the image being dominated by parts of the image with rich textures (usually associated with large gradients), the input image is divided into M×N image blocks using a negotiation mechanism between image blocks, where M is an integer greater than 1 and / or N is an integer greater than 1 (e.g., M=4, N=3). Each image block independently determines its optimal exposure parameter setting, and the global exposure parameter setting will be the result of consensus among all image blocks.

[0070] In some embodiments, the consensus mechanism may include: calculating an average value of target exposure parameters of multiple image blocks to obtain a second exposure parameter.

[0071] The average value may be a weighted average value or an arithmetic average value. In some embodiments, for the weighted average value, weight values ​​of different image blocks may be set for the region of interest of the image.

[0072] In some embodiments, the consensus mechanism may include customized logic to implement specific functions. Exemplarily, the consensus mechanism may also include: determining the exposure state of each image block based on the pixel value of the reference pixel in each image block, the exposure state including one of the following: accurate exposure state, underexposure state and overexposure state; determining the number of image blocks in a first exposure state based on the exposure state of each image block; when the number of image blocks is greater than a first preset value, adjusting the target exposure parameters of the image blocks in the first exposure state to reduce the number of image blocks in the first exposure state.

[0073] In some embodiments, the first exposure state may be an overexposed state. In some embodiments, the first exposure state may be an underexposed state, or an overexposed state and an underexposed state. Overexposed and / or underexposed areas have no or provide little scene information, and these two areas are generally unavoidable due to the limited perceptible brightness or dynamic range of the camera. For example, lights on the ceiling are generally overexposed or saturated. The embodiments of the present application customize special processing strategies for overexposed and / or underexposed areas to reduce or eliminate these areas, thereby enabling the camera to capture images with sufficiently rich texture details and increase the amount of visual information.

[0074] The method uses a consensus mechanism between image blocks to limit overexposed and / or underexposed areas. First, the second preset value of the overexposed and underexposed areas can be defined by the user or preconfigured. For example, if the number of overexposed pixels in the image block is greater than or equal to the second preset value, the image block is defined as an overexposed state. The definition of underexposed image blocks is similar. Then, in the consensus stage, the first preset value of the overexposed and underexposed areas can be defined by the user or preconfigured. Logic about the number of image blocks in the overexposed state or underexposed state is implemented. For example, if 3 image blocks are overexposed, the exposure time is reduced to reduce the number of overexposed image blocks, and there is no need to pay attention to the exposure of the remaining (MxN-3) image blocks. In this way, the overall area of ​​the overexposed and underexposed areas can be effectively limited to increase the amount of visual information of the shooting environment.

[0075] In some embodiments, the consensus mechanism may also include: determining the exposure state of each image block based on the pixel value of the reference pixel in each image block, the exposure state including one of the following: accurate exposure state, underexposure state and overexposure state; determining the number of image blocks in a first exposure state based on the exposure state of each image block; when the number of image blocks is greater than a first preset value, adjusting the second exposure parameter to reduce the number of image blocks in the first exposure state.

[0076] That is to say, the exposure parameter can be adjusted by adjusting the target exposure parameter of the local image block and then calculating the average value of all image blocks, or the average value of all image blocks can be calculated first and then adjusted to reduce the number of image blocks in the first exposure state.

[0077] In order to avoid sudden changes in exposure control, the second exposure parameter can be optionally temporally filtered. In some embodiments, the temporal filtering can be implemented by a median filter, obtaining a first reference exposure parameter of a multi-frame reference image, wherein the multi-frame reference image includes a continuous multi-frame image of the first image; using a median filter, determining the median of the exposure parameter among the second exposure parameter and the first reference exposure parameter of the multi-frame reference image; and using the median of the exposure parameter as the target exposure parameter of the camera. The AE control decisions of the continuous multi-frames are stored in a buffer, and the median of the buffered decisions is used as the final decision of the exposure control.

[0078] Figure 3 Schematic diagram of the spatial distribution decision method and the time average method in the embodiment of the present application, such as Figure 3 As shown in the figure, based on the spatial distribution AE decision method, the input image is divided into M×N image blocks. Each image block independently determines its optimal exposure setting, the "+" sign indicates an increase in the exposure parameter, the "-" sign indicates a decrease in the exposure parameter, and the "0" sign indicates that the current exposure parameter remains unchanged. According to the optimal exposure settings of all image blocks, it is determined whether to adjust the exposure setting of the camera. Furthermore, in order to avoid sudden changes in exposure control, temporal filtering can be optionally implemented to determine the final exposure setting of the camera.

[0079] Step 206: Control the camera outflow according to the target exposure parameter of the camera.

[0080] In some embodiments, the method further includes: acquiring a second image captured by the camera under target exposure parameters; performing exposure compensation on the second image according to the target exposure parameters of the camera to obtain a second image after exposure compensation.

[0081] The AE control method adjusts the settings of exposure time and gain to allow the system to perceive the optimal amount of visual information from the camera. If the brightness consistency assumption is used, adjustments are considered in the SLAM front end. For example, for a SLAM front end based on feature tracking, the implicit assumption is that the content (brightness and color) of the scene is constant within the frame. Changes in AE settings break this assumption. Although the SLAM front end can tolerate small changes in AE settings, the exposure compensation method allows for higher accuracy.

[0082] In some embodiments, the exposure compensation includes: determining one or more exposure compensation parameters of the second image according to the target exposure parameter and the second reference exposure parameter of the camera, the exposure compensation parameters including a first ratio of the reference exposure time to the target exposure time, and / or a second ratio of the reference gain to the target gain; performing exposure compensation on the second image according to the one or more exposure compensation parameters to obtain a compensated second image. The exposure compensation parameters may be exposure parameters used by the reference image for realizing the brightness consistency assumption for the second image.

[0083] That is, exposure compensation involves directly correcting the brightness of the current image based on the current exposure time and gain. To compensate the current image I t (x,y), where T ref and T t is the reference exposure time and the current exposure time, g ref and g t is the reference gain and the current gain. If the compensated image has a constant exposure setting, it can be used as the reference exposure setting.

[0084] In some embodiments, exposure compensation includes injecting metadata including current exposure settings into the SLAM front end and taking into account changes in exposure parameter settings in front end modules such as feature trackers or photometric estimators.

[0085] In some embodiments, the method may further include: preprocessing the exposure compensated second image to obtain a third image; sending the third image to a simultaneous positioning and mapping device to construct an environmental map and estimate its own position and posture relative to the environment.

[0086] Exemplarily, the preprocessing may include at least one of the following: image enhancement, filtering, feature extraction, etc.

[0087] An automatic exposure control method optimized for a SLAM system provided in an embodiment of the present application enables the system to respond intelligently under a wide range of lighting conditions, thereby allowing users to use augmented reality (AR) / virtual reality (VR) systems under challenging lighting conditions (e.g., low light indoors, outdoors, HDR environments, etc.). More specifically, the method provides the following advantages: 1. It is conducive to feature detection; 2. It reduces the chance of problematic exposure, thereby improving the robustness of the system; 3. A spatially distributed AE consensus mechanism allows for globally coordinated AE control; 4. It is stable in time; 5. Exposure compensation for stable feature tracking.

[0088] In order to better reflect the purpose of this application, further examples are given based on the above embodiments of this application. Figure 4 As shown, the method specifically includes:

[0089] Step 401: Start;

[0090] Step 402: Initialization;

[0091] Step 403: every N frames of image;

[0092] Step 404: performing AE measurement based on image gradient;

[0093] Step 405: determine whether AE adjustment is triggered; if yes, execute step 406; if no, the camera continues to set the output stream with the current exposure parameters;

[0094] Step 406: Calculate the optimal AE metric;

[0095] Step 407: Calculate the change of exposure time / gain;

[0096] Step 408: If it is not a minor change, execute step 409; if it is a minor change, the camera continues to set the current exposure parameters to output the stream;

[0097] Step 409: adjusting the exposure time / gain according to the change of the exposure time / gain;

[0098] Step 410: Restart the camera stream.

[0099] Specifically, the AE module initializes itself using predetermined settings, and then the AE module periodically reads the current image and associated metadata including the current exposure time and gain, calculates the AE metric based on the current image frame and the metadata, calculates the desired total gain based on the AE metric, and then performs a local search to determine the step size and direction of the desired change in exposure time and / or gain. Finally, a set of optimal exposure times and / or gains are applied to the camera and the camera stream is restarted.

[0100] In some embodiments, the image output by the camera is exposure compensated and preprocessed, and AE measurement is performed on the exposure compensated image, or AE measurement is performed on the preprocessed image, and the preprocessed image is sent to a core SLAM device (such as a smartphone, a VR headset, AR glasses, a robot, etc.) to build an environmental map and estimate its own position and posture relative to the environment.

[0101] With the above technical solution, automatic exposure control is performed using an image gradient-based AE metric, so that the camera can capture images with sufficiently rich texture details, maximize the amount of information provided by the visual front-end system, and thus improve the perception ability and performance of the visual back-end system.

[0102] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides an automatic exposure control device, such as Figure 5 As shown, the automatic exposure control device 50 includes:

[0103] An acquisition unit 501 is configured to acquire a first image captured by a camera under a first exposure parameter;

[0104] A processing unit 502 is configured to determine a gradient of one or more image blocks in the first image based on a pixel value of a reference pixel in the first image;

[0105] The processing unit 502 is further configured to determine an automatic exposure metric according to the gradient of the image block and the objective function of calculating the automatic exposure metric, where the automatic exposure metric is used to measure the quality of the exposure state of the image block; optimize the automatic exposure metric based on the optimization algorithm, and determine a target exposure parameter of the image block corresponding to the optimal automatic exposure metric;

[0106] The processing unit 502 is further configured to determine a target exposure parameter of the camera according to the target exposure parameter of the one or more image blocks;

[0107] The control unit 503 is configured to control the camera outflow according to the target exposure parameter of the camera.

[0108] In some embodiments, the processing unit 502 is specifically configured to determine the sum of square gradients of the image block based on the horizontal component and the vertical component of the gradient of the image block; and determine the automatic exposure metric by using the sum of square gradients of the image block as a variable of the objective function for calculating the automatic exposure metric.

[0109] In some embodiments, the processing unit 502 is specifically configured to use the square of the gradient of the image block as a variable of one or more transformation functions in the objective function, determine one or more gradient transformation parameters of the image block; and determine the automatic exposure metric based on the one or more gradient transformation parameters. The one or more gradient transformation parameters include at least one of the following: a weighted average of the square of the gradient and local statistics, a weighted average of the square of the gradient and global statistics, a regularization term, and a frequency domain feature of a frequency domain range of interest.

[0110] In some embodiments, the processing unit 502 is specifically configured to make decisions on target exposure parameters of multiple image blocks based on a consensus mechanism between image blocks to determine a second exposure parameter; and use a time filter to smooth the second exposure parameter to determine the target exposure parameter of the camera.

[0111] In some embodiments, the consensus mechanism includes: calculating an average value of target exposure parameters of multiple image blocks to obtain a second exposure parameter.

[0112] In some embodiments, the consensus mechanism also includes: determining the exposure state of each image block based on the pixel value of the reference pixel in each image block, the exposure state including one of the following: accurate exposure state, underexposure state and overexposure state; determining the number of image blocks in a first exposure state based on the exposure state of each image block; when the number of image blocks is greater than a first preset value, adjusting the target exposure parameters of the image blocks in the first exposure state to reduce the number of image blocks in the first exposure state.

[0113] In some embodiments, the consensus mechanism also includes: determining the exposure state of each image block based on the pixel value of the reference pixel in each image block, the exposure state including one of the following: accurate exposure state, underexposure state and overexposure state; determining the number of image blocks in a first exposure state based on the exposure state of each image block; when the number of image blocks is greater than a first preset value, adjusting the second exposure parameter to reduce the number of image blocks in the first exposure state.

[0114] In some embodiments, the time filter is a median filter, and the processing unit 502 is specifically configured to obtain a first reference exposure parameter of a plurality of reference frames, where the plurality of reference frames include a plurality of consecutive frames of the first image; use a median filter to determine a median value of the exposure parameters among the second exposure parameter and the first reference exposure parameter of the plurality of reference images; and use the median value of the exposure parameter as a target exposure parameter of the camera.

[0115] In some embodiments, the acquisition unit 501 is further configured to acquire a second image captured by the camera under the target exposure parameter;

[0116] The processing unit 502 is further configured to perform exposure compensation on the second image according to a target exposure parameter of the camera to obtain an exposure-compensated second image.

[0117] In some embodiments, exposure compensation includes: determining one or more exposure compensation parameters of the second image based on a target exposure parameter and a second reference exposure parameter of the camera, the exposure compensation parameters including a first ratio of a reference exposure time to a target exposure time, and / or a second ratio of a reference gain to a target gain; performing exposure compensation on the second image based on the one or more exposure compensation parameters to obtain a compensated second image.

[0118] In some embodiments, the processing unit 502 is further configured to pre-process the exposure compensated second image to obtain a third image; and send the third image to a simultaneous positioning and mapping device to construct an environmental map and estimate its own position and posture relative to the environment.

[0119] In practical applications, the automatic exposure control device can be an image acquisition device, or a chip applied to an image acquisition device. The image acquisition device can be independent of the visual backend system, or it can be a part of the visual backend system. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the automatic exposure control method provided in any of the above embodiments. And the technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the automatic exposure control method, and this application will not go into details one by one.

[0120] The above-mentioned device uses the AE metric based on image gradient for automatic exposure control, so that the camera can collect images with sufficiently rich texture details, maximize the amount of information provided by the visual front-end system, and thus improve the perception ability and performance of the visual back-end system.

[0121] Based on the hardware implementation of each unit in the above automatic exposure control device, the embodiment of the present application also provides an automatic exposure control device, such as Figure 6 As shown, the automatic exposure control device 600 includes: a processor 601 and a memory 602 configured to store a computer program that can be run on the processor;

[0122] The processor 601 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0123] Of course, in practical applications, Figure 6As shown, the various components in the automatic exposure control device 600 are coupled together via a bus system 603. It can be understood that the bus system 603 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 603 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 603 in the figure.

[0124] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0125] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0126] Optionally, the automatic exposure control device 600 may be a chip, and the chip may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0127] Optionally, the chip may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0128] Optionally, the chip may be a chip integrated in a camera, or a chip integrated in a simultaneous positioning and mapping device.

[0129] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of the automatic exposure control device to complete the steps of the aforementioned method.

[0130] An embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the methods described in the embodiments of the present application are implemented.

[0131] Optionally, the computer program product can be applied to the automatic exposure control device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the automatic exposure control device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0132] The embodiment of the present application also provides a computer program.

[0133] Optionally, the computer program can be applied to the automatic exposure control device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the automatic exposure control device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0134] The present application also provides a simultaneous positioning and map building system, such as Figure 7 As shown, the simultaneous positioning and mapping system 700 includes: a camera 701, an automatic exposure control device 702 and a simultaneous positioning and mapping device 703; wherein,

[0135] The camera 701 is configured to capture images and output images to the automatic exposure control device 702;

[0136] The automatic exposure control device 702 is configured to execute the steps of any one of the methods provided in the embodiments of the present application to output an image to the simultaneous positioning and mapping device 703; and

[0137] The simultaneous positioning and mapping device 703 is configured to construct an environment map according to the received images and estimate its own position and posture relative to the environment.

[0138] The simultaneous positioning and mapping device described in this application may include, for example, a smart phone, a tablet computer, a VR headset, an AR glasses, a robot, etc. The camera and the automatic exposure control device may be independent of the simultaneous positioning and mapping device, or may be part of the simultaneous positioning and mapping device, that is, the simultaneous positioning and mapping system may belong to one device or to multiple devices.

[0139] It should be understood that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0140] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0141] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0142] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0145] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0146] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An automatic exposure control method, characterized in that: The method comprises: Acquire a first image captured by a camera under a first exposure parameter; determining a gradient of one or more image blocks in the first image according to a pixel value of a reference pixel in the first image; Determining an automatic exposure metric according to the gradient of the image block and an objective function for calculating the automatic exposure metric, wherein the automatic exposure metric is used to measure the quality of the exposure state of the image block; Optimizing the automatic exposure metric based on an optimization algorithm, and determining a target exposure parameter of the image block corresponding to the optimal automatic exposure metric; Determining a target exposure parameter of the camera according to the target exposure parameters of the one or more image blocks; The camera outflow is controlled according to a target exposure parameter of the camera.

2. The method according to claim 1, characterized in that: The determining the automatic exposure metric according to the gradient of the image block and the objective function of calculating the automatic exposure metric comprises: Determining a square sum of the gradients of the image block according to a horizontal component and a vertical component of the gradient of the image block; The automatic exposure metric is determined by taking the square sum of the gradients of the image block as a variable of the objective function for calculating the automatic exposure metric.

3. The method according to claim 2, characterized in that The step of using the squared gradient of the image block as a variable of the objective function for calculating the automatic exposure metric to determine the automatic exposure metric comprises: Using the square of the gradient of the image block as a variable of one or more transformation functions in the objective function to determine one or more gradient transformation parameters of the image block; determining the automatic exposure metric based on transforming the one or more gradient parameters; The one or more gradient transformation parameters include at least one of the following: a weighted average of the square of the gradient and local statistics, a weighted average of the square of the gradient and global statistics, a regularization term, and a frequency domain feature of a frequency domain range of interest.

4. The method according to claim 1, characterized in that: The determining the target exposure parameter of the camera according to the target exposure parameter of the one or more image blocks includes: Making decisions on target exposure parameters of the plurality of image blocks according to a consensus mechanism between the image blocks, and determining a second exposure parameter; The second exposure parameter is smoothed by using a time filter to determine a target exposure parameter of the camera.

5. The method according to claim 4, characterized in that The consensus mechanism includes: An average value of the target exposure parameters of the multiple image blocks is calculated to obtain the second exposure parameter.

6. The method according to claim 5, characterized in that The consensus mechanism also includes: Determining an exposure state of each image block according to a pixel value of a reference pixel in each image block, wherein the exposure state includes one of the following: an accurate exposure state, an underexposure state, and an overexposure state; Determining the number of image blocks in a first exposure state according to the exposure state of each image block; When the number of the image blocks is greater than a first preset value, the target exposure parameters of the image blocks in the first exposure state are adjusted to reduce the number of the image blocks in the first exposure state.

7. The method according to claim 5, characterized in that The consensus mechanism also includes: Determining an exposure state of each image block according to a pixel value of a reference pixel in each image block, wherein the exposure state includes one of the following: an accurate exposure state, an underexposure state, and an overexposure state; Determining the number of image blocks in the first exposure state according to the exposure state of each image block; When the number of image blocks is greater than a first preset value, the second exposure parameter is adjusted to reduce the number of image blocks in the first exposure state.

8. The method according to claim 4, characterized in that The time filter is a median filter, and the step of using the time filter to smooth the second exposure parameter to determine the target exposure parameter of the camera includes: Acquire a first reference exposure parameter of a plurality of reference image frames, wherein the plurality of reference image frames include a plurality of consecutive image frames of the first image; Using the median filter, determine a median value of the exposure parameters among the second exposure parameter and the first reference exposure parameters of the multiple frames of reference images; The median of the exposure parameters is used as the target exposure parameter of the camera.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire a second image captured by the camera under target exposure parameters; Exposure compensation is performed on the second image according to the target exposure parameter of the camera to obtain a second image after exposure compensation.

10. The method according to claim 9, characterized in that The exposure compensation includes: Determining one or more exposure compensation parameters of the second image according to a target exposure parameter and a second reference exposure parameter of the camera, the exposure compensation parameters comprising a first ratio of a reference exposure time to a target exposure time, and / or a second ratio of a reference gain to a target gain; Exposure compensation is performed on the second image according to the one or more exposure compensation parameters to obtain a compensated second image.

11. The method according to claim 9, characterized in that The method further comprises: Preprocessing the exposure-compensated second image to obtain a third image; The third image is sent to a simultaneous localization and mapping device to construct an environment map and estimate the position and posture of the device relative to the environment.

12. An automatic exposure control device, characterized in that: The device comprises: An acquisition unit is configured to acquire a first image captured by a camera under a first exposure parameter; a processing unit configured to determine a gradient of one or more image blocks in the first image based on a pixel value of a reference pixel in the first image; The processing unit is further configured to determine an automatic exposure metric according to the gradient of the image block and an objective function for calculating the automatic exposure metric, wherein the automatic exposure metric is used to measure the quality of the exposure state of the image block; optimize the automatic exposure metric based on an optimization algorithm, and determine a target exposure parameter of the image block corresponding to the optimal automatic exposure metric; The processing unit is further configured to determine a target exposure parameter of the camera according to the target exposure parameters of the one or more image blocks; A control unit is configured to control the camera outflow according to a target exposure parameter of the camera.

13. An automatic exposure control device, characterized in that: a processor and a memory configured to store a computer program capable of being executed on the processor, Wherein, the processor is configured to execute the steps of the method described in any one of claims 1 to 11 when running the computer program.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A simultaneous positioning and mapping system, characterized in that: The system includes: a camera, an automatic exposure control device and a simultaneous positioning and mapping device; wherein, The camera is configured to capture images and output images to the automatic exposure control device; The automatic exposure control device is configured to perform the steps of the method according to any one of claims 1 to 11 to output an image to the simultaneous positioning and mapping device; and The simultaneous positioning and mapping device is configured to construct an environment map based on the received images and estimate its own position and posture relative to the environment.

Images