Method and apparatus for synchronously adjusting the exposure of a multi-lens camera
By calculating the brightness differences between multiple lenses and adjusting the exposure parameters, the problem of brightness differences between lenses in panoramic images was solved, improving the stitching quality and visual effect, and achieving synchronous exposure adjustment.
Patent Information
- Application Number
- CN202510581837.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies have failed to effectively address the issue of brightness differences between lenses in panoramic image stitching, resulting in decreased stitching quality and an inability to synchronously adjust the exposure of multi-lens cameras, thus affecting the overall visual effect of panoramic images.
By calculating the brightness difference between multiple images to be stitched, and using an exposure compensation engine and image signal processor, a cost function is calculated and exposure parameters are adjusted to achieve synchronous exposure adjustment of multiple lenses, thereby reducing the brightness difference in overlapping areas and improving stitching quality.
It accelerates the convergence speed of panoramic images, improves stitching quality and overall visual effect, and provides greater convenience for subsequent applications.
Smart Images

Figure CN123123729A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to panoramic image processing, and more particularly to a method and apparatus for synchronously adjusting the exposure of a multi-lens camera. Background Technology
[0002] Panoramic images typically have an extremely wide horizontal field of view (covering 180 degrees or more) and can also have a significant vertical field of view. Panoramic images are usually stitched together from multiple images to present a scene that is wider than the range of human vision at a time, thus creating an immersive experience.
[0003] As panoramic imaging technology matures, it has been widely applied in various fields such as video conferencing, video surveillance, virtual reality, robot navigation, and smart factories. The basic principle of panoramic imaging technology is to find similar overlapping areas in multiple images and then stitch these overlapping images together to form a complete panoramic image. However, if there are significant differences in brightness / luminance between two or more images, it will negatively impact the stitching quality of the panoramic image. For example, it can cause uneven brightness / luminance in the overlapping areas after stitching, leading to a reduction in overall image quality.
[0004] Chinese Patent Publication No. 111741228 discloses an exposure adjustment method and apparatus for panoramic images. It adjusts the exposure of all images before stitching, but because it does not consider the brightness of overlapping areas, the adjustment effect is not ideal. Furthermore, Chinese Patent Publication No. 104125410 discloses a panoramic video multi-lens exposure compensation method and apparatus. Referring to Figure 3 of that patent document, the rightmost panoramic partition image 4 is not aligned with the leftmost panoramic partition. Figure 1 The stitching process is straightforward, making adjustments significantly easier. However, when panoramic partitioning... Figure 1 When Figure 4 requires splicing and there are brightness differences in the splicing area, this patent document cannot provide any solution.
[0005] Therefore, there is an urgent need for a new solution that can simultaneously adjust the exposure of multi-lens cameras to reduce brightness / luminosity differences in overlapping areas of panoramic images, accelerate convergence speed, and improve stitching quality. This invention is a solution to these needs. Summary of the Invention
[0006] In view of the above problems, the purpose of this invention is to provide an apparatus and method for synchronously adjusting the exposure parameters of a multi-lens camera. By calculating a cost function based on the main brightness differences of multiple images to be stitched together, and by determining the new exposure parameters of each lens for capturing the next frame based on these main brightness differences and the cost function, the brightness / exposure of the multi-lens camera can be synchronously and adaptively adjusted. This reduces the difference in brightness / luminosity values in overlapping areas of panoramic (or wide-angle) images, accelerates convergence speed, improves stitching quality, enhances the overall visual effect of panoramic (or wide-angle) images, and provides greater convenience for subsequent applications.
[0007] According to an embodiment of the present invention, an apparatus for synchronously adjusting the exposure of a multi-lens camera is provided, comprising a plurality of image signal processors and an exposure compensation engine. Each image signal processor includes an exposure statistics circuit and is coupled to the multi-lens camera, wherein each exposure statistics circuit is configured to measure the brightness information of each lens image in a plurality of lens images captured by the multi-lens camera. The exposure compensation engine is coupled to the multi-lens camera and the image signal processors and is configured to perform a set of operations, the set of operations comprising: (1) calculating a principal brightness difference based on the brightness information of each lens image in the plurality of lens images and a target brightness; (2) calculating a cost function based on the principal brightness difference of the plurality of lens images; and (3) if the cost function is greater than a first threshold value, obtaining a first parameter adjustment amount for each of at least one lens image in the lens images based on its own principal brightness difference, and obtaining and transmitting an exposure parameter to the multi-lens camera based on the first parameter adjustment amount for lens exposure adjustment.
[0008] According to another embodiment of the present invention, a method for synchronously adjusting the exposure of a multi-lens camera is provided, comprising the following steps: (a) measuring the brightness information of each lens image in a plurality of lens images by means of a plurality of image signal processors coupled to the multi-lens camera; (b) calculating a principal brightness difference based on the brightness information of each lens image in the plurality of lens images and a target brightness; (c) calculating a cost function based on the principal brightness differences of the plurality of lens images; and (d) if the cost function is greater than a first threshold value, obtaining a first parameter adjustment amount for each of at least one lens image in the lens images based on its own principal brightness difference, and obtaining and transmitting an exposure parameter to the multi-lens camera based on the first parameter adjustment amount for lens exposure adjustment.
[0009] The above and other objects and advantages of the present invention will be described in detail below with reference to the following figures, detailed descriptions of the embodiments and claims. Attached Figure Description
[0010] Figure 1 This is a block diagram showing an exposure adjustment device for a multi-lens camera according to an embodiment of the present invention.
[0011] Figure 2A Two different side views of an outward-facing quad-lens camera mounted on architecture 1A are shown.
[0012] Figure 2B An example of an outward-facing triple-lens camera mounted on architecture 1B.
[0013] Figure 2C An example of an inward-facing triple-lens camera mounted on architecture 1C.
[0014] Figure 2D An example of an outward-facing dual-lens camera mounted on a 1D architecture.
[0015] Figure 2E An example of an inward-facing dual-lens camera mounted on architecture 1E.
[0016] Figure 3A A flowchart of an exposure adjustment method for a multi-lens camera 11 is shown for use in accordance with a first embodiment of the present invention.
[0017] Figure 3B A detailed flowchart of the parameter decision operation performed by the exposure compensation engine 150 is shown for an embodiment of the present invention.
[0018] Figure 4A and Figure 4B A flowchart of an exposure adjustment method for a multi-lens camera 11 is shown for use in accordance with a second embodiment of the present invention.
[0019] Figure 5A Displaying an isometric rectangular panoramic image with a limited vertical field of view, originating from an installation mounted on... Figure 2A The four lenses captured by the four-lens camera 11A on the four working faces of the cubic structure are equidistant rectangular projections, wherein the field of view captured by the four-lens camera 11A covers the horizontal field of view of X degrees and the vertical field of view of Y degrees, and X = 360 and Y < 180.
[0020] Figure 5B Displaying an equidistant rectangular panoramic image, originating from a device mounted on... Figure 2A The equidistant rectangular projection of four lens images captured by a four-lens camera 11A on four working faces of a cubic structure, wherein the field of view captured by the four-lens camera 11A covers an X-degree horizontal field of view and a Y-degree vertical field of view, and X = 360 and Y = 180.
[0021] Figure 5C and Figure 5D A flowchart of an exposure adjustment method for a multi-lens camera is shown for use in accordance with a third embodiment of the present invention.
[0022] Figure 6A and Figure 6B A flowchart of an exposure adjustment method for a multi-lens camera is shown according to a fourth embodiment of the present invention.
[0023] Figure 7A and Figure 7B A flowchart of an exposure adjustment method for a multi-lens camera is shown according to a fifth embodiment of the present invention.
[0024] Figure 8 Displaying a wide-angle stitched image, originating from a device erected on... Figure 2B The equidistant rectangular projection of three lens images captured by a three-lens camera on three working faces of a cubic structure, wherein the field of view captured by the three-lens camera covers an X-degree horizontal field of view and a Y-degree vertical field of view, and X = 270 and Y = 180.
[0025] Icon labels:
[0026] 1A, 1B, 1C, 1D, 1E Cube Architecture
[0027] 11 Multi-lens camera
[0028] 11A Quad Lens Camera
[0029] 11B and 11C triple-lens cameras
[0030] 11D and 11E dual-lens cameras
[0031] 110-11N lens
[0032] 100 Exposure Adjustment Device
[0033] 120 Input Port
[0034] 130 Image Signal Processor
[0035] 131 Exposure Statistics Unit
[0036] 140 System Bus
[0037] 150 exposure compensation units
[0038] 160 splicing engine
[0039] 170 Volatile Memory Devices
[0040] 180 Non-volatile memory devices
[0041] 190 Output Port
[0042] Overlapping areas of JR0~JR3 and JL0~JL3
[0043] w1 overlap area width
[0044] w2 Single lens image width Detailed Implementation
[0045] The use of the term "and / or" throughout this specification and in the following claims includes all combinations of any one or more of the related listed items. The singular forms of the terms "a" and "the" throughout this specification and in the following claims include both singular and plural meanings, unless otherwise specified in this specification. Related terms used throughout this specification and in the following claims are defined as follows, unless otherwise specified in this specification. Throughout this specification, elements with the same function use the same reference numerals.
[0046] The terms used throughout this specification and in the following claims are defined as follows, unless otherwise specified in this specification. The term "panoramic image" refers to an image with an extremely wide horizontal field of view (HFOV) (covering 180 degrees or greater) and simultaneously possessing a significant vertical field of view. The term "wide-angle image" refers to an image with a field of view larger than that of the human eye. A panoramic image is a type of wide-angle image. Both panoramic and wide-angle images can be obtained by stitching together images from multiple lenses (captured by multiple lenses of a camera at different angles). This invention... Figures 3A-3BThe exposure adjustment methods shown in 4A-4B, 5C-5D, 6A-6B, and 7A-7B operate equivalent to the Automatic Exposure (AE) algorithm. The term "AE algorithm" refers to an algorithm that operates like a server system, continuously monitoring the exposure value of each frame generated by the image signal processor (ISP); if the measured exposure value exceeds the permissible range around the target value, the algorithm immediately intervenes and adjusts the exposure adjustment device 100 to quickly restore the exposure value to the permissible operating range. Within a single response cycle, the specific tasks that the AE algorithm (or the exposure adjustment method of this invention) needs to perform include: evaluating the exposure quality of the current frame based on the exposure statistics of the current lens image generated by the ISP; if the exposure quality needs adjustment, generating new exposure parameters for the next frame based on the current exposure parameters and the ideal exposure target value; writing the new exposure parameters into the corresponding hardware devices of the camera to drive the aperture to different sizes, adjust the sensor shutter speed or exposure duration, and / or adjust the gain (digital gain and analog gain). The term "response cycle" refers to the time required for the exposure adjustment device 100 to complete the following processes sequentially: receiving the current exposure parameters from the multi-lens camera 11; the multi-lens camera 11 outputting image data; the image signal processor 130 processing the lens image; the exposure compensation engine 150 performing exposure compensation based on the lens image; generating a projected image with the stitching engine 160; and updating the exposure parameters with the exposure compensation engine 150.
[0047] One feature of this invention is that it calculates a cost function based on luminance statistics of the images to be stitched. Another feature of this invention is that it determines the new exposure parameters (e.g., exposure time) of each lens of the multi-lens camera for the next frame of the projected image based on the luminance statistics and the cost function, thereby synchronously and adaptively adjusting the luminance of each lens of the multi-lens camera to improve the stitching quality.
[0048] Figure 1 This is a block diagram showing an exposure adjustment device 100 for a multi-lens camera according to an embodiment of the present invention. (Reference) Figure 1The exposure adjustment device 100 is configured to synchronously adjust the exposure of the multi-lens camera 11, and includes (N+1) input ports 120, (N+1) image signal processors (ISPs) 130, a system bus 140, an exposure compensation engine 150, a volatile memory (VM) device 170, and a non-volatile memory (NVM) device 180, where N ≥ 1. The lenses 110 to 11N of the multi-lens camera 11 are coupled to each image signal processor 130 via the input ports 120. According to the design of this embodiment, each image signal processor 130 includes an exposure statistics unit 131, and each exposure statistics unit 131 can be implemented using an integrated circuit device such as a programmable processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated processor component, or any combination thereof. In one embodiment, the exposure adjustment device 100 may be a system on a chip (SoC) integrated into a computing device (e.g., a mobile phone, tablet computer, wearable computer, etc.) to perform synchronous exposure adjustment operations of the multi-lens camera 11 and generate (N+1) processed lens images in the volatile memory device 170.
[0049] The multi-lens camera 11 can capture still images or moving images, and the multi-lens camera 11 can be a panoramic multi-lens camera (e.g., Figure 2A The quad-lens camera 11A shown) or a wide-angle multi-lens camera (e.g.) Figure 2B Outward-facing triple-lens camera 11B Figure 2C The inward-facing triple-lens camera 11C Figure 2D Outward-facing twin-lens camera 11D, and Figure 2E The inward-facing twin-lens camera 11E is shown. Figure 2B , 2C In the architecture of the multi-lens camera 11B / 11C, the three lenses (lens-0, lens-1, and lens-2) mounted on the 1B / 1C form a 120-degree angle between any two adjacent edges / faces. Figure 2D , Figure 2EIn the multi-lens camera 11D / 11E, the two lenses (lens-0, lens-1) mounted on the architecture 1D / 1E form a 120-degree angle between any two adjacent edges / surfaces. Note that this 120-degree angle is merely illustrative and not a limitation of the invention; in actual implementation, any two adjacent edges / surfaces of the architecture 1B / 1C / 1D / 1E may present other angles. The multi-lens camera 11 can obtain a field of view that simultaneously covers an X-degree horizontal field of view (HFOV) and a Y-degree vertical field of view (VFOV) to generate multiple lens images, where X≦360 and Y≦180 (e.g., 360×160 or 180×90). For example, Figure 2A The four-lens camera 11A has four lenses (lens-0 to lens-3) mounted on the four working surfaces (left, right, front, and back) of a cubic structure 1A to obtain an external field of view with a 360-degree horizontal FOV and a 90-degree vertical FOV, and to generate four lens images. Note that it is only necessary to capture a field of view covering an X-degree horizontal FOV and a Y-degree vertical FOV (X≦360, Y≦180). The number of lenses in the multi-lens camera 11 of this invention is not limited, but the fields of view of any two adjacent lenses must form a sufficient overlap area to facilitate image stitching.
[0050] The input port 120 and output port 190 of the exposure adjustment device 100 can be of a conventional design and include circuitry for adjusting data to conform to a high-speed serial interface standard, such as Mobile Industry Processor Interface (MIPI), Universal Serial Bus (USB), or Ethernet. The following embodiment uses an input port 120 as a MIPI port and an output port 190 as a USB port as an example, but the invention is not limited to this and can be extended to any existing or future high-speed serial interface standard.
[0051] The volatile memory (VM) device 170 is used to store various data required by the (N+1) image signal processors 130 during operation, including programs or image data captured by the multi-lens camera 11. The non-volatile memory (NVM) device 180 contains multiple programs or instructions executed by the (N+1) image signal processors 130 respectively to complete... Figures 3A-3B , Figures 4A-4B , Figures 5C-5D , Figures 6A-6B and Figures 7A-7BThe complete steps of the exposure adjustment method are shown below. Furthermore, each image signal processor 130 can operate independently and execute programs stored in the volatile memory (VM) device 170 and the non-volatile memory device 180, performing various data processing and exposure statistics calculations on image data from the corresponding lenses 11k (k = 0 to N) stored in the volatile memory device 170. Specifically, these image signal processors 130 receive electronic signals from sensors (not shown) of the corresponding lenses 110 to 11N in the multi-lens camera 11 via input ports 120, and then convert the electronic signals into corresponding digital lens images. Then, each image signal processor 130 or each exposure statistics unit 131 performs exposure statistics calculations on its corresponding digital lens image to generate current exposure statistics information. The exposure compensation engine 150 then evaluates the exposure quality of each lens image based on the current exposure statistics. If the exposure quality of a certain lens image needs adjustment, the exposure compensation engine 150 generates and transmits updated exposure parameters to the multi-lens camera 11 for use in the next frame (of the projected image) based on the current exposure parameters and the ideal exposure value to adjust the exposure of each lens, such as adjusting the aperture size, sensor shutter speed, and / or digital / analog gain of each lens. Next, the stitch engine 160 executes the program stored in the non-volatile memory device 180 and performs image stitching based on (N+1) lens images in the volatile memory device 170 to form a projection image, such as a panoramic image or a wide-angle image. Finally, the projection image is output through the output port 190. According to the present invention, the above-mentioned exposure parameters include, but are not limited to, an aperture value, an exposure time, a digital gain value, and an analog gain value. The analog gain value is used to directly amplify the light signal received by the CCD sensor, while the digital gain value is used to amplify the digital signal (obtained by converting the CCD sensor output through an analog-to-digital converter (ADC)). Generally, to achieve the required image brightness, the order of adjusting the camera's exposure parameters is as follows: aperture value, exposure time, analog gain value, and digital gain value. Therefore, the aperture size is adjusted first, and the shutter speed is adjusted only when the aperture cannot be increased further (i.e., the aperture value cannot be decreased further).However, the longer the exposure time, the more severe the motion blur will be. Therefore, a maximum exposure time is usually set. When the exposure time reaches the limit, the analog gain value will be adjusted. Finally, when the previous three parameters (aperture value, exposure time and analog gain value) have all reached their limits, the digital gain value will be adjusted to achieve the desired target brightness.
[0052] Many projection methods are applicable to the exposure adjustment device 100 of the present invention. These projection methods include, but are not limited to, equidistant rectangular projection, cylindrical projection, and modified cylindrical projection. Modified cylindrical projection includes, but is not limited to, Miller projection, Mercator projection, Lambert cylindrical equal area projection, Pannini projection, etc. Accordingly, the above-mentioned projected image includes, but is not limited to, an equidistant rectangular projection image, a cylindrical projection image, and a modified cylindrical projection image. For ease of description, the examples and embodiments below will be described using equidistant rectangular projection and equidistant rectangular panoramic images, and it is assumed that (1) the exposure parameters are equal to the above-mentioned exposure time and (2) the panoramic multi-lens camera 11A has four lenses 110-113, respectively mounted on the four working surfaces (left, right, front, and back) of the cubic structure 1A. The operation of the exposure adjustment device 100 of the present invention is also applicable to the technical scope of wide-angle cameras, cylindrical projection, or modified cylindrical projection.
[0053] Figure 3A A flowchart illustrating an exposure adjustment method for a multi-lens camera 11 according to a first embodiment of the present invention is shown. Reference will be made below. Figure 1 and Figure 3A The document details the exposure adjustment method performed by four image signal processors 130 (or four exposure statistics units 131) and an exposure compensation engine 150.
[0054] Step S301: The four image signal processors 130 respectively receive electronic signals from the image sensors of lenses 110 to 113 and convert these electronic signals into four lens images.
[0055] Step S302: The four image signal processors 130 or the four exposure statistics units 131 measure the four average brightness (Y0, Y1, Y2 and Y3) of the four lens images respectively, where N=3.
[0056] Step S304: Using exposure compensation engine 150, calculate the average brightness Y for each lens image. k The principal luminance difference between the target luminance Tg and the target luminance. Where k = 0, 1, 2, 3. In one embodiment,
[0057] Step S306: Using the exposure compensation engine 150, based on the above four main brightness differences... Calculate a cost function C. In one embodiment, In another embodiment,
[0058] Step S308: Using the exposure compensation engine 150, determine whether the cost function C is greater than a critical value TH1. If yes, proceed to step S310; otherwise, proceed to step S314.
[0059] Step S310: Using the exposure compensation engine 150, based on the current exposure step... Dynamic adjustment quantity and main brightness difference Perform a parameter decision operation on the image from lens -k to determine the next exposure step. and one parameter adjustment amount Where k = 0, 1, 2, 3. For simplicity, step S310 is... Let k = 0, 1, 2, 3, and func() represent the parameter decision operation performed by the exposure compensation engine 150 (see details). Figure 3B In this invention, func() can be considered as being generated by... Figure 3A , 4A In 5C, 6A, and 7A, the main process calls or invokes a subroutine, where... and These represent the input arguments passed to the three subroutines. These represent the two output arguments returned from this subroutine. This subroutine is only executed when `func()` is called. Indicates the step size used to adjust the current exposure. The dynamic adjustment amount, This indicates the exposure step size of the current frame (of the isometric rectangular panoramic image). This indicates the exposure step size of the next frame (of the isometric rectangular panoramic image), and This indicates the amount of adjustment to the exposure parameters.
[0060] Step S312: Using exposure compensation engine 150, according to The exposure parameters of lenses 110-113 are updated, and the updated exposure parameters of the next frame (of the isometric rectangular panoramic image) are transmitted to the multi-lens camera 11 for lens exposure adjustment, such as adjusting the aperture value, exposure time, and / or digital / analog gain of each lens, where k = 0, 1, 2, 3. For example, the exposure compensation engine 150 can update the exposure time of lenses 110-113 according to the following formula and transmit the updated exposure time to the camera 11: Where T k T′ represents the current exposure time of lens 11k. k This indicates the exposure time for the 11k lens to capture the next frame. Then, return to step S301 to process the next frame (the isometric rectangular panoramic image).
[0061] Step S314: Keep the exposure parameters unchanged, for example, the exposure time T0 to T3 remains unchanged. Then, return to step S301 to process the next frame (the equidistant rectangular panoramic image).
[0062] Figure 3B To illustrate a detailed flowchart of the parameter decision operation performed by the exposure compensation engine 150 according to an embodiment of the present invention, the following references are made. Figure 1 and Figure 3A , 3B This describes the parameter decision operation (i.e., (S',Δ)=func(S,ε,D)), where ε represents the dynamic adjustment amount used to correct the input exposure step size S, D represents the brightness difference, S' represents the output exposure step size, and Δ represents the parameter adjustment amount of the exposure parameter.
[0063] Step S362: Determine if the brightness difference D is greater than zero. If yes, proceed to step S364; otherwise, proceed to step S374.
[0064] Step S364: Determine whether the absolute value of the brightness difference D (abs(D)) is less than a critical value TH2. If yes, proceed to step S370; otherwise, proceed to step S366. Here, abs(D) represents a function used to return the absolute value of D.
[0065] Step S366: Determine whether the absolute value of the brightness difference D (abs(D)) is greater than a critical value TH3. If yes, proceed to step S372; otherwise, proceed to step S368. Here, the critical value TH3 is greater than the critical value TH2.
[0066] Step S368: Keep the exposure step size S unchanged, and set the parameter adjustment amount Δ to be equal to S′, that is, S′=S and Δ=S′, and then proceed to step S384.
[0067] Step S370: Decrease the exposure step size S by a value ε, that is, S' = S - ε, and set the parameter adjustment amount Δ to be equal to S' (i.e., Δ = S'), and then proceed to step S384.
[0068] Step S372: Increase the exposure step size S by a value ε, that is, S' = S + ε, and set the parameter adjustment amount Δ to be equal to S' (i.e., Δ = S'), and then proceed to step S384.
[0069] Step S374: Determine whether the absolute value of the brightness difference D (abs(D)) is less than the critical value TH2. If yes, proceed to step S380; otherwise, proceed to step S376.
[0070] Step S376: Determine whether the absolute value of the brightness difference D (abs(D)) is greater than the critical value TH3. If yes, proceed to step S382; otherwise, proceed to step S378.
[0071] Step S378: Keep the exposure step size S unchanged (i.e., S' = S), and set the parameter adjustment amount Δ to be equal to (-S') (i.e., Δ = -S'), and then proceed to step S384.
[0072] Step S380: Decrease the exposure step size S by a value ε, that is, S' = S - ε, and set the parameter adjustment amount Δ to be equal to (-S') (that is, Δ = -S'), and then proceed to step S384.
[0073] Step S382: Increase the exposure step size S by a value ε, that is, S' = S + ε, and set the parameter adjustment amount Δ to be equal to (-S') (that is, Δ = -S'), and then proceed to step S384.
[0074] Step S384: Return the above S' value and Δ value.
[0075] Example 1 (corresponding) Figure 3A First embodiment)
[0076] Assume Tg = 100, TH1 = 50, TH2 = 10, TH3 = 50, Furthermore, the cost function C has four main brightness differences. The square root of the sum of squares And assume that the following four average brightness values are measured in step S302: Y0 = 5, Y1 = 130, Y2 = 10, Y3 = 99. Then, according to step S304, the following is obtained... And the cost function is obtained according to step S306. Because the cost function C is greater than TH1, the process proceeds to step S310. The average brightness Y... k When the value is much smaller than Tg (i.e., D>TH3) (e.g., Y0=5, Y2=10), the exposure step size is increased according to step S372, resulting in... and (k=0 and 2). When the average luminance Y k is close to Tg (i.e., D<TH2) (for example, Y3=99), the exposure step size is reduced according to step S370, such that and when the average luminance Y k is greater than Tg, and TH2<|D|<TH3 (for example, Y1=130), the current exposure step size S is maintained unchanged according to step S378, and the parameter adjustment amount Δ is set to -S, such that Finally, according to step S312, the exposure compensation engine 150 updates and transmits the following four exposure times of the lenses 110 to 114 to the camera 11 for capturing the next frame: T′0=T0+6, T′1=T1-5, T′2=T2+6, T′3=T3+4.
[0077] Example 2 (corresponding to Figure 3A the first embodiment):
[0078] Assume that Tg=55, TH1=100, TH2=45, TH3=50, and the cost function C is equal to and assume that the following four average luminance values Y0=5, Y1=135, Y2=15, Y3=95 are measured in step S302. Then, according to step S304, and the cost function is obtained according to step S306 Since the cost function C is less than TH1, the process proceeds to step S314, and the exposure times T1 to T4 remain unchanged. It should be noted that in Example 1 and Example 2, the basis for calculating the main luminance difference and the manner of calculating the cost function C based on the main luminance difference are only illustrative and not intended to limit the present invention. In actual implementation, other calculation methods may be adopted for the main luminance difference and the cost function C, which still fall within the scope of the present invention.
[0079] Figure 4A and Figure 4B are flowcharts showing an exposure adjustment method for the multi-lens camera 11 according to the second embodiment of the present invention. In the following, reference will be made to Figure 1 and Figure 4A , 4B to describe in detail the exposure adjustment method executed by the four image signal processors 130 (or the four exposure statistics units 131) and the exposure compensation engine 150. It should be noted that in all the flowcharts, functional blocks with the same function are denoted by the same reference numerals, and thus repeated descriptions of the same functional modules are omitted below.
[0080] Step 406: Using exposure compensation engine 150, from Among the absolute values of the four, find the largest brightness difference. (i.e., maximum brightness difference)
[0081] Step S408: Using exposure compensation engine 150, based on the maximum brightness difference The cost function C is calculated. In one embodiment, the cost function...
[0082] Step S410: Using exposure compensation engine 150, set the k value to 0, that is, k = 0.
[0083] Step S412: Using exposure compensation engine 150, determine if the k value is less than 4. If yes, proceed to step S416; otherwise, skip to step S314.
[0084] Step S414: Using exposure compensation engine 150, determine... Is the absolute value equal to? If yes, proceed to step S415; otherwise, proceed to step S420.
[0085] Step S415: Using exposure compensation engine 150, based on the current exposure step size Dynamic adjustment amount and main brightness difference Perform parameter decision operations on the lens-k image to determine the next exposure step. and one parameter adjustment amount Δ k For simplicity, step S415 can be represented as follows: Furthermore, func() represents the parameter decision operation performed by the exposure compensation engine 150 (see details). Figure 3B As mentioned earlier, func() can be viewed as a function of... Figure 4A , 4B The subroutine called or loaded by the main process.
[0086] Step S416: Using exposure compensation engine 150, according to The exposure parameters of lens 11k are updated, and the updated exposure parameters for the next frame (of the isometric panoramic image) are transmitted to multi-lens camera 11 for lens exposure adjustment. For example, exposure compensation engine 150 can update the exposure time of lens 11k and transmit the updated exposure time to camera 11 for capturing the next frame according to the following formula:
[0087] Step S418: Using exposure compensation engine 150, increment the k value by 1, i.e., k = k + 1, and then return to step 412 to process the next shot image.
[0088] Step S420: Keep the exposure parameters constant. For example, the exposure time T... k Keep it unchanged. Then, proceed to step S418.
[0089] Example 3 (corresponding) Figure 4A , 4B Second embodiment):
[0090] Assume Tg = 55, TH1 = 60, TH2 = 30, TH3 = 90. And the cost function C equals And assume that the following four average brightness values are measured in step S302: Y0 = 5, Y1 = 130, Y2 = 10, Y3 = 99. Then, according to step S304, the following is obtained... And the cost function is obtained according to step S408. Because C is greater than TH1 and Equal to the maximum brightness difference Therefore, the exposure time of lens 111 needs to be adjusted, while the exposure times of the other lenses remain unchanged. Next, according to... Figure 3B Step S378, because Therefore, the exposure step size remains unchanged. And will Set equal to that is Finally, according to step S416 or S420, the exposure compensation engine 150 updates and transmits the exposure times of lenses 110 to 113: T′0 = T0, T′1 = T1-5, T′2 = T2, T′3 = T3.
[0091] Back Figure 2A The quad-lens camera 11A comprises four lenses 110-113, mounted on the four working faces (right, left, front, and back) of the cubic structure 1A. Any two adjacent working faces of this cubic structure are substantially orthogonal, for example, facing directions of 0°, 90°, 180°, and 270° longitude on a virtual sphere, respectively, to instantly capture images with a 360-degree horizontal field of view and a 90-degree vertical field of view, generating four images. These four lenses 110-113 must satisfy the requirement of sufficient overlap between the fields of view of any two adjacent lenses to facilitate image stitching. (Reference) Figure 5A and Figure 5BPixels located in the overlapping regions JR0-JR3 and JL0-JL3 are formed by overlapping images from two adjacent shots, while pixels located in the non-overlapping region 15 come from images from a single shot. The stitching engine 160 performs stitching / blending operations on the overlapping regions JR0-JR3 and JL0-JL3 respectively to form the equidistant rectangular panoramic image. Generally, the width w2 in each overlapping region JR0-JR3 and JL0-JL3 is less than (1 / 3 × w1), where w1 represents the width of a single shot image.
[0092] Figure 5C and Figure 5D A flowchart of an exposure adjustment method for a multi-lens camera 11 is shown to illustrate the method according to a third embodiment of the present invention. Reference will be made below. Figure 1 , Figures 5A-5D The method for adjusting the exposure is described in detail through four image signal processors 130 (or four exposure statistics units 131) and an exposure compensation engine 150, where N=3.
[0093] Step S502: Measure the image signal processors 130 or the exposure statistics units 131 respectively. Figure 5A or Figure 5B The average brightness (L0, L1, L2, L3, R0, R1, R2, R3) of the eight overlapping regions JR0~JR3 and JL0~JL3 shown.
[0094] Step S504: Using exposure compensation engine 150, calculate the left overlap region JL of the lens-k image. k Average brightness and right overlap region JR of the lens-((k+3)mod 4) image (k+3)mod Brightness difference between 4 average brightness And calculate the right overlap region JR of the lens-k image. k Average brightness and left overlap region JL of lens-((k+1)mod 4) image (k+1)mod Brightness difference between 4 average brightness in And k = 0, 1, 2, 3. (Reference) Figure 5A and Figure 5B The image of lens -((k+3)mod 4) is located to the left of the image of lens -k, and the image of lens -((k+1)mod 4) is located to the right of the image of lens k.
[0095] Step S506: Using exposure compensation engine 150, based on each lens image... and Calculate the auxiliary brightness difference Where k = 0, 1, 2, 3. In one embodiment, In another embodiment,
[0096] Step S508: With exposure compensation engine 150, based on and The cost function C is calculated, where k = 0, 1, 2, 3. In one embodiment, In another embodiment,
[0097] Step S510: Using exposure compensation engine 150, based on the current exposure step size Dynamic adjustment amount and auxiliary brightness difference Perform parameter decision operations on the image from lens -k to determine the next exposure step. and parameter adjustment amount This step is similar to step S310, and is based on... This indicates that k = 0, 1, 2, 3 and func() represents the parameter decision operation performed by the exposure compensation engine 150 (e.g., Figure 3B (As shown).
[0098] Step S512: Using exposure compensation engine 150, according to as well as Calculate Δ k Where k = 0, 1, 2, 3. In one embodiment,
[0099] Step S514: Using exposure compensation engine 150, according to Δ k The exposure parameters of lenses 110-113 are updated, and the updated exposure parameters of the next frame (of the isometric rectangular panoramic image) are transmitted to the multi-lens camera 11 for lens exposure adjustment. For example, the exposure time of lenses 110-113 can be updated and transmitted to camera 11 according to the following formula:
[0100] Example 4 (corresponding) Figure 5C , 5D The third embodiment and Figure 5A or Figure 5B (Isometric rectangular panoramic image):
[0101] Assume Tg = 55, TH1 = 9, TH2 = 4, TH3 = 96. and (Step S506), and assume that the following four average brightness values were measured in step S302: Y0 = 90, Y1 = 50, Y2 = 95, Y3 = 95; and that the average brightness values of the eight overlapping regions JR0~JR3 and JL0~JL3 were measured in step S502: L0 = 80, L1 = 50, L2 = 95, L3 = 80, R0 = 90, R1 = 40, R2 = 80, R3 = 80. Then, according to step 304, the following is obtained... And according to step 504, Then, according to step 506, the following is obtained: And the cost function is calculated according to step 508. If α = 0.5 and n = 2, then C ≒ 125. Because the cost function C is greater than TH1, the exposure parameters need to be determined based on... and To adjust. According to step S310, we obtain... and According to step S510, we obtain Next, let's assume (Step S512) yields Δ0 = -2, Δ1 = 2, Δ2 = -2, and Δ3 = -7. Finally, the exposure compensation engine 150 updates the following exposure times for lenses 110 to 113 and transmits the updated exposure times to camera 11: T′0 = T0 - 2, T′1 = T1 + 2, T′2 = T2 - 2, and T′3 = T3 - 7.
[0102] Example 5 (corresponding) Figure 5C , 5D The third embodiment and Figure 5A or Figure 5B (Isometric rectangular panoramic image):
[0103] Assume Tg = 55, TH1 = 95, TH2 = 4, TH3 = 96. and (Step S506), and assume that the following four average brightness values were measured in step S302: Y0 = 90, Y1 = 50, Y2 = 95, Y3 = 95, and the average brightness values of eight overlapping regions JR0~JR3 and JL0~JL3 were measured in step S502: L0 = 80, L1 = 50, L2 = 95, L3 = 80, R0 = 90, R1 = 40, R2 = 80, R3 = 80. Then, according to step 304, the following is obtained... And according to step 504, Then, according to step S506, we obtain And the cost function is calculated according to step 508. If α = 0.5 and n = 2, then C ≒ 90. Since C is less than TH1, the exposure parameters remain unchanged.
[0104] Figure 6A , Figure 6B A flowchart of an exposure adjustment method for a multi-lens camera 11 is shown to illustrate the method according to a fourth embodiment of the present invention. Reference will be made below. Figure 1 , 5A 5B and Figure 6A , 6B The method for adjusting the exposure is described in detail through four image signal processors 130 (or four exposure statistics units 131) and an exposure compensation engine 150, where N=3.
[0105] Step 602: Using exposure compensation engine 150, determine whether the image of lens-k meets one of the following two conditions: (a) the right overlap region JR in the image of lens-k k The left overlap area JL of the image from the right-hand camera (k+1)mod4 Brighter, and simultaneously overlapping to the left JL k JR, compared to the right overlap area of the left-hand lens image (k+3)mod4 Brighter; or (b) the right overlap region JR in the lens-k image. k The left overlap area JL of the image from the right-hand camera (k+1)mod4 Darker, and simultaneously overlapping to the left JL k JR, compared to the right overlap area of the left-hand lens image (k+3)mod4 Darker. If either of the above two conditions is met, then... If the process proceeds to step S606, then the process proceeds to step S604.
[0106] Step S604: Using exposure compensation engine 150, based on Calculate Δ k .
[0107] Step S606: Using exposure compensation engine 150, based on the current exposure step size Dynamic adjustment amount and auxiliary brightness difference Perform parameter decision operations on the lens-k image to determine the next exposure step. and one parameter adjustment amount Δ k Similar to step S415, this step S606 can be represented as follows: Furthermore, func() represents the parameter decision operation performed by the exposure compensation engine 150 (see details). Figure 3B As mentioned earlier, func() can be viewed as a function of... Figure 6A , 6B The subroutine called or loaded by the main process.
[0108] Step S608: Using exposure compensation engine 150, based on and Calculate Δ k In one embodiment,
[0109]
[0110] Step S610: Using exposure compensation engine 150, according to Δ k The exposure parameters of lens 11k are updated, and the updated exposure parameters of the next frame (of the isometric panoramic image) are transmitted to multi-lens camera 11 for lens exposure adjustment. For example, the exposure time of lens 11k can be updated and transmitted to camera 11 according to the following formula: T′ k =T k +Δ k .
[0111] Example 6 (corresponding) Figure 6A , 6B The fourth embodiment and Figure 5A or Figure 5B (Isometric rectangular panoramic image):
[0112] Assume Tg = 55, TH1 = 60, TH2 = 4, TH3 = 96. (Step S506), and assuming that the following four average brightness values were measured in step S302: Y0 = 90, Y1 = 50, Y2 = 95, Y3 = 95, and the average brightness values of eight overlapping regions JR0~JR3 and JL0~JL3 were measured in step S502: L0 = 80, L1 = 50, L2 = 95, L3 = 80, R0 = 90, R1 = 40, R2 = 80, R3 = 80. Next, according to step S504, the following is obtained... Then, according to step S506, we obtain And the cost function is calculated according to step 508.
[0113] (If α = 1). Because the cost function C is greater than TH1, it is necessary to base it on... To adjust the exposure parameters. According to step S415, perform the parameter decision operation: Where k = 0, 1, 2, 3, to obtain and In step S602, only the image of lens-1 satisfies the condition. Therefore, based on the conditions, the parameter decision operation is performed in step S606: Where k = 1, so as to obtain and According to step S608, we obtain On the other hand, because the images from the other three shots did not meet the requirements. The conditions are met, so step S604 is performed. According to step S604, we can obtain... Where k = 0, 2, 3, so and Finally, according to step S312, the exposure compensation engine 150 updates and transmits the updated exposure times of lenses 110 to 113 to camera 11: T′0=T0-5, T′1=T1+2, T′2=T2-5, T′3=T3-5.
[0114] Figures 7A-7B A flowchart of an exposure adjustment method for a multi-lens camera 11 is shown to illustrate the fifth embodiment of the present invention. Reference will be made below. Figure 1 and Figure 7A , Figure 7B The method for adjusting the exposure is described in detail through four image signal processors 130 (or four exposure statistics units 131) and an exposure compensation engine 150, where N=3.
[0115] Step S702: Using exposure compensation engine 150, based on the brightness difference of each lens image. and Calculate a geometric mean, that is... Where k = 0, 1, 2, 3.
[0116] Step S704: Using exposure compensation engine 150, based on each lens image... and Calculate the auxiliary brightness difference In one embodiment, In another embodiment, Where k = 0, 1, 2, 3.
[0117] Step S706: Using exposure compensation engine 150, based on and The cost function C is calculated, where k = 0, 1, 2, 3. In one embodiment,
[0118] Step S708: Using exposure compensation engine 150, based on the current exposure step size Dynamic adjustment amount and geometric mean Perform parameter decision operations on the lens-k image to determine the next exposure step. and one parameter adjustment amount For simplicity, step S708 can be represented as follows: Where k = 0, 1, 2, 3, and func() represents the parameter decision operation performed by the exposure compensation engine 150 (see details). Figure 3B As mentioned earlier, func() can be viewed as a function of... Figures 7A-7B The subroutine called or loaded by the main process.
[0119] Step S710: Using exposure compensation engine 150, based on the current exposure step size Dynamic adjustment amount and auxiliary brightness difference Perform parameter decision operations on the image from lens -k to determine the next exposure step. and one parameter adjustment amount For simplicity, step S710 can be represented as follows: Where k = 0, 1, 2, 3, and func() represents the parameter decision operation performed by the exposure compensation engine 150 (see details). Figure 3B ).
[0120] Step S712: Using exposure compensation engine 150, based on and Calculate Δ k In one embodiment,
[0121] Step S714: Using exposure compensation engine 150, based on and Calculate Δ k In one embodiment,
[0122] Example 7 (corresponding) Figure 7A , Figure 7B The fifth embodiment and Figure 5A or Figure 5B (Isometric rectangular panoramic image):
[0123] Assume Tg = 55, TH1 = 60, TH2 = 4, TH3 = 96. and (Step S704), and assuming that the following four average brightness values were measured in step S302: Y0 = 90, Y1 = 50, Y2 = 95, Y3 = 95, and the average brightness values of eight overlapping regions JR0~JR3 and JL0~JL3 were measured in step S702: L0 = 80, L1 = 50, L2 = 95, L3 = 80, R0 = 90, R1 = 40, R2 = 80, R3 = 80. Based on step S304, the following values are obtained: and And according to step S504, Then, the geometric mean is obtained according to step S702. And the auxiliary brightness difference is obtained according to step S704. Furthermore, the cost function is calculated according to step 706.
[0124] (If α = 1; β = 0.5).
[0125] Because the cost function C is greater than TH1, the exposure parameters need to be based on... and / or Adjustment. Perform parameter decision operation according to step S415. To obtain Perform parameter decision operation according to step S708. To obtain In step 602, only the image of lens-1 satisfies the condition. The conditions are met. Therefore, the parameter decision operation is performed according to step S710. Where k = 1, so as to obtain and And according to step 712, On the other hand, because the images from the other three lenses did not meet the requirements. The conditions are met, so step S714 is performed. Based on step S714, the following is obtained: Where k = 0, 2, 3, so Finally, according to step S312, the exposure compensation engine 150 updates the exposure times of lenses 110 to 113 and transmits the updated exposure times to the camera 11 to capture the next frame: T′0=T0+3, T′1=T1+6, T′2=T2+3, T′3=T3+3.
[0126] Although Figures 3A-3B , Figures 4A-4B , Figures 5B-5C , Figures 6A-6B and Figures 7A-7B The functional blocks shown are presented as discrete blocks, but depending on the desired implementation, these functional blocks can be divided into more blocks, merged into fewer blocks, or directly deleted, and their execution order can also be... Figures 3A-3B The order shown in 4A-4B, 5B-5C, 6A-6B, and 7A-7B is different.
[0127] Figure 8Displaying a 270-degree stitched wide-angle image, the image is sourced from Figure 2B A three-lens camera 11B, mounted on three working surfaces of a cubic structure 1B, captures three images and generates a view after equidistant rectangular projection. The three-lens camera 11B can capture a field of view encompassing a 270-degree horizontal field of view and a 180-degree vertical field of view. (Reference) Figure 8 The image contains three lens images and there are only four overlapping areas: JR0~JR1 and JL1~JL2.
[0128] Example 8 (corresponding) Figure 5C , 5D The third embodiment and Figure 8 (Stitched wide-angle image):
[0129] Assume Tg = 55, TH1 = 95, TH2 = 4, TH3 = 96. (Step S506), and assume that the following three average brightness values were measured in step S302: Y0 = 90, Y1 = 50, Y2 = 95, and that the average brightness values of the four overlapping regions JR0~JR1 and JL1~JL2 were measured in step S502: L1 = 50, L2 = 95, R0 = 90, R1 = 40. Furthermore, according to step S304... It should be noted that, Figure 8 There is no overlap between the left side of the image from lens-0 and the right side of the image from lens-2, therefore... Set it to the first preset value (e.g., 0), and... Set it to the second preset value (e.g., 0). Then, according to step S504, obtain... According to step S506, And the cost function is calculated according to step 508. (If α = 0.5 and n = 2). Because the cost function C is greater than TH1, it is necessary to base it on... and To adjust the exposure parameters. According to step S310... According to step S510, Next, let's assume We can obtain Δ0 = -2, Δ1 = 2, Δ2 = -2. Finally, according to step S312, the exposure compensation engine 150 updates the exposure times of lenses 110 to 113 and transmits the updated exposure times to the camera 11 to capture the next frame: T′0 = T0 - 2, T′1 = T1 + 2, T′2 = T2 - 2.
[0130] Example 9 (corresponding) Figure 7A , Figure 7B The third embodiment and Figure 8 (Stitched wide-angle image): Assuming Tg = 55, TH1 = 60, TH2 = 4, TH3 = 96, (Step S702) and (Step S704), and assuming that the following three average luminances were measured in step S302: Y0 = 90, Y1 = 50, Y2 = 95, and the following average luminances JR0~JR1 and JL1~JL2 of the four overlapping regions were measured in step S502: L1 = 50, L2 = 95, R0 = 90, R1 = 40. Then, according to step S304, the following is obtained... It should be noted that, Figure 8 There is no overlap between the left side of the image from lens-0 and the right side of the image from lens-2, therefore... Set it to the first preset value (e.g., 0), and... Set it to the second preset value (e.g., 0). Then, according to step S504, obtain... Then, the geometric mean is obtained according to step S702. And the auxiliary brightness difference is obtained according to step S704. Furthermore, the cost function is calculated according to step S706.
[0131] (If α = β = 0.5).
[0132] Because the cost function C is greater than TH1, it needs to be based on and Included or not included To adjust the exposure parameters. Perform the parameter decision operation according to step S415. To obtain And perform parameter decision operation according to step S708. To obtain In step 602, only the image of lens-1 satisfies the condition. Therefore, according to the conditions, the parameter decision operation is performed according to step S710. Where k = 1, so as to obtain and According to step 712, On the other hand, because the images from the other two shots did not meet the requirements. The conditions are met, so step S714 is performed. Based on step S714, the following is obtained: Where k = 0, 2, therefore Finally, according to step S312, the exposure compensation engine 150 updates the exposure times of lenses 110 to 113 and transmits the updated exposure times to the camera 11 to capture the next frame: T′0=T0-7, T′1=T1+6, T′2=T2-7.
[0133] Although the exposure parameters in the above examples and embodiments are illustrated using exposure time as an example, it should be understood that the exposure parameters are not limited thereto, and also include aperture value, analog gain value, and digital gain value. In other words, the present invention... Figures 3A-3B , Figures 4A-4B , Figures 5C-5D , Figures 6A-6B and Figures 7A-7B The exposure adjustment method shown also applies to adjusting aperture, analog gain, and digital gain. Similar to exposure time, when the final parameter adjustment Δ (as in steps S310 and S415) is obtained... For example, Δ in steps S512, S604, S608, S712 and S714 k When T' increases, both the analog gain and digital gain values will also increase, for example: T' k =T k +Δ; Unlike exposure time, when this final parameter adjustment Δ increases, the aperture value decreases, for example: A′ k =A k -Δ (by reversing the sign of the parameter adjustment Δ), where A k This represents the current aperture value of the lens at 11k, A′. k This indicates the aperture value for the next frame when the lens is at 11k.
[0134] Example 10 (corresponding to Example 1 and this exposure parameter is equal to the aperture value):
[0135] According to step S310 of Example 1 above, we obtain as well as Finally, according to step S312, the exposure compensation engine 150 updates and transmits the following four aperture values of the lenses 110 to 114 to the camera 11 to capture the next frame: A′0=A0-6, A′1=A1+5, A′2=A2-6, A′3=A3-4.
[0136] In the first and second embodiments, the parameter adjustment amount Δ for each lens image k Difference from main brightness (or average brightness Y) k Related to this. In the third embodiment, the parameter adjustment amount Δ for each lens image is... k Difference from main brightness and auxiliary brightness difference Related. In the fourth embodiment, the parameter adjustment amount Δ for each lens image is... k Difference from main brightness Related; however, the parameter adjustment amount Δ k Difference with auxiliary brightness The relationship between them depends on Is it greater than 0? In the fifth embodiment, for each lens image, the parameter adjustment amount Δ k Difference from main brightness and geometric mean Related, however, its parameter adjustment amount Δ k Difference with auxiliary brightness The relationship between them depends on Is it greater than 0?
[0137] The exposure statistics units 131 and exposure compensation engine 150 of the present invention can be implemented in hardware, software, or a combination of hardware and software (or firmware). A simple example of a pure hardware solution is a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In one embodiment, the exposure compensation engine 150 is implemented by a first general-purpose processor and a first program memory, and each exposure statistics unit 131 is implemented by a second general-purpose processor and a second program memory, wherein the first program memory stores a first processor executable program, and the second program memory stores a second processor executable program. When the first general-purpose processor executes the first processor executable program, the first general-purpose processor can be configured to operate as the exposure compensation engine 150; when the second general-purpose processor executes the second processor executable program, the second general-purpose processor can be configured to operate as the exposure statistics unit 131.
[0138] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the patent application of the present invention; any equivalent changes or modifications made without departing from the spirit disclosed in the present invention shall be included in the claims.
Claims
1. A device for synchronously adjusting the exposure of a multi-lens camera, characterized in that, Include: Multiple image signal processors, each image signal processor including an exposure statistics circuit and coupled to the multi-lens camera, wherein each exposure statistics circuit is configured to measure the brightness information of each lens image in multiple lens images captured by the multi-lens camera; as well as An exposure compensation engine, coupled to the multi-lens camera and the plurality of image signal processors, is configured to perform a set of operations, the set of operations including: (1) For each of the multiple lens images, calculate a major brightness difference between the brightness information and a target brightness; (2) Calculate a cost function based on the main brightness difference of the lens image; and (3) If the cost function is greater than a first critical value Then for each of at least one of the multiple shot images, Based on the main brightness difference, a first parameter adjustment amount is obtained, and Based on the adjustment amount of the first parameter, an exposure parameter is obtained and transmitted to the multi-lens camera for lens exposure adjustment.
2. The apparatus as claimed in claim 1, characterized in that, The multi-lens camera is configured to capture a field of view that covers an X-degree horizontal field of view and a Y-degree vertical field of view, where X≦360 and Y≦180.
3. The apparatus as described in claim 1, characterized in that, The multi-lens camera is one of a wide-angle camera and a panoramic camera.
4. The apparatus as claimed in claim 1, characterized in that, This group of operations further includes: (4) If the cost function is less than or equal to the first critical value, then the exposure parameters remain unchanged.
5. The apparatus as claimed in claim 1, characterized in that, The operation of obtaining the adjustment amount of the first parameter includes: Based on the main brightness difference D of the kth image in the lens image, the current exposure step size S and the dynamic adjustment amount ε, the next exposure step size S' of the kth image and the first parameter adjustment amount Δ are obtained. Where k≧0, and the dynamic adjustment amount ε is used to adjust the current exposure step size S to obtain the next exposure step size S'.
6. The apparatus as claimed in claim 5, characterized in that, The operation of obtaining the next exposure step size S' and the first parameter adjustment amount Δ of the k-th image further includes: If D>0 and the absolute value of D (|D|) is less than a second critical value TH2, then the next exposure step size S' is calculated according to the following formula: S'=S-ε, and the first parameter adjustment amount Δ is set to be equal to the next exposure step size S'. If D>0 and |D| is greater than a third critical value TH3, then the next exposure step size S' is calculated according to the following formula: S'=S+ε, and the first parameter adjustment amount Δ is set to be equal to the next exposure step size S'; If D>0 and TH2≦|D|≦TH3, then the next exposure step size S' is set to be equal to the current exposure step size S, and the first parameter adjustment amount Δ is set to be equal to the next exposure step size S'. If D < 0 and |D| is less than the second critical value TH2, then the next exposure step size S' is calculated according to the following formula: S' = S - ε, and the first parameter adjustment amount Δ is set to be equal to (-S'); If D < 0 and |D| is greater than the third critical value TH3, then the next exposure step size S' is calculated according to the following formula: S' = S + ε, and the first parameter adjustment Δ is set to be equal to (-S'); and If D < 0 and TH2 ≦ |D| ≦ TH3, then the next exposure step size S' is set to be equal to the current exposure step size S, and the first parameter adjustment amount Δ is set to be equal to (-S').
7. The apparatus as claimed in claim 1, characterized in that, Operation (3) further includes: If the cost function is greater than the first critical value and the absolute value of the principal brightness difference |D| of the k-th image in the lens image is equal to the maximum value among the absolute values of brightness differences in the lens images. Then, based on the main brightness difference D of the k-th image, the adjustment amount of the first parameter is obtained; and The exposure parameters are obtained and transmitted to the multi-lens camera based on the adjustment amount of the first parameter, wherein the cost function is related to the maximum value and k ≧ 0.
8. The apparatus as claimed in claim 1, characterized in that, Each of the lens images includes at least one of a left overlapping region and a right overlapping region located at the left and right ends of the lens image, respectively, and prior to operation (2), the group of operations further includes: (a) If a left-side lens image exists to the left of the k-th image of the lens image, then calculate the left brightness difference between the average brightness of the left overlapping region of the k-th image and the average brightness of the right overlapping region of the left-side lens image. Where k≧0; (b) If a right-side shot image exists to the right of the k-th image, then calculate the right brightness difference between the average brightness of the right overlapping region of the k-th image and the average brightness of the left overlapping region of the right-side shot image. as well as (c) Based on the aforementioned left brightness difference With the aforementioned right brightness difference Calculate the auxiliary brightness difference of the k-th image; The brightness information of each lens image includes the average brightness of the entire lens image, the average brightness of the left overlapping region, and the average brightness of the right overlapping region.
9. The apparatus as claimed in claim 8, characterized in that, Prior to operation (c), this group of operations further includes: If there are no left-side camera images to the left of the k-th image, then the left brightness difference... Set it to equal a first preset value; as well as If there is no right-side camera image to the right of the k-th image, then the right brightness difference will be... Set to a second preset value.
10. The apparatus as claimed in claim 8, characterized in that, Operation (2) includes: The cost function is calculated based on the primary brightness difference and the secondary brightness difference of the lens image.
11. The apparatus as claimed in claim 10, characterized in that, Operation (3) includes: If the cost function is greater than the first critical value. For the k-th image, The adjustment amount of the first parameter is obtained based on the main brightness difference. The adjustment amount of the second parameter is obtained based on the inherent auxiliary brightness difference; and The exposure parameters are obtained and transmitted to the multi-lens camera based on the first parameter adjustment amount and the second parameter adjustment amount.
12. The apparatus as claimed in claim 10, characterized in that, Operation (3) includes: If the cost function C is greater than the first critical value TH1, then the first parameter adjustment amount is obtained based on the main brightness difference of the k-th image; If C>TH1 and Then, based on the auxiliary brightness difference of the kth image, the second parameter adjustment amount is obtained, and based on the first parameter adjustment amount and the second parameter adjustment amount, the exposure parameter is obtained and transmitted to the multi-lens camera. as well as If C>TH1 and The exposure parameters are then obtained and transmitted to the multi-lens camera based on the adjustment amount of the first parameter.
13. The apparatus as claimed in claim 8, characterized in that, Prior to operation (c), this group of operations further includes: Based on the left brightness difference of the k-th image With the aforementioned right brightness difference Calculate a geometric mean.
14. The apparatus as claimed in claim 13, characterized in that, Operation (2) includes: The cost function is calculated based on the primary brightness difference, the geometric mean, and the secondary brightness difference of the lens image.
15. The apparatus as claimed in claim 14, characterized in that, Operation (3) includes: If the cost function C is greater than the first critical value TH1 Then, based on the main brightness difference of the k-th image, the adjustment amount of the first parameter is obtained, and A second parameter adjustment amount is obtained based on the geometric mean of the k-th image; If C>TH1 and Based on the auxiliary brightness difference of the k-th image, a third parameter adjustment amount is obtained, and The exposure parameters are obtained and transmitted to the multi-lens camera based on the first parameter adjustment amount, the second parameter adjustment amount, and the third parameter adjustment amount. as well as If C>TH1 and Then, based on the first parameter adjustment amount and the second parameter adjustment amount of the k-th image, the exposure parameters are obtained and transmitted to the multi-lens camera.
16. The apparatus as claimed in claim 1, characterized in that, The exposure parameter is at least one of an aperture value, an exposure time, an analog gain value, and a digital gain value.
17. The apparatus as claimed in claim 16, characterized in that, If the exposure parameter is the aperture value, the aperture value will decrease when the adjustment amount of the first parameter increases. If the exposure parameter is one of the exposure time, the analog gain value, and the digital gain value, the exposure time, the analog gain value, and the digital gain value will also increase when the adjustment amount of the first parameter increases.
18. A method for synchronously adjusting the exposure of a multi-lens camera, characterized in that, Include: (a) Using multiple image signal processors, the brightness information of each lens image in multiple lens images is measured, wherein the image signal processors are coupled to the multi-lens camera; (b) For each shot image, calculate a principal brightness difference between the brightness information and the brightness of a target; (c) Calculate a cost function based on the principal brightness difference of the lens image; and (d) If the cost function is greater than a first critical value Then, for at least one of the multiple shot images, each image is obtained. Based on the main brightness difference, a first parameter adjustment amount is obtained, and Based on the adjustment amount of the first parameter, an exposure parameter is obtained and transmitted to the multi-lens camera for lens exposure adjustment.
19. The method as described in claim 18, characterized in that, It also includes: The multi-lens camera captures a field of view encompassing an X-degree horizontal field of view and a Y-degree vertical field of view, where X ≦ 360 and Y ≦ 180; and The image signal processor acquires the lens image originating from the multi-lens camera.
20. The method as described in claim 18, characterized in that, It also includes: If the cost function is less than or equal to the first critical value, then the exposure parameters remain unchanged.
21. The method as described in claim 18, characterized in that, The step of obtaining the first parameter adjustment amount includes: Based on the main brightness difference D of the kth image in the lens image, a current exposure step size S, and a dynamic adjustment amount ε, the next exposure step size S' of the kth image and the first parameter adjustment amount Δ are obtained, where k≧0, and the dynamic adjustment amount ε is used to adjust the current exposure step size S to obtain the next exposure step size S'.
22. The method as described in claim 21, characterized in that, The steps of obtaining the next exposure step size S' of the k-th image and the first parameter adjustment amount Δ include: If D>0 and the absolute value of D (|D|) is less than a second critical value TH2, then the next exposure step size S' is calculated according to the following formula: S'=S-ε, and the first parameter adjustment amount Δ is set to be equal to S'; If D>0 and |D| is greater than a third critical value TH3, then the next exposure step size S' is calculated according to the following formula: S'=S+ε, and the first parameter adjustment amount Δ is set to be equal to S'; If D>0 and TH2≦|D|≦TH3, then set the next exposure step size S' to the current exposure step size S, and set the first parameter adjustment amount Δ to be equal to S'; If D < 0 and |D| is less than the second critical value TH2, then the next exposure step size S' is calculated according to the following formula: S' = S - ε, and the adjustment amount Δ of the first parameter is set to be equal to (-S'); If D < 0 and |D| is greater than the third critical value TH3, then the next exposure step size S' is calculated according to the following formula: S' = S + ε, and the first parameter adjustment Δ is set to be equal to (-S'); and If D < 0 and TH2 ≦ |D| ≦ TH3, then set the next exposure step size S' to the current exposure step size S, and set the first parameter adjustment amount Δ to be equal to (-S').
23. The method as described in claim 18, characterized in that, Step (d) includes: If the cost function is greater than the first critical value and the absolute value of the principal brightness difference |D| of the k-th image in the lens image is equal to the maximum value among the absolute values of brightness differences in the lens images. Then, based on the main brightness difference D of the k-th image, the adjustment amount of the first parameter is obtained; and The exposure parameters are obtained and transmitted to the multi-lens camera based on the adjustment amount of the first parameter, wherein the cost function is related to the maximum value and k ≧ 0.
24. The method as described in claim 18, characterized in that, Each of the lens images includes at least one of a left overlapping region and a right overlapping region located at the left and right ends of the lens image, respectively, and prior to step (c), the method further includes: (1) If a left-side lens image exists to the left of the k-th image of the lens image, then calculate the left brightness difference between the average brightness of the left overlapping region of the k-th image and the average brightness of the right overlapping region of the left-side lens image. Where k≧0; (2) If there is a right-side camera image to the right of the k-th image, then calculate the right brightness difference between the average brightness of the right overlapping region of the k-th image and the average brightness of the left overlapping region of the right-side camera image. as well as (3) Based on the aforementioned left brightness difference The difference between the right average brightness Calculate the auxiliary brightness difference of the k-th image; The brightness information of each lens image includes the average brightness of the entire lens image, the average brightness of the left overlapping region, and the average brightness of the right overlapping region.
25. The method as described in claim 24, characterized in that, The method further includes, prior to step (3): If there are no left-side camera images to the left of the k-th image, then the left average brightness difference will be... Set to a first preset value; as well as If there are no right-side camera images to the right of the k-th image, then the right average brightness difference will be... Set it to a second preset value.
26. The method as described in claim 24, characterized in that, Step (c) includes: The cost function is calculated based on the primary brightness difference and the secondary brightness difference of the multiple lens images.
27. The method as described in claim 26, characterized in that, Step (d) includes: If the cost function is greater than the first critical value. For the k-th image, The adjustment amount of the first parameter is obtained based on the main brightness difference. The adjustment amount of the second parameter is obtained based on the inherent auxiliary brightness difference; and The exposure parameters are obtained and transmitted to the multi-lens camera based on the first parameter adjustment amount and the second parameter adjustment amount.
28. The method as described in claim 26, characterized in that, Step (d) includes: If the cost function C is greater than the first critical value TH1, then the first parameter adjustment amount is obtained based on the main brightness difference of the kth image; If C>TH1 and Then, based on the auxiliary brightness difference of the kth image, the second parameter adjustment amount is obtained, and based on the first parameter adjustment amount and the second parameter adjustment amount, the exposure parameters are obtained and transmitted to the multi-lens camera. as well as If C>TH1 and Based on the first parameter adjustment amount of the k-th image, the exposure parameters are obtained and transmitted to the multi-lens camera.
29. The method as described in claim 24, characterized in that, Prior to step (3), the method further comprises: Based on the left brightness difference of the k-th image Difference in brightness with the right Calculate a geometric mean.
30. The method as described in claim 29, characterized in that, Step (c) includes: The cost function is calculated based on the primary brightness difference, the geometric mean, and the secondary brightness difference of the lens image.
31. The method as described in claim 30, characterized in that, Step (d) includes: If the cost function C is greater than the first critical value TH1 Then, based on the main brightness difference of the k-th image, the adjustment amount of the first parameter is obtained, and A second parameter adjustment amount is obtained based on the geometric mean of the k-th image; If C>TH1 and Then, based on the auxiliary brightness difference of the k-th image, a third parameter adjustment amount is obtained, and The exposure parameters are obtained and transmitted to the multi-lens camera based on the first parameter adjustment amount, the second parameter adjustment amount, and the third parameter adjustment amount. as well as If C>TH1 and Then, based on the first parameter adjustment amount and the second parameter adjustment amount of the k-th image, the exposure parameters are obtained and transmitted to the multi-lens camera.
32. The method as described in claim 18, characterized in that, The multi-lens camera is one of a wide-angle camera and a panoramic camera.
33. The method as described in claim 18, characterized in that, The exposure parameter is at least one of an aperture value, an exposure time, an analog gain value, and a digital gain value.
34. The method as described in claim 33, characterized in that, If the exposure parameter is the aperture value, the aperture value will decrease when the adjustment amount of the first parameter increases. If the exposure parameter is one of the exposure time, the analog gain value, and the digital gain value, the exposure time, the analog gain value, and the digital gain value will also increase when the adjustment amount of the first parameter increases.
