Brightness consistency control method of multi-objective zoom system and computer storage medium

Abstract

The application relates to the technical field of image signal processing, in particular to a brightness consistency control method of a multi-lens zoom system and a computer storage medium. The method comprises the following steps: acquiring an electronic zoom ratio, determining a clipping region in an original image according to the electronic zoom ratio, and associating a brightness statistical region of an automatic exposure algorithm to the clipping region. Brightness statistics are performed on the brightness statistical region to obtain exposure parameters of a current active sensor. According to a pre-calibrated exposure parameter conversion relationship between different sensors, the exposure parameters of the current active sensor are mapped into target exposure parameters of a target sensor. The target exposure parameters are written into a register of the target sensor. When a sensor switching instruction is received, the target sensor is connected to an image signal processor, and the target sensor outputs an image by using the target exposure parameters pre-stored in the register. The application can realize seamless zooming with consistent brightness, thereby improving the visual continuity and user experience of the multi-lens zoom system.

Description

Technical Field

[0001] This application relates to the field of image signal processing technology, and in particular to a brightness consistency control method for a multi-view zoom system and a computer storage medium. Background Technology

[0002] With the increasing demands for image quality from mobile terminals and security monitoring equipment, multi-camera architecture has become a common solution for achieving wide focal length coverage. Currently, traditional technical solutions generally use a hardware architecture of multiple sensors with different focal lengths combined with a single ISP (Image Signal Processor), achieving seamless zoom coverage from wide-angle to telephoto through the combination of wide-angle and telephoto lenses.

[0003] In terms of automatic exposure control, traditional technologies generally suffer from the following drawbacks: During electronic zoom, the AE (Auto Exposure) statistical window is fixed to a fixed area of ​​the original full-frame image of the sensor and does not dynamically adjust with changes in the cropped field of view, resulting in a mismatch between the statistical brightness and the user's actual observation area. When switching sensors, the newly connected sensor lacks effective prior exposure information for the current environment and requires a closed-loop convergence process of several frames to several seconds from the default parameters to reach stable brightness. In addition, due to limitations in register update mechanisms and image processing pipeline frame buffer latency, parameters calculated and issued at the moment of switching are unlikely to take effect in the first frame, causing brightness jumps and flickering, affecting visual continuity. Summary of the Invention

[0004] This application provides a brightness consistency control method and computer storage medium for a multi-view zoom system, which can eliminate brightness statistical deviation and switching convergence delay in the electronic zoom and optical zoom processes, and achieve seamless zoom with consistent brightness, thereby improving the visual continuity and user experience of the multi-view zoom system.

[0005] Firstly, this application provides a brightness consistency control method for a multi-view zoom system, comprising: acquiring an electronic zoom ratio; determining a cropped region in the original image based on the electronic zoom ratio; associating a brightness statistics region of an automatic exposure algorithm with the cropped region; performing brightness statistics on the brightness statistics region using the automatic exposure algorithm to obtain the exposure parameters of the currently active sensor; mapping the exposure parameters of the currently active sensor to the target exposure parameters of the target sensor according to a pre-calibrated conversion relationship between different sensors; writing the target exposure parameters into the register of the target sensor; and when a sensor switching command is received, connecting the target sensor to an image signal processor, and outputting an image using the pre-stored target exposure parameters in the register.

[0006] In some implementations, associating the brightness statistics region of the automatic exposure algorithm with the cropping region includes: calculating the coordinate information and window size of the cropping region based on the electronic zoom ratio; and proportionally mapping the coordinate information and window size of the brightness statistics region according to the size ratio between the cropping region and the original image.

[0007] In some implementations, the exposure parameter conversion relationship is obtained based on the optical parameters of each sensor, including lens aperture value and transmittance. Alternatively, it is obtained by calibrating different sensors to the same uniform light source and adjusting the exposure parameters to make the output brightness consistent; the exposure parameters include at least one of exposure time, analog gain, and digital gain.

[0008] In some implementations, mapping the exposure parameters of the currently active sensor to the target exposure parameters of the target sensor includes: obtaining the current exposure parameters of the currently active sensor; and multiplying the current exposure parameters by a conversion ratio corresponding to the exposure parameter conversion relationship to obtain the target exposure parameters.

[0009] In some implementations, before writing the target exposure parameters into the target sensor's register, the method further includes: detecting whether a moving target exists in the current scene using a motion detection module. If a moving target is detected, the relaxation range of the exposure time is limited, and the target exposure parameters are corrected using gain compensation. If no moving target is detected, the target exposure parameters are corrected using exposure time compensation.

[0010] In some implementations, detecting the presence of a moving target in the current scene includes: comparing the current frame with the previous frame using a motion detection module, and outputting a determination result indicating whether a moving target exists based on the comparison result. Limiting the relaxation range of exposure time includes: limiting the relaxation range of exposure time to a preset proportion of the maximum exposure time.

[0011] In some implementations, exposure time compensation is used to correct the target exposure parameters, including temporarily relaxing the upper limit of exposure time from a first limit value defaulted to by the image signal processor algorithm to a second limit value supported by the sensor hardware, wherein the second limit value is greater than the first limit value.

[0012] In some implementations, connecting the target sensor to the image signal processor includes: when the hardware architecture is such that each sensor has an independent MIPI channel and each channel is connected to a different video interface, performing a rebinding operation between the video interface and the image signal processor, binding the video interface corresponding to the target sensor to the image signal processor. When the hardware architecture is such that multiple sensors share a single MIPI bus via a MIPI switch, performing a MIPI bus switching operation, connecting the target sensor to the image signal processor via the MIPI bus.

[0013] Secondly, this application provides a brightness consistency control device for a multi-view zoom system, comprising: The acquisition module is used to acquire the electronic zoom ratio, determine the cropped area in the original image based on the electronic zoom ratio, and associate the brightness statistics area of ​​the automatic exposure algorithm with the cropped area.

[0014] The processing module performs brightness statistics on the brightness statistical area using an automatic exposure algorithm to obtain the exposure parameters of the currently active sensor. Based on a pre-calibrated conversion relationship between different sensors' exposure parameters, it maps the exposure parameters of the currently active sensor to the target exposure parameters of the target sensor. The target exposure parameters are then written into the target sensor's register. When a sensor switching command is received, the target sensor is connected to the image signal processor, and the target sensor outputs an image using the pre-stored target exposure parameters in its register.

[0015] Thirdly, this application provides a chip for performing the method described in the first aspect above.

[0016] Fourthly, this application provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the method as described in the first aspect above. Alternatively, the electronic device may include a chip as described in the third aspect.

[0017] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0018] In a sixth aspect, this application provides a computer program product that stores a computer program, which, when executed by a processor, implements the method described in the first aspect above.

[0019] In the technical solution provided in this application, the system can dynamically associate the AE statistical window with the cropping area according to the electronic zoom magnification, and map the exposure parameters of the currently active sensor based on the pre-calibrated conversion relationship and pre-write them into the register of the target sensor. When switching, the system outputs the pre-stored parameters directly, which enables the AE algorithm to always evaluate the brightness based on the user's actual observation of the picture, eliminates the interference of light outside the field of view on the exposure decision, and enables the target sensor to have accurate exposure parameters in the first frame after accessing the ISP. This completely eliminates the closed-loop convergence process of several frames to several seconds in the traditional solution, thereby achieving accurate metering and seamless switching from accurate first frame throughout the entire process of electronic zoom and optical zoom, fundamentally eliminating brightness jumps and flicker, and improving the brightness consistency and visual experience of the multi-lens zoom system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a brightness consistency switching process for a brightness consistency control method for a multi-view zoom system provided in this application embodiment; Figure 2 This is a schematic diagram of a multi-view zoom system pipeline under a VIF rebinding architecture, which is a brightness consistency control method for a multi-view zoom system provided in this application embodiment. Figure 3 This is a schematic diagram of a multi-view zoom system pipeline under the MIPI switching architecture of a brightness consistency control method for a multi-view zoom system provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a brightness consistency control device for a multi-view zoom system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] As security monitoring and mobile shooting equipment increasingly demand higher zoom ranges, a single camera can no longer simultaneously meet the requirements of a wide field of view and high-resolution long-distance imaging. The video surveillance and multi-camera imaging technology fields typically employ a hardware architecture that combines multiple sensors with different focal lengths with a single ISP, using a combination of wide-angle and telephoto lenses to achieve wide-magnification zoom coverage from wide-angle to telephoto.

[0029] In traditional technical architectures, zoom functionality can be achieved in two ways. The first is electronic zoom, which maintains the sensor's full-frame output while the image scaling and cropping module in the ISP (Internet Service Provider) selects the ROI (Region of Interest) at the software level for cropping and magnification. In this case, the sensor hardware itself does not physically switch. The second is optical zoom (i.e., lens switching), where the system uses a multiplexer or software commands to disconnect the data channel between the current sensor and the ISP, and connects a sensor with a different focal length to the same ISP, thus achieving a physical lens switching.

[0030] In terms of AE control, existing systems typically employ the following operating modes: Fixed statistical area: Regardless of the change in electronic zoom magnification, the brightness statistical window of the AE algorithm is fixed in a certain central area of ​​the original full-frame image of the sensor, or the full-frame image is statistically analyzed with fixed weights for each area. The position and size of the statistical window are not dynamically adjusted with the change of the actual output field of view corresponding to the electronic zoom.

[0031] Independent convergence mechanism. When the system performs a physical sensor switch, the newly connected sensor lacks initial exposure parameters that match the current ambient lighting conditions. Its AE algorithm usually starts from the default parameters or historical parameters saved before the last power failure. Based on the brightness statistics of the first frame or several frames of the image from the new sensor, it gradually adjusts the exposure time and gain through closed-loop feedback. After multiple frame iterations, the brightness can converge to the target value. This process is called the AE convergence process.

[0032] The aforementioned traditional technical solutions have the following shortcomings in practical applications: The problem of brightness assessment distortion during electronic zoom.

[0033] Because electronic zoom only crops and enlarges the original image, the user actually observes only a local area of ​​the original full-frame image. However, the AE statistics window still covers the original full-frame or fixed center area, resulting in a discrepancy between the statistics area and the user's actual field of view.

[0034] For example, when a user zooms in on details in a dark area of ​​an image using electronic zoom, if there is a strong light source in the outer area of ​​the original full-frame image, the AE algorithm will determine that the current image brightness is sufficient because it detects the strong light information, and thus reduce the exposure, resulting in severe underexposure of the zoomed-in area that the user is actually focusing on, or even causing unexpected flickering of the image brightness.

[0035] Secondly, there is the problem of sudden brightness changes during sensor switching, i.e., the cold start problem.

[0036] Lenses with different focal lengths have different aperture values ​​(F-numbers) and lens transmittance (T-numbers). For example, an 8mm lens has an aperture of F1.6, while a 25mm lens has an aperture of F2.4, resulting in a significant difference in the amount of light they can capture. At the moment of sensor switching, the newly connected target sensor lacks initial exposure parameters that match the current ambient light and must undergo the aforementioned closed-loop convergence process (typically lasting 0.5 to 2 seconds). During this period, the image experiences drastic brightness jumps, resulting in overexposure (whitening) or underexposure (blackening), severely disrupting the continuity of the video and the viewing experience.

[0037] Third, there is the issue of latency and contention in the real-time parameter distribution during the switching process.

[0038] In scenarios where multiple sensors share a driver or rely on the underlying SDK for asynchronous operations, attempting to calculate the target sensor's exposure parameters in real time and send them to its registers simultaneously with the switching command is often limited by the following factors: write-to-effectiveness delays caused by register double buffering, timing lags caused by image processing pipeline frame buffering, and scheduling delays in inter-thread communication. These factors collectively result in the first frame of the switching operation failing to carry valid new exposure parameters; that is, the sudden brightness changes in the first frame cannot be eliminated through real-time parameter sending.

[0039] In summary, traditional technical solutions have significant shortcomings in controlling the brightness consistency of multi-lens zoom systems.

[0040] In view of this, this application provides a brightness consistency control method for a multi-view zoom system, which can eliminate brightness statistical deviation and switching convergence delay in the electronic zoom and optical zoom processes, achieve seamless zoom with consistent brightness, and thus improve the visual continuity and user experience of the multi-view zoom system.

[0041] The technical solutions provided in this application can be applied to any imaging device or equipment equipped with multiple cameras with different focal lengths that need to maintain consistent image brightness during electronic zoom or optical zoom. Specifically, they can be applied to scenarios such as smartphones, security monitoring cameras, multi-camera video production equipment, and special machine vision systems.

[0042] The technical solutions provided in this application are described below with reference to the accompanying drawings.

[0043] Figure 1 This is a schematic diagram of a brightness consistency switching process for a brightness consistency control method in a multi-view zoom system provided in an embodiment of this application. Figure 1 As shown, the method may include the following steps S101 to S105: Step S101: Obtain the electronic zoom ratio, determine the cropping area in the original image based on the electronic zoom ratio, and associate the brightness statistics area of ​​the automatic exposure algorithm with the cropping area.

[0044] During electronic zoom, the system can acquire the current electronic zoom level in real time. This electronic zoom level can be triggered by the user sliding the zoom slider or clicking the zoom switch button through the application interface, or it can be automatically generated by the system based on a preset zoom strategy. After acquiring this electronic zoom level, the system can combine it with the current size information of the original image to calculate the position and size of the cropped area in the original image.

[0045] For example, if the original image has a width of W0 and a height of H0, and the current electronic zoom is Z, with the image center as the cropping reference, then the width w and height h of the cropping area are w=W0 / Z and h=H0 / Z, respectively, and the coordinates (x, y) of the upper left corner of the cropping area are x=(W0-w) / 2 and y=(H0-h) / 2, respectively.

[0046] After calculating the coordinates and dimensions of the cropped region, the system can associate the brightness statistics region of the automatic exposure algorithm with that cropped region. Specifically, the system can obtain the configuration information of the brightness statistics window currently used by the automatic exposure algorithm, including the coordinates of the top-left corner of the statistics window and the window size. Then, the system performs a proportional mapping of this configuration information based on the size ratio between the cropped region and the original image.

[0047] For example, if the top-left corner coordinates of the current brightness statistics window are (Xs, Ys), the width is Ws, the height is Hs, the cropping region width is w, and the original image width is W0, then the top-left corner coordinates (Xs', Ys') of the mapped new statistics window are: Xs'=Xs×(w / W0), Ys'=Ys×(h / H0); The width Ws' and height Hs' of the new statistical window are respectively: Ws'=Ws×(w / W0), Hs'=Hs×(h / H0).

[0048] It should be understood that the aforementioned proportional mapping can be a scaling transformation of the coordinate system, used to ensure that the statistical window scales proportionally with the scaling of the cropped area, thus ensuring that the statistical window always covers the corresponding image area within the cropped area. After the mapping is completed, the system writes the new statistical window configuration into the statistical configuration register of the automatic exposure algorithm, enabling the automatic exposure algorithm to use the updated statistical window in the brightness statistics of subsequent frames.

[0049] The technical solution provided in this application embodiment ensures that the brightness statistics area of ​​the automatic exposure algorithm remains consistent with the cropped area of ​​the image actually observed by the user, and will not affect the exposure decision of the current zoom image due to brightness changes in the outer area of ​​the field of view.

[0050] Step S102: Perform brightness statistics on the brightness statistics area using an automatic exposure algorithm to obtain the exposure parameters of the currently active sensor.

[0051] After updating the brightness statistics region, the system can perform brightness statistics on the pixels within the updated statistics window using an automatic exposure algorithm. Specifically, the system can use the automatic exposure algorithm to read the brightness values ​​of each pixel in the current frame image that falls within the statistics window range, and calculate the statistical characteristic values ​​of these pixel brightness values. The statistical characteristic values ​​can be the brightness mean or a weighted average.

[0052] The system compares the calculated statistical characteristic values ​​with preset target brightness values. Based on the difference, it calculates a new set of exposure parameters using a closed-loop feedback control mechanism (e.g., proportional-integral-derivative control). These exposure parameters may include exposure time, analog gain, and digital gain. Exposure time determines the charge accumulation time of the sensor's photosensitive element; analog gain amplifies the analog signal when it is read from the photosensitive element; and digital gain amplifies the digital signal after analog-to-digital conversion. All three factors together determine the final image brightness level.

[0053] In the technical solution provided in this application embodiment, the automatic exposure algorithm can be implemented using any closed-loop feedback control method known in the art, which can make the actual brightness approach the target brightness. After the calculation is completed, the system reads the exposure parameter value of the currently active sensor from the register corresponding to the automatic exposure module in the image signal processor and uses it as the input for step S103.

[0054] Step S103: Based on the pre-calibrated exposure parameter conversion relationship between different sensors, map the exposure parameters of the currently active sensor to the target exposure parameters of the target sensor.

[0055] In the technical solution provided in this application, the system has pre-calibrated the exposure parameter conversion relationship between each sensor before leaving the factory or during the initialization stage. This conversion relationship characterizes the proportional relationship between the exposure parameters of different sensors when they achieve the same output brightness under the same ambient lighting conditions.

[0056] This transformation relationship can be obtained in advance in two ways: Firstly, it is calculated based on the lens optical parameters of each sensor.

[0057] The optical parameters of a lens include its aperture (F-number) and transmittance (T-value). The aperture (F-number) determines the size of the lens's opening; a smaller F-number allows more light to enter per unit time. The transmittance (T-value) reflects the optical system's efficiency in transmitting light. For any two sensors A and B, the conversion ratio Ratio_AB = (F_A² × T_A) / (F_B² × T_B), where F_A and T_A are the aperture (F-number) and transmittance of sensor A, and F_B and T_B are the aperture (F-number) and transmittance of sensor B.

[0058] Secondly, it is obtained through experimental calibration.

[0059] In a production line or laboratory environment, each sensor is aligned with the same uniform light source, and the exposure parameters of each sensor (including at least one of exposure time, analog gain, and digital gain) are adjusted independently to make the brightness of the images output by each sensor consistent. The ratio between the exposure parameters of each sensor recorded at this time is the conversion ratio.

[0060] Regardless of the method used, once calibration is complete, the system can store the conversion ratios between each pair of sensors in a non-volatile storage medium for real-time retrieval in subsequent mapping steps.

[0061] After obtaining the current exposure parameters of the currently active sensor through step S102, the system can identify the identity of the currently active sensor based on the state variables maintained by the zoom control module. Then, based on the target sensor identity indicated by the sensor switching command, the system reads the corresponding conversion ratio from the storage medium. Subsequently, the system multiplies each exposure parameter component (exposure time, analog gain, and digital gain) of the currently active sensor by the conversion ratio to obtain the target exposure parameters that the target sensor should have in the current environment.

[0062] It should be understood that in the technical solutions provided in this application, the above mapping operation can be performed simultaneously on multiple inactive sensors. When there are multiple inactive sensors in the system, the system can read the conversion ratio between each inactive sensor and the currently active sensor, and calculate the target exposure parameters of each inactive sensor in parallel or sequentially, so as to prepare parameters for possible subsequent switching in any direction.

[0063] Step S104: Write the target exposure parameters into the register of the target sensor.

[0064] After calculating the target exposure parameters for the target sensor, the system writes these parameters into the target sensor's register. Specifically, the system can write data to the target sensor's exposure control register via a serial communication interface (such as I²C or SPI bus) between the sensor and the control chip. The data written includes the target exposure time, target analog gain, and target digital gain.

[0065] It should be understood that in the technical solution provided in this application embodiment, the write operation is not executed only once during system operation, but is continuously executed in a loop. That is, whenever the new exposure parameters of the currently active sensor are calculated by the method provided in step S102, and the new target exposure parameters of the target sensor are subsequently calculated in step S103, the system writes the latest target exposure parameters into the register of the target sensor by the method provided in step S104. Since the exposure parameters of the currently active sensor may be updated frame by frame with changes in ambient light, the parameter values ​​in the register of the target sensor in this application are also updated synchronously, always matching the ambient light at the current moment.

[0066] When the system performs this write operation, the target sensor can be in an inactive state (i.e., powered on but not outputting image data). The write operation does not affect the current state of the target sensor, nor does it trigger its image output. This write operation utilizes the sensor's normal read and write capabilities to registers in standby or idle states.

[0067] It should be understood that when there are multiple target sensors in the system, the system performs the above-described write operation independently for each target sensor. Each target sensor's register independently stores its corresponding latest target exposure parameters.

[0068] Step S105: When a sensor switching command is received, the target sensor is connected to the image signal processor, and the target sensor outputs an image with the target exposure parameters stored in the register.

[0069] In the technical solutions provided in this application embodiment, the triggering methods for the sensor switching command include: the user clicks the magnification switching button (e.g., switching from 1× to 2×) through the application interface, and the system detects the operation and identifies the sensor corresponding to the target magnification; or, the user slides the zoom slider across a preset switching magnification threshold, and the system determines that the sensor needs to be switched; or, the system generates a sensor switching event according to a preset automatic zoom strategy.

[0070] Regardless of the triggering method, upon receiving a switching command, the system first identifies the target sensor indicated by the command, and then executes the link switching operation. The method of link switching depends on the specific hardware architecture of the system.

[0071] If the system's hardware architecture allows each sensor to have its own independent MIPI channel, and each MIPI channel is connected to a different video interface, then the system performs a rebinding operation between the video interface and the image signal processor (Image Signal Processor). This rebinds the video interface corresponding to the target sensor to the input port of the Image Signal Processor, allowing the image data from the target sensor to be processed by the Image Signal Processor. In this architecture, image data streams from different sensors are transmitted in parallel through different video interfaces; the switching operation essentially changes the Image Signal Processor's selection of the video interface.

[0072] If the system's hardware architecture consists of multiple sensors connected to a single MIPI bus via a MIPI switch (i.e., multiplexing), the system will perform a MIPI bus switching operation to switch the MIPI switch's on / off state to the branch where the target sensor is located, allowing the target sensor's image data to be transmitted to the image signal processor via that MIPI bus.

[0073] The two link switching methods are selected based on the hardware design, and the system uses only one of them; the two will not be executed simultaneously. After the link switching operation is completed, the target sensor is officially connected to the image signal processor and begins to output image data. The first frame image output by the target sensor is exposed and gain controlled according to the target exposure parameters pre-stored in its register. Therefore, the first frame image already has a brightness that matches the current ambient light when it is output. Since the target exposure parameters are pre-written into the register before switching, the target sensor does not need to wait for the automatic exposure algorithm to complete new brightness statistics and closed-loop convergence after connection. It directly exposes and outputs with the pre-stored parameters, achieving the accurate predetermined effect of the brightness of the first frame after switching.

[0074] In the technical solution provided in this application embodiment, the system can dynamically associate the AE statistical window with the cropping area according to the electronic zoom magnification, and map the exposure parameters of the currently active sensor based on the pre-calibrated conversion relationship and pre-write them into the register of the target sensor. When switching, the system outputs the pre-stored parameters directly, which enables the AE algorithm to always evaluate the brightness based on the user's actual observation of the picture, eliminates the interference of light outside the field of view on the exposure decision, and enables the target sensor to have accurate exposure parameters in the first frame after accessing the ISP. This completely eliminates the closed-loop convergence process of several frames to several seconds in the traditional solution, thereby achieving accurate metering and seamless switching from accurate first frame in the entire process of electronic zoom and optical zoom, fundamentally eliminating brightness jumps and flicker, and improving the brightness consistency and visual experience of the multi-lens zoom system.

[0075] The following section, with reference to the accompanying diagrams, provides a detailed explanation of the two hardware architectures for sensor switching and their corresponding switching methods.

[0076] Figure 2This diagram illustrates a multi-view zoom system pipeline under a VIF rebinding architecture, providing a brightness consistency control method for a multi-view zoom system according to an embodiment of this application. Figure 2 As shown, in the VIF rebinding architecture, the system includes multiple image sensors (sensor 0, sensor 1, sensor 2), each with its own independent MIPI channel (MIPI0, MIPI1, MIPI2), and each MIPI channel is connected to a different video interface (VIF0, VIF1, VIF2). Each video interface is connected to the input port of the image signal processor. In this architecture, the physical connection between each sensor and its corresponding video interface is fixed, and the image signal processor can only receive image data from one video interface at a time. When the system receives a sensor switching command, it performs a rebinding operation between the video interface and the input port of the image signal processor, binding the video interface corresponding to the target sensor to the image signal processor, thereby allowing the image data from the target sensor to enter the image signal processor for processing.

[0077] Figure 3 This is a schematic diagram of a multi-view zoom system pipeline under a MIPI switching architecture, which is provided as an embodiment of the present application for a brightness consistency control method for a multi-view zoom system. Figure 3 As shown, in the MIPI switching architecture, the system includes multiple image sensors (sensor 0, sensor 1, and sensor 2), each connected to a different input port of the MIPI switch. The output port of the MIPI switch is connected to the video interface (VIF0) via a single MIPI bus (MIPI0), and then connected to the image signal processor (Image Signal Processor) via the video interface. In this architecture, multiple sensors share the same MIPI bus and the same video interface. When the system receives a sensor switching command, it executes a MIPI bus switching operation, switching the connection state of the MIPI switch from the branch where the current sensor is located to the branch where the target sensor is located. This allows the image data from the target sensor to be transmitted to the video interface via the MIPI bus, and then sent to the Image Signal Processor for processing.

[0078] In both architectures described above, regardless of the link switching method used, since the target sensor's register has already stored the target exposure parameters that match the current ambient light through the aforementioned steps, the target sensor outputs an image with the target exposure parameters stored in its register after being connected to the image signal processor. The first frame of the output image has the correct brightness, without having to wait for the closed-loop convergence process of the automatic exposure algorithm.

[0079] In some implementations, the system may perform a compensation strategy to optimize the adjustment of exposure parameters based on scene motion state before writing the target exposure parameters into the target sensor's register.

[0080] Specifically, the system uses a motion detection module to detect whether there are moving targets in the current scene. The motion detection module can use any one or more combinations of inter-frame differencing, background subtraction, or optical flow methods to detect moving targets. When the motion detection module outputs that there is a moving target in the current scene, the system determines that there is a moving target in the current scene; when the motion detection module outputs that there is no moving target in the current scene, the system determines that there is no moving target in the current scene.

[0081] If a moving target is detected in the scene, the system limits the relaxation range of the exposure time when calculating the target's exposure parameters. Specifically, the relaxation range of the exposure time is limited to a preset percentage of the maximum exposure time (e.g., 30%). The remaining exposure gap after the exposure time compensation is limited is filled by gain compensation, that is, brightness compensation is mainly achieved by gain adjustment, thereby avoiding motion blur or trailing in the image due to excessive exposure time.

[0082] If no moving target is determined in the scene, the system prioritizes using exposure time compensation for brightness adjustment. Specifically, the system temporarily relaxes the upper limit of exposure time from the first limit value defaulted by the image signal processor algorithm to a second limit value supported by the sensor hardware, where the second limit value is greater than the first limit value.

[0083] For example, the first limit could be the upper limit of the exposure time of the automatic exposure algorithm in the image signal processor under normal operating conditions (e.g., 80ms), and the second limit could be the maximum exposure time allowed by the target sensor hardware specifications (e.g., 200ms). By utilizing a wider range of exposure time adjustment, the system can compensate for differences in light intake between different lenses without introducing excessive gain noise, thereby achieving a better signal-to-noise ratio in low-light scenes.

[0084] It should be understood that the above compensation strategy remains in effect for a predetermined number of frames after the switch is completed. Subsequently, the system will gradually restore the upper limit of the exposure time to the first limit value to avoid image blurring caused by prolonged use of excessively long exposure times.

[0085] It should be understood that, provided there are no logical conflicts, the above-described embodiments can be combined and implemented to adapt to actual application needs. These combined embodiments or implementation schemes are still within the scope of protection of this application.

[0086] Corresponding to the brightness consistency control method of the multi-view zoom system in the above embodiments, this application provides a brightness consistency control device 40 for a multi-view zoom system. The brightness consistency control device 40 for the multi-view zoom system can be implemented by software, hardware or a combination of both as part or all of a computer device, and is used to execute the steps in the brightness consistency control method of the multi-view zoom system in the above embodiments.

[0087] Figure 4 The diagram shows a schematic of the brightness consistency control device 40 of a multi-view zoom system provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0088] Reference Figure 4 The brightness consistency control device 40 of the multi-lens zoom system includes an acquisition module 410 and a processing module 420.

[0089] The acquisition module 410 is used to acquire the electronic zoom ratio, determine the cropped area in the original image based on the electronic zoom ratio, and associate the brightness statistics area of ​​the automatic exposure algorithm with the cropped area.

[0090] The processing module 420 is used to perform brightness statistics on the brightness statistics area through an automatic exposure algorithm to obtain the exposure parameters of the currently active sensor.

[0091] The processing module 420 is also used to map the exposure parameters of the currently active sensor to the target exposure parameters of the target sensor according to the pre-calibrated exposure parameter conversion relationship between different sensors.

[0092] The processing module 420 is also used to write the target exposure parameters into the register of the target sensor.

[0093] The processing module 420 is also used to connect the target sensor to the image signal processor when a sensor switching command is received, and the target sensor outputs an image with the target exposure parameters pre-stored in the register.

[0094] In some embodiments, the acquisition module 410 is specifically used to: calculate the coordinate information and window size of the cropping region based on the electronic zoom ratio; and proportionally map the coordinate information and window size of the brightness statistics region according to the size ratio between the cropping region and the original image.

[0095] In some embodiments, the exposure parameter conversion relationship is obtained based on the optical parameters of each sensor, including lens aperture value and transmittance. Alternatively, it is obtained by calibrating different sensors to the same uniform light source and adjusting the exposure parameters to make the output brightness consistent; the exposure parameters include at least one of exposure time, analog gain, and digital gain.

[0096] In some embodiments, the processing module 420 is specifically configured to: obtain the current exposure parameters of the currently active sensor; and multiply the current exposure parameters by the conversion ratio corresponding to the exposure parameter conversion relationship to obtain the target exposure parameters.

[0097] In some embodiments, the processing module 420 is specifically configured to: detect whether a moving target exists in the current scene using the motion detection module; if a moving target is detected, limit the relaxation range of the exposure time and correct the target exposure parameters using gain compensation; if no moving target is detected, correct the target exposure parameters using exposure time compensation.

[0098] In some embodiments, the processing module 420 is specifically used to: compare the current frame with the previous frame using the motion detection module, and output a determination result indicating whether a moving target exists based on the comparison result. Limiting the relaxation range of exposure time includes: limiting the relaxation range of exposure time to a preset proportion of the maximum exposure time.

[0099] In some embodiments, the processing module 420 is specifically configured to: temporarily relax the upper limit of exposure time from a first limit value defaulted to by the image signal processor algorithm to a second limit value supported by the sensor hardware, wherein the second limit value is greater than the first limit value.

[0100] In some embodiments, the processing module 420 is specifically configured to: when the hardware architecture is such that each sensor has an independent MIPI channel and each channel is connected to a different video interface, perform a rebinding operation between the video interface and the image signal processor, binding the video interface corresponding to the target sensor to the image signal processor. When the hardware architecture is such that multiple sensors share a single MIPI bus via a MIPI switch, perform a MIPI bus switching operation, connecting the target sensor to the image signal processor via the MIPI bus.

[0101] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0102] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0103] Based on the same inventive concept, embodiments of this application also provide an electronic device.

[0104] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 5As shown, the electronic device 50 of this embodiment includes: at least one processor 510 ( Figure 5 Only one is shown in the diagram), memory 520, and communication module 530. Memory 520 stores a computer program 540 that may run on processor 510. When processor 510 executes computer program 540, it implements the steps in the above-described embodiment of the brightness consistency control method for the multi-view zoom system, for example... Figure 1 Steps S101 to S105 are shown. Alternatively, when processor 510 executes computer program 540, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of the acquisition module 410 to the processing module 420 are shown. The communication module 530 can be a separate communication unit used to communicate with external servers or terminal devices.

[0105] Electronic device 50 may include, but is not limited to, a processor 510 and a memory 520. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 50 and does not constitute a limitation on electronic device 50. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 50 may also include input transmitting devices, network access devices, buses, etc.

[0106] The processor 510 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0107] In some embodiments, memory 520 may be an internal storage unit of electronic device 50, such as a hard disk or memory of electronic device 50. Memory 520 may also be an external storage device of electronic device 50, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on electronic device 50. Memory 520 may include both internal and external storage units of electronic device 50. Memory 520 is used to store operating system, applications, bootloader, data, and other programs, such as the program code of computer program 540. Memory 520 may also be used for temporary storage of data that has been sent or will be sent.

[0108] Furthermore, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. In the various embodiments of this application, each functional unit can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] This application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the steps described in the various method embodiments above.

[0110] This application provides a chip, which includes a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps in the various method embodiments described above.

[0111] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the steps described in the various method embodiments above.

[0112] It should be understood that the processor mentioned in the embodiments of this application can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0113] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0115] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0116] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0120] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a large-screen device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0121] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A brightness consistency control method for a multi-view zoom system, characterized in that, include: Obtain the electronic zoom ratio, determine the cropped area in the original image based on the electronic zoom ratio, and associate the brightness statistics area of ​​the automatic exposure algorithm with the cropped area; The brightness of the brightness statistics area is statistically analyzed using the automatic exposure algorithm to obtain the exposure parameters of the currently active sensor. Based on the pre-calibrated exposure parameter conversion relationship between different sensors, the exposure parameters of the currently active sensor are mapped to the target exposure parameters of the target sensor; Write the target exposure parameters into the register of the target sensor; When a sensor switching command is received, the target sensor is connected to the image signal processor, and the target sensor outputs an image using the target exposure parameters pre-stored in the register.

2. The brightness consistency control method for a multi-lens zoom system according to claim 1, characterized in that, The step of associating the brightness statistics region of the automatic exposure algorithm with the cropping region includes: Calculate the coordinate information and window size of the cropping area based on the electronic zoom ratio; The coordinate information and window size of the brightness statistics region are mapped proportionally according to the size ratio between the cropped area and the original image.

3. The brightness consistency control method for a multi-lens zoom system according to claim 1, characterized in that, The exposure parameter conversion relationship is obtained based on the optical parameters of each sensor, including the lens aperture value and light transmittance; Alternatively, the calibration can be obtained by aligning different sensors with the same uniform light source and adjusting the exposure parameters to make the output brightness consistent. The exposure parameters include at least one of exposure time, analog gain, and digital gain.

4. The brightness consistency control method for a multi-lens zoom system according to claim 1, characterized in that, The step of mapping the exposure parameters of the currently active sensor to the target exposure parameters of the target sensor includes: Obtain the current exposure parameters of the currently active sensor; The target exposure parameter is obtained by multiplying the current exposure parameter by the conversion ratio corresponding to the exposure parameter conversion relationship.

5. The brightness consistency control method for a multi-lens zoom system according to claim 1, characterized in that, Before writing the target exposure parameters into the register of the target sensor, the method further includes: The motion detection module detects whether there are moving targets in the current scene; If the moving target is detected, the relaxation range of the exposure time is limited, and the exposure parameters of the target are corrected by gain compensation; If the moving target is not detected, exposure time compensation is used to correct the target's exposure parameters.

6. The brightness consistency control method for a multi-lens zoom system according to claim 5, characterized in that, The detection of whether a moving target exists in the current scene includes: The motion detection module compares the current frame with the previous frame and outputs a determination result on whether a moving target exists based on the comparison result. The relaxation of the exposure time limit includes: The relaxation range of the exposure time is limited to a preset percentage of the maximum exposure time.

7. The brightness consistency control method for a multi-lens zoom system according to claim 5, characterized in that, The step of correcting the target exposure parameters using exposure time compensation includes: The upper limit of exposure time is temporarily relaxed from the first limit value defaulted to by the image signal processor algorithm to a second limit value supported by the sensor hardware, which is greater than the first limit value.

8. The brightness consistency control method for a multi-lens zoom system according to claim 1, characterized in that, The step of connecting the target sensor to the image signal processor includes: When the hardware architecture is such that each sensor has an independent MIPI channel and each channel is connected to a different video interface, a rebinding operation is performed between the video interface and the image signal processor to bind the video interface corresponding to the target sensor to the image signal processor. When the hardware architecture allows multiple sensors to share a single MIPI bus via a MIPI switch, a MIPI bus switching operation is performed to connect the target sensor to the image signal processor via the MIPI bus.

9. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement a brightness consistency control method for a multi-view zoom system as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the brightness consistency control method of the multi-view zoom system as described in any one of claims 1 to 8.

Images