Image system control method, computer storage medium and device thereof

Abstract

This invention provides an imaging system control method, related apparatus, and computer-readable medium. The imaging system control method includes: operating a first camera module among a plurality of camera modules to generate at least one first frame; operating a second camera module among the plurality of camera modules to generate at least one second frame; generating first statistical data based on at least one first frame and generating second statistical data based on at least one second frame; and determining a first adjustment target for imaging control parameters of the first camera module based on at least the first statistical data. The imaging system control method, related apparatus, and computer-readable medium of this invention enhance the user experience during photographic and video recording sessions.

Description

[Technical Field]

[0001] The present invention relates to imaging systems, and more particularly, to a method and apparatus for controlling an imaging system having multiple camera modules. [Previous Technology]

[0002] Spatial Alignment Transformation (SAT) zoom is widely used in multi-camera systems to achieve seamless transitions between different focal lengths. However, maintaining 3A (auto-exposure, auto-white balance, auto-focus) consistency among camera modules presents significant challenges due to inherent differences in sensor characteristics. Traditional methods use a viewing camera module to determine the 3A adjustment target, which often leads to inconsistencies in color and brightness during camera module switching. To address this issue, a background master decision mechanism is proposed, which simultaneously activates at least two camera modules: one acts as a foreground viewing camera module for real-time preview and capture, while another well-adjusted camera module (i.e., adjusted for 3A consistency decisions) acts as a background master camera module to determine the 3A adjustment target for all camera modules. This approach reduces the adjustment workload and may improve consistency, but it also introduces challenges such as increased power consumption and potential latency in 3A consistency. [Summary of the Invention]

[0003] In view of this, one object of the present invention is to provide a 3A consistency decision and control mechanism for controlling an imaging system having multiple camera modules. In embodiments of the present invention, multiple camera modules of the imaging system can be activated simultaneously, wherein at least one activated camera module operates in a low-power state (e.g., low frames per second (FPS) mode, low-resolution mode, and / or non-high dynamic range (non-HDR) or standard dynamic range (SDR) capture mode), while at least one activated camera module operates in a normal power state. In the 3A consistency decision and control mechanism of the present invention, the 3A adjustment target of the imaging control parameters of the camera module is determined by considering statistical data of frame generation based on more than one activated camera module.

[0004] An embodiment of the present invention provides an imaging system control method. The imaging system has multiple camera modules. The imaging system control method includes: operating a first camera module among the multiple camera modules to generate at least one first frame; operating a second camera module among the multiple camera modules to generate at least one second frame; generating first statistical data based on at least one first frame and generating second statistical data based on at least one second frame; and determining a first adjustment target for imaging control parameters of the first camera module based on at least the first statistical data.

[0005] Another embodiment of the present invention provides a computer-readable medium storing a plurality of instructions, responsive to execution on a computing device, to cause the computing device to operate in such a way as to: operate a first camera module of a plurality of camera modules to generate at least one first frame; operate a second camera module of a plurality of camera modules to generate at least one second frame; generate first statistics based on at least one first frame and generate second statistics based on at least one second frame; and determine a first adjustment target for imaging control parameters of the first camera module based on at least the first statistics.

[0006] Another embodiment of the present invention provides an apparatus for controlling an imaging system, comprising: at least one memory configured to store a plurality of instructions; and at least one processor communicatively coupled to the at least one memory and configured to execute the plurality of instructions to perform the following operations: generating first statistics based on at least one first frame obtained by operating a first camera module of a plurality of camera modules of the imaging system; generating second statistics based on at least one second frame obtained by operating a second camera module of the plurality of camera modules; and determining a first adjustment target for imaging control parameters of the first camera module based on at least the first statistics.

[0007] The technical solution of the present invention enhances the user experience during photography and video recording sessions.

Implementation Method

[0009] It will be readily understood that the components of the present invention, as generally described and illustrated in the accompanying drawings, can be arranged and designed in a variety of different configurations. Therefore, a more detailed description of embodiments of the system and method of the present invention as shown in the figures follows.

[0010] Several embodiments will now be described in detail. Although the subject matter will be described in conjunction with alternative embodiments, it should be understood that they are not intended to limit the claimed subject matter to these embodiments. Rather, the claimed subject matter is intended to cover alternatives, modifications, and equivalents that may be included within the spirit and scope of the claimed subject matter as defined by the appended claims.

[0011] In the following description, numerous specific details are set forth in order to provide a comprehensive understanding of this embodiment. However, it will be apparent to those skilled in the art that no such specific detail is required to practice this embodiment. In other instances, well-known materials or methods have not been described in detail to avoid obscuring this embodiment.

[0012] Throughout this specification, references to “one embodiment,” “embodiment,” or “some embodiments” mean that a particular feature, structure, or characteristic described in relation to an embodiment or example is included in at least one embodiment of this embodiment. Therefore, the phrases “in one embodiment,” “in an embodiment,” or “in some embodiments” appearing throughout this specification do not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any suitable combination and / or sub-combination in one or more embodiments.

[0013] Overview

[0014] Figure 1 illustrates an imaging system and a corresponding electronic device according to an embodiment of the present invention. The imaging system of the present invention can be applied to and configured in mobile phones, tablets, laptops, digital cameras, camcorders, augmented or virtual reality headsets, home security cameras, facial recognition access control systems, vehicle dashcams, satellite imaging systems, drone cameras, 360-degree surround view systems, cinema cameras, broadcast-grade cameras, virtual studio systems, or other types of electronic devices for photographic or video capture purposes. As shown in Figure 1, the imaging system 100 may be disposed in the electronic device 10 and may include multiple camera modules 110, 120, and 130. In some embodiments, each camera module 110, 120, and 130 may include lenses (i.e., 111, 121, and 131), image sensors (i.e., 112, 122, and 132), and control circuitry (i.e., 113, 123, and 133). Note that the number of camera modules described in this embodiment is for illustrative purposes only and should not be considered a limitation of the invention. According to various embodiments of the present invention, the imaging system 100 may include fewer than three or more than three camera modules.

[0015] In some embodiments, lenses 111, 121, and 131 may be configured with different optical characteristics, such as focal length, aperture size, and field of view, to provide different imaging capabilities. In some embodiments, image sensors 112, 122, and 132 may be solid-state electronic devices, such as complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) sensors, capable of converting optical images into electronic signals. Image sensors 112, 122, and 132 may differ in resolution, pixel size, and sensitivity to accommodate different imaging requirements. Control circuits 113, 123, and 133 may include a microprocessor, digital signal processor (DSP), or dedicated application-specific integrated circuit (ASIC) for managing the operation of each camera module 110, 120, and 130. Control circuits 113, 123, and 133 may control functions such as autofocus, automatic white balance, automatic exposure adjustment, image stabilization, and data processing.

[0016] Specifically, each camera module 110, 120, and 130 can correspond to a specific focal length and can be used to capture photos or videos at that specific focal length. For example, camera module 110 can be configured for ultra-wide-angle shooting with a shorter focal length (e.g., 16mm equivalent), camera module 120 can be configured for wide-angle shooting with a medium focal length (e.g., 28mm equivalent), and camera module 130 can be configured for telephoto shooting with a longer focal length (e.g., 85mm equivalent). In some embodiments, each of camera modules 110, 120, and 130 is equipped with high dynamic range (HDR) imaging capabilities. Each of camera modules 110, 120, and 130 is designed to be flexible enough to operate in HDR or standard dynamic range (SDR) capture modes depending on scene conditions and specific imaging requirements.

[0017] In some embodiments, at least two of camera modules 110, 120, and 130 are simultaneously activated for 3A consistency decision-making and control. Each of the activated camera modules operates in either a foreground or background state. That is, each of camera modules 110, 120, and 130 can be configured to operate in either a foreground or background state (if activated). When a camera module operates in the foreground state, i.e., as a "foreground viewing" camera module, at least one frame generated by the foreground viewing camera module is used for real-time preview and image / video capture, as well as 3A consistency decision-making and control. Furthermore, when a camera module operates in the background state, i.e., as a "background master" camera module, at least one frame generated by the background master camera module is used for 3A consistency decision-making and control.

[0018] For power consumption considerations, power control strategies can be implemented to reduce the power consumption of the activated camera module. In some embodiments, the background main camera module can operate in a low-power state, while the foreground viewing camera module can operate in a normal-power state. In some embodiments, the background main camera module can operate in a low frame rate mode (e.g., 20, 10, or 5 frames per second (FPS)) to achieve lower power consumption, while the foreground viewing camera module can operate in a normal frame rate mode (e.g., 30 FPS). In some embodiments, the background main camera module can operate in a low-resolution mode to achieve lower power consumption, while the foreground viewing camera module can operate in a high-resolution or native resolution mode. Specifically, the background main camera module can utilize techniques such as pixel binning or skip sampling to output frames with a lower resolution compared to the original image sensor output resolution. In some embodiments, the background main camera module can operate in a non-HDR or SDR shooting mode to achieve low power consumption, while the foreground viewing camera module can operate in an HDR shooting mode.

[0019] In some embodiments, the processing unit 140 may be communicatively coupled to camera modules 110, 120, and 130. The processing unit 140 may be configured to execute instructions and / or reference materials or information stored in memory 150 to perform tasks such as image fusion, noise reduction, color correction, lens distortion correction, image stabilization, and / or HDR processing. In some embodiments, the processing unit 140 may be used to achieve depth mapping for portrait mode effects, super-resolution imaging through multi-frame synthesis, and low-light performance enhanced through image stacking. Furthermore, the processing unit 140 may be configured to coordinate the operation of camera modules 110, 120, and 130 to achieve seamless switching between different focal lengths and ensure optimal image quality in various shooting scenarios.

[0020] According to various embodiments of the present invention, the processing unit 140 can be implemented in a variety of configurations to optimize performance, power efficiency, and integration within the imaging system 100 or electronic device 10. In some embodiments, the processing unit 140 can be integrated into the imaging system 100 or disposed within the electronic device 10. For example, the processing unit 140 can be a specific imaging processing circuit embedded in a system-on-a-chip (SoC) of the electronic device 10, such as part of an image signal processor (ISP) in a mobile phone SoC. Alternatively, the processing unit 140 can be implemented as a standalone image signal processor. The processing unit 140 can also employ a distributed architecture, where some low-level processing tasks are performed by circuitry close to the image sensor, while more complex calculations are handled by a central processing unit. In some embodiments, to handle the parallel processing of multiple image streams, the processing unit 140 can have a multi-core architecture, thereby allowing simultaneous processing of data from image sensors 112, 122, and 132.

[0021] In summary, the imaging system 100 can be used as a zoom imaging system, capable of capturing high-quality images and videos over a wide focal length range without the need for a mechanical zoom mechanism. The imaging system 100 not only enhances the versatility of imaging capabilities but also contributes to the compact form factor of the electronic device 10.

[0022] 3A Consistent Decision-Making and Control

[0023] Due to the inherent characteristics of different image sensors and lenses, the images output by camera modules 110, 120, and 130 may exhibit different trends in brightness and color rendering. When a user switches between camera modules 110, 120, and 130 (i.e., changes the shooting focal length), this difference can lead to noticeable and unnatural changes in the live preview. This inconsistency can degrade the user experience and the overall quality of the imaging system. To address this issue, the present invention proposes a method for controlling an imaging system with multiple camera modules. The method of the present invention can be used to achieve seamless transitions between camera modules, ensuring excellent consistency in brightness and color rendering across different camera modules.

[0024] According to one embodiment, a method for controlling an imaging system having multiple camera modules includes the following steps as shown in Figure 2. Please refer to Figure 2 in conjunction with Figure 3. In step S101, a first camera module FCM (i.e., one of camera modules 110, 120, and 130) is operated to generate at least one first frame. In step S102, a second camera module SCM (i.e., the other of camera modules 110, 120, and 130) is operated to generate at least one second frame.

[0025] In some embodiments, the first camera module FCM may be one of a background main camera module and a foreground viewing camera module, while the second camera module SCM may be one of a background main camera module and a foreground viewing camera module. In some embodiments, at least one first frame generated by the first camera module FCM may be used for 3A consistency decision-making, while at least one second frame generated by the second camera module SCM may be used for 3A consistency decision-making, and / or for real-time preview, photo / video capture. In some embodiments, the first camera module FCM operates in a low frame rate mode, such as 20, 10, or 5 FPS, while the second camera module SCM operates in a normal frame rate mode, such as 30 FPS. In some embodiments, the first camera module operates in a low resolution mode, while the second camera module operates in a high resolution or native resolution mode. In some embodiments, the first camera module FCM operates in a non-HDR or non-SDR capture mode, while the second camera module SCM operates in an HDR capture mode. In some embodiments, the focal length of the lens of the first camera module FCM is longer or shorter than the focal length of the lens of the second camera module SCM.

[0026] In step S103, first statistical data is generated based on at least one first frame, and second statistical data is generated based on at least one second frame. In some embodiments, the first and second statistical data can be calculated based on a block-based algorithm or a histogram-based algorithm. In some embodiments, the first and second statistical data may include R (red), G (green), B (blue), and / or Y (brightness) information, wherein the Y information can be determined by a weighted sum of the R, G, and B information, and the weight values ​​can be configured according to different requirements.

[0027] In some embodiments, the first statistical data may be generated by the image signal processing circuit (not shown), processing unit 140, and / or first statistical circuit (e.g., functional block of control circuit 113, 123, or 133) of the imaging system 100 or electronic device 10 based on image data (e.g., brightness and / or color information) of at least one first frame. In some embodiments, the second statistical data may be generated by the image signal processing circuit (not shown), processing unit 140, and / or second statistical circuit (e.g., functional block of control circuit 113, 123, or 133) of the imaging system 100 or electronic device 10, or second statistical circuit (e.g., functional block of control circuit 113, 123, or 133) of the second camera module SCM.

[0028] In step S104, a first (3A) adjustment target for the imaging control parameters of the first camera module FCM is generated based on at least the first statistical data. Specifically, the imaging control parameters may be 3A control parameters, including at least one of automatic white balance control parameters, automatic exposure control parameters, and automatic focus control parameters for controlling the camera module. Furthermore, each camera module 110, 120, and 130 may have its own imaging control parameters. In some embodiments, the first adjustment target for the imaging control parameters of the first camera module FCM may also be determined based on the first statistical data of the first camera module FCM and the current imaging control parameters.

[0029] In some embodiments, the first adjustment target of the imaging control parameters of the first camera module FCM can be determined by the processing unit 140 based on first statistical data and second statistical data. Furthermore, the first adjustment target of the imaging control parameters of the first camera module FCM can also be determined based on the first statistical data, the second statistical data, and the current imaging control parameters of at least one of the first camera module FCM and the second camera module SCM. In some embodiments, the determination of the first adjustment target can be based on a function f1, which can be expressed as: T1 = f1(S1, S2, C1, C2), where T1 is the first adjustment target, S1 is the first statistical data, S2 is the second statistical data, C1 is the current imaging control parameter of the first camera module FCM, and C2 is the current imaging control parameter of the second camera module SCM.

[0030] Furthermore, after generating a first adjustment target for the imaging control parameters of the first camera module FCM, the processing unit 140 performs inter-module synchronization control between the first camera module FCM and the second camera module SCM. Specifically, the processing unit 140 is configured to achieve inter-module synchronization control by determining a second (3A) adjustment target for the imaging control parameters of the second camera module SCM. In some embodiments, the second adjustment target is determined based on the first adjustment target, first statistics, and second statistics (or further based on the current imaging control parameters of at least one of the first camera module FCM and the second camera module SCM). In some embodiments, the second adjustment target is determined based on a function f2, which can be expressed as: T2 = f2(T1, S1, S2, C1, C2), where T1 is the first adjustment target, T2 is the second adjustment target, S1 is the first statistics, S2 is the second statistics, C1 is the current imaging control parameter of the first camera module FCM, and C2 is the current imaging control parameter of the second camera module SCM.

[0031] Due to the power control strategy, the second camera module SCM may be more sensitive to dynamic scene conditions. This increased sensitivity is primarily based on the higher operating frame rate of the second camera module SCM compared to the first camera module FCM. For example, the frame rate at which the second camera module SCM operates to generate at least one second frame (e.g., 30 fps or higher) is significantly higher than the frame rate at which the first camera module FCM operates to generate at least one first frame (e.g., 20 fps or less than 30 fps). The frame rate difference between the two camera modules results in a significant advantage: the at least one second frame generated by the second camera module SCM exhibits a better ability to reflect or capture rapidly changing scene conditions, particularly in terms of brightness and color changes. Utilizing this improved sensitivity, the processing unit 140 is configured with an adaptive mechanism to re-evaluate and adjust 3A consistency decisions and control in real time.

[0032] Specifically, in some embodiments, the processing unit 140 is configured to redetermine a second adjustment target based on second statistics in response to changes in scene conditions. This redetermination operation ensures that the imaging system 100 maintains the optimal configuration for the current environmental conditions. Furthermore, to maintain 3A consistency among camera modules, the processing unit 140 is configured to update the first adjustment target based on the redetermined second adjustment target, the first statistics, and the second statistics after the redetermination of the second adjustment target.

[0033] Decision-making in HDR capture mode

[0034] When the first camera module FCM operates in a non-HDR or non-SDR capture mode and the second camera module SCM operates in an HDR capture mode (due to the power control strategy), the processing unit 140 can determine the 3A adjustment target based on second statistics based on at least one frame generated by the second camera module. That is, the first camera module FCM operates in a non-HDR or non-SDR capture mode to generate at least one first frame, while the second camera module SCM operates in an HDR capture mode to generate at least one second frame.

[0035] In some cases, first statistics based on at least one first frame cannot capture comprehensive scene information, especially in highlight and shadow areas, which is crucial for accurate 3A consistency decisions and control in HDR scenes. Therefore, processing unit 140 needs to refer to second statistics based on at least one second frame generated in the HDR capture mode. This HDR-based statistics provides a more complete scene representation, including detailed information on high-brightness and low-brightness areas, enabling processing unit 140 to correctly determine the imaging control parameters for optimal HDR capture (i.e., the second adjustment target of the second camera module SCM).

[0036] Therefore, in some embodiments, the processing unit 140 is configured to determine the first adjustment target based solely on the second statistical data, and accordingly perform inter-module synchronization control by determining the second adjustment target of the second camera module SCM based on the first adjustment target, the first statistical data, and the second statistical data. The second camera module SCM then operates in HDR shooting mode based on the second adjustment target.

[0037] Furthermore, in some embodiments, the processing unit 140 may determine the 3A adjustment target and the control parameters for HDR capture, respectively. Specifically, the processing unit 140 is configured to determine the control parameters (e.g., HDR operating mode, digital conversion gain (DCG) or LMBF) of the HDR capture mode of the second camera module SCM based on second statistical data. On the other hand, the processing unit 140 is also configured to perform inter-module synchronization control by determining a second adjustment target of the second camera module SCM based on the first adjustment target, the first statistical data, and the second statistical data. Then, based on the second adjustment target and the control parameters of the HDR capture mode, the second camera module SCM operates in the HDR capture mode.

[0038] FOV Alignment

[0039] In the 3A consistency decision and control mechanism of the present invention, since the generation of frames for determining the first and second statistics relies on camera modules with different focal lengths, field of view (FOV) alignment between at least one first frame and at least one second frame is necessary. Specifically, the processing unit 140 is configured to determine a region of interest (ROI) containing an overlapping area (e.g., an image patch) of at least one first frame and at least one second frame. The determination of the ROI is crucial because the live preview is generated based on one of the at least one first frame and at least one second frame, while the other frame may contain areas that are not visible in the live preview due to FOV differences.

[0040] One of the purposes of establishing ROI is to exclude or minimize the impact of image data from areas invisible in the live preview on the calculation of the first and second statistics. By focusing on the ROI, it is ensured that 3A consistency decisions and controls are primarily based on the image content actually visible to the user in the preview, thereby improving the accuracy of 3A consistency decisions and controls.

[0041] Please refer to Figure 4 for further understanding. In condition (a) shown in Figure 4, it is assumed that the first frame FF is generated by a first camera module FCM with a shorter focal length (i.e., a wider FOV), while the second frame SF is generated by a second camera module SCM (also used as a foreground viewing camera module) with a longer focal length (i.e., a narrower FOV). Since the second camera module SCM is used as a foreground viewing camera module, determining the first statistical data requires excluding or reducing the influence of image data (e.g., brightness and / or color information) from regions outside the FOV of the second frame SF from the first frame FF. During FOV alignment, only overlapping or partially overlapping regions are used to provide meaningful data for the 3A consistency decision (through weight allocation). FOV alignment makes it possible to make more accurate comparisons and analyses of frames generated by different camera modules.

[0042] In view of the above, according to one embodiment, Figure 5 shows a flowchart of a method for determining the first statistical data and the second statistical data. In step S301, processing unit 140 determines the ROI corresponding to the overlapping blocks between at least one first frame (e.g., first frame FF) and at least one second frame (e.g., second frame SF). In step S302, processing unit 140 assigns a weight to each overlapping block in at least one first frame based on the alignment state relative to the ROI. This weighting mechanism is designed to take into account different degrees of overlap. For example, as shown in Figure 4(a), the weight of a block in the first frame FF that does not completely overlap with the second frame SF (i.e., a block that partially overlaps with the ROI) is less than 1 (i.e., 0 < weight < 1), and the specific weight value can be proportional to the percentage of overlap. The weight of a block in the first frame FF that completely overlaps with the second frame SF (i.e., completely within the ROI) is equal to 1. The weight of a block in the first frame FF that does not overlap with the second frame SF (i.e., completely outside the ROI) is equal to 0. Step S302 ensures that the contribution of partially overlapping areas to statistical data is proportional, thereby improving the accuracy of 3A consistency decision-making and control.

[0043] In step S303, processing unit 140 determines first statistical data based on the weights (determined in step S302) and image data of overlapping blocks in at least one first frame. In step S304, processing unit 140 determines second statistical data based on image data of overlapping blocks in at least one second frame, since the blocks in at least one second frame are entirely within the ROI, they are typically processed without additional weighting.

[0044] Furthermore, in condition (b) shown in Figure 4, it is assumed that the first frame FF is generated by a first camera module FCM (also used as a foreground viewing camera module) with a longer focal length (i.e., a narrower FOV), while the second frame SF is generated by a second camera module SCM with a shorter focal length (i.e., a wider FOV). Since the first camera module FCM is used as a foreground viewing camera module, determining the second statistics requires excluding or reducing the influence introduced by image data from certain regions of the second frame SF, which are located outside the FOV of the first frame FF. In view of the above, according to one embodiment, Figure 6 shows a flowchart of a method for determining the first and second statistics. In step S401, the processing unit 140 determines the ROI corresponding to the overlapping blocks between at least one first frame and at least one second frame. In step S402, the processing unit 140 assigns weights to each overlapping block in the frame of at least one second frame based on the alignment state relative to the ROI, wherein blocks completely within the ROI are assigned a weight of 1, blocks partially overlapping the ROI are assigned a weight between 0 and 1 proportional to their overlap percentage, and blocks completely outside the ROI are assigned a weight of 0. In step S403, processing unit 140 determines first statistical data based on image data of overlapping blocks of at least one first frame. In step S404, processing unit 140 determines second statistical data based on the weights of overlapping blocks of at least one second frame and image data.

[0045] In some embodiments, FOV alignment can be more than just simple rectangular overlap, because 3D points are projected onto frames of different camera modules via their respective matrices, resulting in FOV alignment between two frames across different camera modules potentially having arbitrary shapes, such as quadrilaterals. Furthermore, lens distortion is considered through matrix transformations to ensure accurate mapping between two frames from different camera modules. A mapping table can be used to record the proportion of each image patch falling within the ROI, allowing for accurate calculation of statistics even with partial overlap.

[0046] The aforementioned FOV alignment ensures that statistical data for camera modules with different focal lengths / FOVs are determined based on comparable scene areas, thereby contributing to accurate 3A consistency decision-making and control. The weighted method for wider FOV camera modules can compensate for partial overlap, while direct calculation for narrower FOV camera modules can maintain the integrity of their full coverage data.

[0047] Smooth Control

[0048] In the 3A consistency decision control mechanism, the processing unit 140 is also configured to implement a smooth control mechanism for adjustments across different frame rates between different camera modules. Specifically, in some embodiments, the first camera module FCM can act as a background main camera module and operate at a lower frame rate (e.g., 20, 10, or 5 FPS), while the second camera module SCM can act as a foreground viewing camera module and operate at a standard frame rate (e.g., 30 FPS).

[0049] Processing unit 140 is configured to determine a 3A target adjustment step size based on the frame rate of the first camera module FCM, where a lower FPS results in a larger adjustment step size to ensure responsiveness to changes in the first adjustment target of the first camera module FCM. Subsequently, processing unit 140 is configured to interpolate the determined 3A target adjustment step according to the ratio of the operating frame rate of the first camera module FCM to the operating frame rate of the second camera module SCM to obtain an intermediate 3A target adjustment step. In this way, by using intermediate steps to facilitate a more gradual adjustment towards the second adjustment target, a smoother transition in the live preview can be ensured. This method effectively balances the need for responsive adjustment with the requirement for a visually smooth transition in the live preview.

[0050] For example, when the first camera module FCM is running at 10 FPS and the second camera module SCM is running at 30 FPS, the processing unit 140 is configured to interpolate the 3A target adjustment steps of the first camera module FCM. Specifically, each 3A target adjustment step of the first camera module FCM is divided into three smaller intermediate 3A target adjustment steps of the second camera module SCM.

[0051] Furthermore, in some embodiments, unlike the 3A target adjustment step of the first camera module FCM, the processing unit 140 is allowed to independently determine the smooth adjustment direction of the intermediate 3A target adjustment step of the second camera module SCM. This solves potential problems such as obstructed field of view of the first camera module FCM or significant brightness differences between the FOV of the first camera module FCM and the second camera module SCM. By achieving this independent smooth direction determination, the 3A consistency decision and control mechanism of the present invention prioritizes user experience by aligning the 3A adjustment with the actual scene visible in the real-time preview, ensuring that the second camera module SCM (e.g., the foreground viewing camera module) can adapt to its immediate environmental conditions.

[0052] In summary, the 3A consistency decision and control mechanism of the present invention represents a significant advancement in mobile imaging technology. The present invention offers two advantages, addressing key challenges in modern mobile imaging systems. First, it demonstrates efficient multi-camera operation, maintaining low power consumption of a single camera module while simultaneously activating multiple camera modules. This feature ensures energy efficiency without compromising the accuracy of 3A consistency across all camera modules, thereby optimizing performance and power management in complex multi-camera module configurations. Second, the 3A consistency decision and control mechanism guarantees a seamless visual transition in real-time preview when switching between camera modules. This smooth transition maintains consistency in real-time preview by eliminating noticeable changes in exposure, color balance, or focus during camera module switching, thus significantly enhancing the user experience during photographic and video recording sessions.

[0053] Embodiments of the present invention can be implemented as apparatus, method, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, which are generally referred to herein as “modules” or “systems.” Furthermore, this embodiment can take the form of a computer program product embodied in any tangible medium, wherein computer-usable program code is embodied. In terms of hardware, the present invention can be implemented by applying any of the following techniques or related combinations: individual operating logic having logic gates capable of performing logical functions according to data signals, and a dedicated application-specific integrated circuit (ASIC), programmable gate array (PGA), or field-programmable gate array (FPGA) having appropriate combinational logic.

[0054] The flowcharts and block diagrams in the flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this embodiment. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware system or a combination of dedicated hardware and computer instructions that perform the specified functions or actions. These computer program instructions may be stored in a computer-readable medium that instructs a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means that implement the functions / actions specified in the flowchart and / or block diagram blocks.

[0055] Those skilled in the art will readily observe that many modifications and changes can be made to the apparatus and method while retaining the teachings of the present invention. Therefore, the above disclosure should be understood as being limited only by the boundaries and scope of the appended claims.

[0056] The subjects described herein sometimes show different components contained within or connected to other different components. It should be understood that the architectures depicted in this way are merely exemplary, and many other architectures can actually be implemented to achieve the same functionality. Conceptually, any arrangement of components that achieve the same functionality is effectively “associated” to achieve the desired functionality. Therefore, any two components combined in this document to obtain a particular function can be considered “associated” with each other to achieve the desired functionality, regardless of the architecture or intermediate components. Similarly, any two such associated components can also be considered “operably connected” or “operably coupled” with each other to achieve the desired functionality, and any two components that can be suchly associated can also be considered “operably coupled” with each other to achieve the desired functionality. Specific examples of “operably coupled” include, but are not limited to: components that are physically connectable and / or physically interact with each other, and / or components that are wirelessly interactive and / or wirelessly interact with each other, and / or components that are logically interactive and / or logically interactive with each other.

[0057] Furthermore, regarding the use of virtually any plural and / or singular terms in the text, those skilled in the art can convert plural to singular and / or singular to plural, provided that it is appropriate for the context and / or application.

[0058] Those skilled in the art will understand that, generally, the terms used herein, particularly those used in the appended claims (e.g., the subjects in the appended claims), are generally intended as “open-ended” terms (e.g., the term “comprising” should be interpreted as “comprising but not limited to”, the term “having” should be interpreted as “having at least”, the term “comprising” should be interpreted as “comprising but not limited to”, etc.). Those skilled in the art will also understand that if a specific number of the objects described in the claims is intended, such intention will be explicitly stated in the claims; in the absence of such a statement, such intention does not exist. For example, to aid understanding, the appended claims may include the use of introductory phrases such as “at least one” and “one or more” to describe the objects described in the claims. However, the use of such phrases should not be interpreted as limiting any patent application containing such a description of a claim subject to an invention containing only one such description, even if the same patent application contains the introductory phrases "one or more" or "at least one" and introductory articles such as "one (a)" or "one (an)" (e.g., "one (a)" and / or "one (an)" should generally be interpreted as meaning "at least one" or "one or more"); the same applies to the use of definite articles to describe claims. Furthermore, even if a specific number of claims is explicitly stated, those skilled in the art will recognize that such a statement should generally be interpreted as meaning at least the stated number (e.g., a statement of "two claims" without other modifiers generally means at least two claims, or two or more claims). Furthermore, when using idioms such as "at least one of A, B, and C," the structure is generally intended to convey the meaning of the idiom as understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" would include, but is not limited to, systems having a single A, a single B, a single C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). When using idioms such as "at least one of A, B, or C," the structure is generally intended to convey the meaning of the idiom as understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" would include, but is not limited to, systems having a single A, a single B, a single C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.).Those skilled in the art will further understand that, whether in the specification, the claims, or the drawings, virtually any extractive word and / or phrase representing two or more alternative terms should be understood to take into account the possibility of including one, any, or both of the terms. For example, the phrase “A or B” should be understood to include the possibility of including “A”, “B”, or “A and B”.

[0059] Although various methods, apparatuses, and systems have been used to describe and illustrate some exemplary techniques herein, it should be understood by those skilled in the art that various other modifications and equivalent substitutions can be made without departing from the claimed subject matter. Furthermore, many modifications can be made to adapt particular situations to the teachings of the claimed subject matter without departing from the central concept described herein. Therefore, it is intended that the claimed subject matter is not limited to the specific examples disclosed, and that such claimed subject matter may also include all implementations and their equivalents falling within the scope of the appended claims. The above descriptions are merely preferred embodiments of the present invention, and all equivalent variations and modifications made within the scope of the claims of this invention should be considered within the scope of this invention. [Simplified Explanation of the Diagram]

[0008] The accompanying drawings, which are included in and constitute a part of this specification, illustrate embodiments of the invention and, together with the specification, serve to explain the principles of the invention: Figure 1 illustrates an imaging system and corresponding electronic equipment according to an embodiment of the invention. Figure 2 illustrates a flowchart of a method for controlling an imaging system having multiple camera modules according to an embodiment of the invention. Figure 3 illustrates how adjustment targets for different camera modules are determined according to an embodiment of the invention. Figure 4 illustrates how FOV alignment is performed according to an embodiment of the invention. Figure 5 illustrates a flowchart of a method for determining first and second statistical data according to a first embodiment of the invention. Figure 6 illustrates a flowchart of a method for determining first and second statistical data according to a second embodiment of the invention.

Claims

1. An imaging system control method, the imaging system having multiple camera modules, the imaging system control method comprising: Operate the first camera module among the plurality of camera modules to generate at least one first frame; Operate the second camera module among the plurality of camera modules to generate at least one second frame; Generating first statistical data based on at least one first frame and generating second statistical data based on at least one second frame includes: calculating the first statistical data and the second statistical data using the image signal processing circuit of the imaging system; Alternatively, the first statistical data can be calculated using the first statistical circuit of the first camera module, and the second statistical data can be calculated using the second statistical circuit of the second camera module; and a first adjustment target for the imaging control parameters of the first camera module can be determined based on the at least the first statistical data.

2. The imaging system control method as described in claim 1 further includes: By determining a second adjustment target for the imaging control parameters of the second camera module based on the first adjustment target, the first statistical data, and the second statistical data, inter-module synchronization control is performed on at least one camera module.

3. The imaging system control method as described in claim 1 further includes: The first adjustment target for the imaging control parameters of the first camera module is determined based on the first statistical data and the second statistical data.

4. The imaging system control method as described in claim 1, wherein, The imaging control parameters include at least one of the following: automatic white balance control parameters, automatic exposure control parameters, and automatic focus control parameters.

5. The imaging system control method as described in claim 2 further includes: In response to changes in scenario conditions, the second adjustment target is redefined based on the second statistical data.

6. The imaging system control method according to claim 5 further includes: The first adjustment target is updated based on the redefined second adjustment target, the first statistical data, and the second statistical data.

7. The imaging system control method as described in claim 1, wherein the step of generating the at least one first frame and the at least one second frame includes: The at least one first frame is generated by operating the first camera module at a first frame rate; And the at least one second frame is generated by operating the second camera module at a second frame rate; wherein the second frame rate is greater than the first frame rate.

8. The imaging system control method as claimed in claim 1, wherein the focal length of the lens of the first camera module is longer or shorter than the focal length of the second camera module.

9. The imaging system control method as described in claim 1, further comprising: Determine the region of interest corresponding to the overlapping block between the at least one first frame and the at least one second frame; Weights are assigned to each overlapping block in the at least one first frame based on the alignment state relative to the region of interest; first statistics are determined based on the weights of the overlapping blocks in the at least one first frame and image data; and second statistics are determined based on image data of the overlapping blocks in at least one of the at least one second frame.

10. The imaging system control method as described in claim 1, further comprising: Determine the region of interest corresponding to the overlapping block between the at least one first frame and the at least one second frame; Weights are assigned to each overlapping block in the at least one second frame based on the alignment state relative to the region of interest; first statistics are determined based on image data of the overlapping blocks in the at least one first frame; and second statistics are determined based on the weights and image data of the overlapping blocks in the at least one second frame.

11. The imaging system control method as claimed in claim 1, wherein the steps of generating the at least one first frame and generating the at least one second frame include: The at least one first frame is generated by operating the first camera module in a non-high dynamic range capture mode; And the at least one second frame is generated by operating the second camera module in high dynamic range capture mode.

12. The imaging system control method as described in claim 11 further includes: The first adjustment target is determined solely based on the second statistical data, wherein the second statistical data comes from the at least one second frame generated by the second camera module operating in the high dynamic range capture mode; inter-module synchronization control is performed by determining a second adjustment target for the imaging control parameters of the second camera module based on the first adjustment target, the first statistical data, and the second statistical data; and the second camera module is operated in the high dynamic range capture mode based on the second adjustment target.

13. The imaging system control method as described in claim 11 further includes: Based on the second statistical data, the control parameters of the high dynamic range capture mode of the second camera module are determined; Inter-module synchronization control is performed by determining a second adjustment target for the imaging control parameters of the second camera module based on the first adjustment target, the first statistical data, and the second statistical data; and the second camera module is operated in the high dynamic range capture mode based on the second adjustment target and the control parameters of the high dynamic range capture mode.

14. The imaging system control method as described in claim 1, further comprising: Intermediate steps are used to perform a gradual adjustment process on the second camera module to ensure a smoother transition to the second adjustment target.

15. A computer-readable medium storing a plurality of instructions, responsive to execution on a computing device, causing the computing device to operate in such a way as to: operate a first camera module of a plurality of camera modules to generate at least one first frame; operate a second camera module of the plurality of camera modules to generate at least one second frame; Generating first statistical data based on at least one first frame and generating second statistical data based on at least one second frame includes: The first statistical data and the second statistical data are calculated using the image signal processing circuit of the imaging system; Alternatively, the first statistical data can be calculated using the first statistical circuit of the first camera module, and the second statistical data can be calculated using the second statistical circuit of the second camera module; and a first adjustment target for the imaging control parameters of the first camera module can be determined based on the at least the first statistical data.

16. An apparatus for controlling an imaging system, comprising: At least one memory module configured to store multiple instructions; The system includes at least one processor communicatively coupled to the at least one memory, configured to execute a plurality of instructions to perform the following operations: generating first statistics based on at least one first frame obtained by operating a first camera module of a plurality of camera modules of an imaging system; generating second statistics based on at least one second frame obtained by operating a second camera module of the plurality of camera modules, wherein: the first statistics and the second statistics are calculated using image signal processing circuitry of the imaging system; or the first statistics are calculated using a first statistical circuitry of the first camera module and the second statistics are calculated using a second statistical circuitry of the second camera module; and a first adjustment target for imaging control parameters of the first camera module is determined based on at least the first statistics.

Images