Image processing method and system based on four-camera cross-focal-length continuous zooming fusion

Abstract

The invention relates to the technical field of multi-camera image processing and computational photography, and discloses an image processing method and system based on four-camera cross-focal-length continuous zooming fusion. The image processing method comprises the steps of system initialization, image acquisition, zoom routing, automatic ROI extraction and tracking, field-of-view cutting and geometric alignment, image fusion and binocular depth recognition. Through systematized multi-camera collaborative design, an innovative mechanism is introduced in key links such as zoom routing, geometric alignment, image fusion and depth recognition, smooth zoom, space consistency, detail fidelity, power consumption optimization and high-quality 3D perception are realized, the imaging quality is improved through the effects, the application scene is expanded, and the application prospect is wide. And a comprehensive solution is provided for mobile photography, AR and intelligent visual systems.

Description

Technical Field

[0001] This invention relates to the field of multi-camera image processing and computational photography technology, specifically to an image processing method and system based on four-camera cross-focal-length continuous zoom fusion. Background Technology

[0002] With the widespread use of multi-camera systems in mobile devices, users have placed higher demands on zoom continuity, image quality consistency, and 3D perception capabilities. Traditional single-camera systems achieve magnification through digital zoom, which leads to a decrease in resolution and loss of detail. While existing multi-camera systems can provide different focal lengths, the following shortcomings still exist in the lens switching and fusion stages: (1) Perspective jump and geometric discontinuity: Differences in optical center, field of view and imaging style of different lenses can lead to sudden changes in image position and inconsistent color style when switching directly, and users can perceive obvious "shaking".

[0003] (2) Unsmooth zoom transition: Relying on low-focal-length lenses to crop "bridges" will result in a significant drop in image quality in the transition area, and the critical point will still inevitably experience a jump.

[0004] (3) Inconsistent depth of field: The depth of field varies greatly at different focal lengths, which can easily lead to inconsistent sharpness or sudden changes in sharpness.

[0005] (4) Difficulties in binocular depth acquisition: The short baseline of heterogeneous dual cameras, inconsistent field of view scale, and insufficient synchronization make epipolar correction and matching calculations complex; the traditional block matching SGM has a large computational load, and the computing power and power consumption of deep learning methods are limited.

[0006] To address the aforementioned issues, those skilled in the art have proposed numerous solutions, such as multi-camera zoom fusion technology, which achieves continuous zoom through multi-camera switching and fusion. However, there is still room for improvement in areas such as parallax compensation, switching lag and transition smoothness, and ROI consistency. Another example is cross-lens fusion and AI enhancement solutions from the industry, which attempt to alleviate uneven sharpness and ghosting using neural networks or multi-frame detail transfer. However, these solutions are complex to implement, limited in real-time performance and power consumption, and cannot fundamentally solve physical parallax. Furthermore, under conditions of small baselines and heterogeneous fields of view, depth accuracy, robustness, and real-time performance face challenges, often requiring auxiliary hardware such as ToF or structured light, which limits cost and applicable scenarios.

[0007] In summary, existing technologies still have significant shortcomings in geometric alignment, depth-of-field consistency, and high-quality depth, directly impacting user experience and subsequent computational photography capabilities. This invention proposes a systematic solution to these problems. Summary of the Invention

[0008] The present invention aims to at least partially solve one of the aforementioned technical problems in the prior art.

[0009] Therefore, the present invention provides an image processing method based on four cameras with cross-focal length continuous zoom fusion, comprising the following steps: System initialization steps: Load the intrinsic parameters, distortion parameters, extrinsic parameters and color parameters of the four cameras, and preset the zoom range and fusion corridor parameters; Image acquisition steps: Simultaneously acquire image data from the four cameras; Zoom routing steps: Determine the zoom range based on the current zoom ratio set by the user; if it is in the single-camera working range, select a single dominant camera; if it is in the fusion corridor range, calculate the fusion weights of the two participating cameras according to the preset weight function. Automatic ROI extraction and tracking steps: On the image of the dominant camera or a camera with high weight, automatically extract and track ROIs by calculating the salient regions of the image; Field of view cropping and geometric alignment steps: Based on the current zoom ratio and the automatic ROI, calculate the cropping area required by each participating camera to achieve an equivalent field of view, and perform cropping and resampling to achieve geometric alignment of multiple images; Image fusion steps: Within the fusion corridor, the two geometrically aligned images are fused pixel-level according to the fusion weights, and the white balance of the non-dominant camera is gradually adjusted to approximate that of the dominant camera; within the single-camera working area, the through image of that camera is output. Binocular depth recognition steps: Based on the completed geometrically aligned image pairs, stereo matching is performed using multi-baseline combination to calculate the depth information of the scene.

[0010] Preferably, in the zoom routing step, the fusion corridor is set in the overlapping interval between the zoom ranges of two adjacent cameras; the weighting function is an S-shaped curve, and a hysteresis mechanism and time-dimensional smoothing filtering are introduced to prevent frequent switching or jitter at the corridor boundary.

[0011] Preferably, the width of the fusion corridor is set to be no less than 1.35 times the equivalent focal length span corresponding to the minimum overlapping field of view of the two cameras in this interval.

[0012] Preferably, the automatic ROI extraction and tracking step further includes a lag update mechanism, which satisfies at least one of the following conditions: ROI update is triggered only when the change in the current zoom magnification exceeds a preset threshold, or when the time since the last update exceeds a time threshold. Set a dead zone around the center of the current ROI; ROI drift within this dead zone will not trigger an update. Changes in the center or scale of the ROI will only take effect after a preset dwell time has been maintained. The location of the ROI under rapid motion is predicted and compensated by combining data from the inertial measurement unit.

[0013] Preferably, in the field-of-view cropping and geometric alignment steps, a unified world coordinate system or a reference camera coordinate system is used, and the extrinsic parameters are used to perform micro-compensation on the cropping area to correct parallax caused by different camera optical centers.

[0014] Preferably, in the binocular depth recognition step, the image pairs used come from a combination of cameras with an equivalent field of view, and the multi-baseline combination includes at least one of an ultra-wide-angle-wide-angle combination, a wide-angle-telephoto combination, and a telephoto-periscope telephoto combination.

[0015] Preferably, the stereo matching employs a hybrid method that integrates a semi-global matching algorithm and a lightweight neural network model.

[0016] The present invention also provides an image processing system based on four cameras with cross-focal length continuous zoom fusion, for implementing the above-mentioned image processing method, the image processing system comprising: The quad-camera module includes four optical lenses with different focal lengths and overlapping zoom ranges, as well as an image sensor. The memory is used to store the calibration parameters, fusion corridor parameters, and computer programs of the four-camera module. The processor is configured to execute the computer program to operate the following functional modules: An initialization module is used to load the calibration parameters and configure system parameters; The image acquisition and synchronization module is used to control the four camera modules to acquire image data synchronously; The zoom routing module is used to determine the dominant camera and / or cameras participating in fusion and their fusion weights based on the current zoom level. An automatic ROI processing module is used to extract and track regions of interest on the dominant camera image; The geometric alignment and cropping module is used to perform field-of-view cropping and geometric alignment on the camera images participating in the fusion based on the ROI and the current zoom ratio. An image fusion module is used to perform weighted fusion of aligned images within a fusion corridor and to handle photometric consistency; and The depth calculation module is used to perform stereo matching based on aligned image pairs and output a depth map.

[0017] Preferably, the zoom routing module is configured to: calculate the fusion weight using an S-shaped weight curve within the fusion corridor interval, and introduce a hysteresis mechanism and time-dimensional smoothing filtering.

[0018] Preferably, the automatic ROI processing module is configured to perform a lag update mechanism, including at least one of the following: triggering conditions based on zoom change or time, dead zone setting, dwell time judgment, and motion prediction combined with IMU data.

[0019] Compared with existing technologies, the image processing method based on four-camera cross-focal-length continuous zoom fusion provided by this invention introduces innovative mechanisms in key aspects such as zoom routing, geometric alignment, image fusion, and depth recognition through a systematic multi-camera collaborative design. This achieves smooth zoom, spatial consistency, detail fidelity, power optimization, and high-quality 3D perception. These effects not only improve image quality but also expand application scenarios, providing a comprehensive solution for mobile photography, AR, and intelligent vision systems.

[0020] Other features and advantages of the present invention will be described in detail in the following specific embodiments. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an image processing method based on continuous zoom fusion across four cameras provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an image processing system based on four cameras with cross-focal length continuous zoom fusion, provided by an embodiment of the present invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives, and effects of this invention readily understandable, the invention is further illustrated below with reference to specific embodiments. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the invention.

[0023] Combination Figure 1 As shown, this embodiment of the invention provides an image processing method based on continuous zoom fusion across four focal lengths using four cameras. This image processing method is based on continuous zoom fusion across four focal lengths using four cameras and binocular depth recognition. Specifically, it includes system initialization steps, image acquisition steps, zoom routing steps, automatic ROI extraction and tracking steps, field of view cropping and geometric alignment steps, image fusion steps, and binocular depth recognition steps.

[0024] The following provides a detailed explanation of each step.

[0025] System initialization steps: Load the intrinsic parameters, distortion parameters, extrinsic parameters and color parameters of the four cameras, and preset the zoom range and fusion corridor parameters; Image acquisition steps: Simultaneously acquire image data from the four cameras; Zoom routing steps: Determine the zoom range based on the current zoom ratio set by the user; if it is in the single-camera working range, select a single dominant camera; if it is in the fusion corridor range, calculate the fusion weights of the two participating cameras according to the preset weight function. Automatic ROI extraction and tracking steps: On the image of the dominant camera or a camera with high weight, automatically extract and track ROIs by calculating the salient regions of the image; In this step, the system automatically extracts the Region of Interest (ROI) and tracks its position by calculating the salient regions of the image. Based on the current zoom ratio and ROI, the cropping areas of each participating camera are calculated to achieve an equivalent field of view, ensuring that the subject maintains a stable position and size in the output image. Thus, during zooming, the subject remains centered and proportionally consistent, avoiding the "jumping" problem that occurs when switching between traditional multi-camera systems.

[0026] Field of view cropping and geometric alignment steps: Based on the current zoom ratio and the automatic ROI, calculate the cropping area required by each participating camera to achieve an equivalent field of view, and perform cropping and resampling to achieve geometric alignment of multiple images; Image fusion steps: Within the fusion corridor, the two geometrically aligned images are fused pixel-level according to the fusion weights, and the white balance of the non-dominant camera is gradually adjusted to approximate that of the dominant camera; within the single-camera working area, the through image of that camera is output.

[0027] In this step, within the fusion corridor, the system weightedly fuses images from two cameras, such as a wide-angle camera and a telephoto camera. The optical magnification details provided by the telephoto camera are gradually incorporated, reducing blur and noise caused by reliance on digital magnification. For example, within the 3x-5x zoom range, the optical advantages of the telephoto camera are fully utilized, avoiding the resolution loss associated with pure digital zoom. Throughout the zoom range, the system dynamically selects the optimal camera combination to ensure a smooth transition of texture details from ultra-wide-angle to periscope telephoto. This is achieved through geometric alignment and resampling, ensuring the output image maintains high clarity at all magnifications. By incorporating the above steps, the system timely introduces telephoto optical details in the mid-to-high magnification range, reducing reliance on digital magnification in the transition zone. This results in a more balanced resolution and texture representation across the entire zoom range, addressing the pain point of blurring at intermediate magnifications when zooming on mobile devices. It provides high-quality zoom capabilities similar to SLR cameras, making it particularly suitable for landscape photography and long-distance shooting.

[0028] Furthermore, within the fusion corridor, when two images coexist, the system gradually adjusts the white balance, exposure, and color parameters of the non-dominant camera to match those of the dominant camera. For example, through color mapping curves or matrix transformations, a smooth color transition is achieved, avoiding color differences or brightness jumps at the fusion boundary. Moreover, the weighting function is synchronized with the white balance adjustment, ensuring that the photometric characteristics of the two images are completely consistent at the end of the fusion corridor. This improves video stream consistency and avoids the color flickering problem commonly encountered during traditional multi-camera switching. Through this technical solution incorporating the above steps, the two images coexist and gradually converge during the transition period, avoiding white balance conflicts. Therefore, it enhances the professionalism of video recording and is suitable for fields with high color consistency requirements, such as live streaming and film production.

[0029] In the image processing method provided by this invention, a through image is output during the single-camera working zone, and both images are processed only in the fusion corridor. Thus, the system activates only a single camera most of the time, reducing power consumption of the image sensor, ISP (Image Signal Processor), and memory. Two-way fusion is only activated within the fusion corridor zone, but the fusion algorithm is computationally lightweight, avoiding the high load of complex multi-camera processing. Furthermore, through intelligent scheduling of zoom routing, the system achieves continuous zoom and depth recognition functions with the same hardware resources, without requiring additional dedicated chips, thus reducing costs and making it suitable for mobile devices and IoT devices. In other words, based on the image processing method provided by this invention, the system is in a single-output state most of the time, with only two images fused during the corridor period, achieving higher visual quality and functionality with the same hardware. The above design of this invention aligns with the trend of green computing, extends battery life, and improves user experience, showing broad application prospects in smartphones, drones, and other scenarios.

[0030] Binocular depth recognition steps: Based on the completed geometrically aligned image pairs, stereo matching is performed using multi-baseline combination to calculate the depth information of the scene.

[0031] It is understandable that cameras with different focal lengths have different depth-of-field characteristics; for example, telephoto cameras have a shallow depth of field, while wide-angle cameras have a deep depth of field. The image processing method provided by this invention dynamically adjusts the focus position during zooming through linked focus control and uses a fusion algorithm to smoothly transition the depth-of-field effect. For example, in the fusion area, the algorithm can change the degree of blur through depth-sensing blending gradients, making the background blur change naturally.

[0032] Automatic ROI tracking ensures the subject remains in sharp focus, while fusion weight adjustment prevents focus loss due to abrupt changes in depth of field. This simulates the physical characteristics of optical zoom, enhancing the visual experience in portrait photography and other scenarios. In other words, the image processing method provided by this invention can seamlessly integrate focus and algorithm transitions to bridge depth-of-field differences, resulting in smooth changes in both bokeh and focus sharpness for a natural look.

[0033] In the zoom routing step, the fusion corridor is set in the overlapping area between the zoom ranges of two adjacent cameras; the weighting function is an S-shaped curve, and a hysteresis mechanism and time-dimensional smoothing filter are introduced to prevent frequent switching or jitter at the corridor boundary.

[0034] In embodiments of the present invention, the hysteresis mechanism prevents frequent switching at corridor boundaries due to minute magnification changes by setting different switching thresholds. Time-dimension smoothing filtering smooths the weight values, further suppressing instantaneous fluctuations caused by user operation jitter or environmental changes, effectively improving the stability of video recording, especially in motion scenes, resulting in more natural and smooth zoom transitions.

[0035] Furthermore, the width of the fusion corridor is set to be no less than 1.35 times the equivalent focal length span corresponding to the minimum overlapping field of view of the two cameras in this interval.

[0036] In embodiments of the present invention, the width of the fusion corridor is set to be no less than 1.35 times the equivalent focal length span of the minimum overlapping field of view of the two cameras. This ensures sufficient buffer space for the zoom ratio within the transition range. Understandably, when transitioning from a wide-angle camera to a telephoto camera, the Logistic weighting function smoothly changes the fusion weights from 0 to 1, avoiding image jumps. This design simulates the continuity of optical zoom, reducing the "step" phenomenon commonly found in traditional digital zoom.

[0037] In other words, according to the technical solution provided by the present invention, the long corridor and Logistic weight provide a wide buffer in the magnification domain; the hysteresis mechanism and the smoothing filter in the time dimension suppress jitter in the time domain, realizing a seamless zoom experience in mobile photography and video recording, reducing the discomfort caused by jitter or switching, and improving professional-grade imaging quality.

[0038] The automatic ROI extraction and tracking step further includes a lag update mechanism, which satisfies at least one of the following conditions: ROI update is triggered only when the change in the current zoom magnification exceeds a preset threshold, or when the time since the last update exceeds a time threshold. Set a dead zone around the center of the current ROI; ROI drift within this dead zone will not trigger an update. Changes in the center or scale of the ROI will only take effect after a preset dwell time has been maintained. The location of the ROI under rapid motion is predicted and compensated by combining data from the inertial measurement unit.

[0039] In the field-of-view clipping and geometric alignment steps, a unified world coordinate system or a reference camera coordinate system is used, and the extrinsic parameters are used to perform micro-compensation on the clipping area to correct parallax caused by different camera optical centers.

[0040] In embodiments of the present invention, micro-compensation is performed on the cropped area using camera extrinsics to correct parallax errors caused by different physical positions of the cameras. This significantly reduces misalignment and ghosting artifacts during multi-image fusion, especially in edge regions.

[0041] In conjunction with the foregoing, the technical solution provided by this invention ensures a stable position of the subject in the output image through unified ROI cropping and calibration offset compensation, effectively reducing misalignment and ghosting artifacts. This technology improves the geometric accuracy of image fusion and is suitable for scenarios such as video conferencing and security monitoring, where subject tracking and spatial consistency are key requirements.

[0042] In the binocular depth recognition step, the image pairs used come from a combination of cameras with an equivalent field of view. The multi-baseline combination includes at least one of ultra-wide-angle-wide-angle combination, wide-angle-telephoto combination, and telephoto-periscope telephoto combination. According to the method provided by the present invention, through a geometric alignment step, the images from each camera are cropped and resampled to a unified scene, eliminating parallax errors and providing ideal input for stereo matching, thereby improving the spatial consistency of the depth map. The system utilizes combinations such as ultra-wide-angle-wide-angle and wide-angle-telephoto to provide multiple baseline lengths, where short baselines are suitable for near scenes and long baselines are suitable for distant scenes, thereby optimizing depth accuracy within different distance ranges. For example, the ultra-wide-angle-wide-angle combination is suitable for indoor scenes, while the telephoto-periscope telephoto combination is suitable for distant depth calculation.

[0043] Preferably, the stereo matching employs a hybrid method that combines a semi-global matching algorithm with a lightweight neural network model. In this invention, by setting the stereo matching to use a hybrid method that combines a semi-global matching algorithm with a lightweight neural network model, accuracy and efficiency are balanced. The semi-global matching algorithm provides robust dense matching, while the lightweight neural network handles texture-deficient regions, significantly reducing depth holes and noise.

[0044] In this invention, depth information can be used to improve image fusion. For example, depth perception weights can be adjusted to avoid foreground / background fusion errors, or ROI tracking can be dynamically optimized based on depth maps to form a closed-loop optimization. The method provided by this invention can provide high-precision 3D perception capabilities for AR / VR, autonomous driving, and robot navigation, while improving 2D imaging quality through depth feedback, achieving multi-functional integration.

[0045] In other words, based on the image processing method provided by this invention, the depth accuracy and applicability can be significantly improved by equivalent field-of-view matching and multi-baseline combination, providing stable input for AR, ranging and reconstruction, and can also feed back into the fusion strategy to further improve imaging consistency.

[0046] In embodiments of this invention, an A / B / C / D quad-camera system is constructed. The four cameras undergo unified calibration (pinhole / fisheye models are assigned according to lens type), storing intrinsic parameters K, distortion D, and extrinsic parameters {R, t}, and generating an equivalent field of view. A mapping table is used for magnification-ROI-cropping coordination. Special lenses such as fisheye / periscope lenses are corrected according to OpenCV's fisheye model or a corresponding approximate model. From wide-angle to super-telephoto, the focal length gradient increases progressively, forming a sufficiently overlapping coverage area. Combining sensor resolution and output resolution, a safe zoom limit is planned for each camera, allocating a "dominant range" to prevent image quality degradation caused by a single lens operating beyond its range. The system completes intrinsic and extrinsic parameter calibration and records relative poses for subsequent field-of-view alignment and ROI compensation.

[0047] A "fusion corridor" with minimum width constraints is established between adjacent shots, meaning that both paths participate simultaneously within a relatively wide magnification range. The weights use a Logistic (S-shaped) curve to smoothly transition from the previous shot to the next shot, and a hysteresis mechanism, time smoothing, and rate limit are introduced. The hysteresis mechanism adds a threshold at the corridor boundary to prevent back-and-forth jitter and frequent switching. The time smoothing and rate limit are a first-order low-pass filter for weights plus a maximum rate of change constraint to ensure smoothness in the time dimension.

[0048] The equivalent focal length is z∈[13,140]mm, and the minimum corridor width Wmin is set between adjacent lenses. The estimated data is as follows: A→B corridor: z∈[16,22] (width 6mm); Corridor A→C: z∈[50,72] (width 22mm); Corridor C→D: z∈[110,130] (width 20mm).

[0049] It should be noted that by setting the corridor width to be greater than or equal to 1.35 times the equivalent span of the minimum overlap FOV of the two lenses in that interval, it is ensured that the two channels can output in parallel over a sufficiently long interval, thus avoiding image quality jumps at critical points.

[0050] During field-of-view cropping and geometric alignment, a unified output coordinate system is defined. The cropping scaling factor required for each lens to achieve an equivalent field of view is calculated based on the current magnification. Cropping is then performed around a unified Region of Interest (ROI) on the sensor image, and the image is resampled to the output size. Combined with calibration offset for ROI micro-compensation and lens distortion correction, spatial overlap of the subject area is maximized. In this invention, subject alignment is achieved using "equivalent focal length matching + unified ROI cropping," significantly reducing ghosting and drift.

[0051] During automatic ROI extraction and target tracking, a multi-scale pyramid is constructed on the main road image, and the gradient magnitude "centroid" is calculated as the saliency center. The ROI is obtained by multi-level weighted fusion. The ROI is updated with adaptive step size and lag according to the magnification, and dead zone and dwell time are set to avoid jitter caused by small noise.

[0052] Specifically, a multi-scale pyramid is constructed on the main path (shots with a weight ≥ 0.5), gradient magnitude / saliency map is calculated, and the weighted centroid is used as the ROI. O It can also be used to obtain the ROI by integrating target detection. fuse .

[0053] Delayed updates rely on the following technical solutions: Magnification threshold: |Δz|≥Δz th (Recommended equivalent focal length change of 2.0 mm) or time threshold t≥t th (It is recommended to update the center only after 120 ms) Dead zone: Set a dead zone with a radius of 6-8% (relative to the shorter side) centered on the current ROI center. Small drifts will not trigger updates. Residence: Central or scale changes must continuously satisfy ≥T dwell (It is recommended that it take 240-360 ms to take effect;) IMU assistance: When the angular velocity is greater than the threshold (e.g., 20° / s), predict the ROI offset for one frame and suppress lag during fast moves.

[0054] All parameters above are engineering specifications and can be reproduced at 30 fps.

[0055] Regarding the fusion implementation, in this embodiment of the invention, pixel-level weighting (floating-point superposition followed by quantization) is performed on the cropped images of the participating shots. Normally, only two paths are fused to control computing power and power consumption. During the transition period, exposure is allowed to converge slowly to reduce style abrupt changes. Based on the dominant shot, the white balance of the non-dominant shots gradually converges to that of the dominant shot frame by frame.

[0056] Regarding the software architecture and thread scheduling, in the embodiments of the present invention, acquisition threads are configured for the four cameras respectively, and the main processing thread performs routing, ROI, cropping and fusion, adopting a producer-consumer model and timestamp alignment; and providing interactive interfaces (magnification control, automatic ROI switch, ROI reset, etc.).

[0057] In a specific embodiment of the present invention, the quad-camera system consists of: A (13-16 mm, ultra-wide) / B (30-40 mm, wide-angle) / C (60-70 mm, telephoto) / D (90-100 mm, periscope).

[0058] Connectivity: Four USB ports. The host (SoC / embedded) side adopts a producer-consumer structure with 4 acquisition threads, a main processing line, and a UI / control thread.

[0059] Calibration files include: four intrinsic / distortion and extrinsic parameters (with A as a reference), and three sets of binocular tuning matrices (AB, AC, CD); the 3D-printed monocular frame is directly connected to the computer, and the zoom range and preview results are adjusted via the display screen.

[0060] Operating platform hardware specifications: 1280×720@30 fps preview and processing; end-to-end latency ≤40 ms.

[0061] In embodiments of the present invention, the software process and key steps include: (1) Initialization: Load the calibration and color parameters of the four cameras; set the zoom range, three fusion corridors (AB, BC, CD), hysteresis threshold and upper limit of weight change rate.

[0062] (2) Acquisition and synchronization: Four-way parallel acquisition.

[0063] (3) Zoom routing: Determine whether it is in a single-path zone or a certain fusion corridor based on the current magnification; calculate the weights of the two paths according to the smooth curve in the corridor, and prevent back-and-forth jitter by limiting the rate of change of hysteresis.

[0064] (4) Automatic ROI: The gradient centroid method is used to find the center of the main subject in the main screen; avoid ROI jumping caused by micro-shaking; IMU prediction is used for smooth following during fast movement.

[0065] (5) Alignment and cropping: Make the perspectives of the two participating cameras equivalent, crop around a unified ROI and resample to a unified output size; perform basic distortion and micro pose compensation to reduce ghosting and misalignment.

[0066] (6) Light alignment: Use the dominant shot as the white balance and gradually converge from the shot frame by frame to avoid sudden changes in brightness or color.

[0067] (7) Pixel fusion: In the corridor: the two paths are weighted and pixel-level fusion is performed, followed by light sharpening and edge protection; outside the corridor: single path is straight through.

[0068] In an embodiment of the present invention, the thread and real-time budget are as follows: Based on a target of 30fps, capture ×4: 720p RAW / YUV (NV12) streams from each channel → ISP basic 3A; The main processing is as follows: 1) Ratio routing & corridor determination (<0.1 ms) 2) Auto-ROI (multi-scale gradient + detection fusion, 1.0-1.8 ms) 3) Distortion removal / equivalent scaling / clipping (LUT + NEON, 2.5-4.0 ms / channel) 4) Blend and Sharpen (1.5-2.0 ms) Total budget: 28-34 ms per frame (hardware dependent), to meet 30 fps real-time output requirements.

[0069] Output: Basic real-time configuration (default): Preview = 1280×720 @ 30 fps; end-to-end latency ≤ 40 ms (including acquisition / scheduling / alignment / fusion / display).

[0070] Enhanced configuration (optional): Preview = 1920×1080 @ 30 fps.

[0071] Photo and video recording: Video up to 4K@30 fps; Photos output at the main camera's native resolution.

[0072] like Figure 2 As shown, the present invention also provides an image processing system based on continuous zoom fusion across four focal lengths using four cameras. This image processing system is used to implement the above-mentioned image processing method. The image processing system includes a four-camera module, a memory, a processor, an initialization module, an image acquisition and synchronization module, a zoom routing module, an automatic ROI processing module, a geometric alignment and cropping module, an image fusion module, and a depth calculation module.

[0073] The four-camera module includes four optical lenses with different focal lengths and overlapping zoom ranges, as well as an image sensor. A memory stores calibration parameters, fusion corridor parameters, and a computer program for the four-camera module. A processor is configured to execute the computer program to operate an initialization module, an image acquisition and synchronization module, a zoom routing module, an automatic ROI processing module, a geometric alignment and cropping module, an image fusion module, and a depth calculation module. Specifically, the initialization module loads the calibration parameters and configures system parameters; the image acquisition and synchronization module controls the four-camera module to synchronously acquire image data; the zoom routing module determines the dominant camera and / or participating cameras and their fusion weights based on the current zoom ratio; the automatic ROI processing module extracts and tracks the region of interest (ROI) in the dominant camera image; the geometric alignment and cropping module performs field-of-view cropping and geometric alignment on the participating camera images based on the ROI and the current zoom ratio; the image fusion module performs weighted fusion of the aligned images within the fusion corridor and processes photometric consistency; and the depth calculation module performs stereo matching based on the aligned image pairs and outputs a depth map.

[0074] The zoom routing module is configured to: calculate the fusion weight using an S-shaped weight curve within the fusion corridor interval, and introduce a hysteresis mechanism and time-dimensional smoothing filtering.

[0075] The automatic ROI processing module is configured to perform a lag update mechanism, including at least one of the following: triggering conditions based on zoom change or time, dead zone setting, dwell time judgment, and motion prediction combined with IMU data.

[0076] It should also be noted that the term "comprising," or any other variation thereof, is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0077] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. An image processing method based on four cameras and continuous zoom fusion across focal lengths, characterized in that, Includes the following steps: System initialization steps: Load the intrinsic parameters, distortion parameters, extrinsic parameters and color parameters of the four cameras, and preset the zoom range and fusion corridor parameters; Image acquisition steps: Simultaneously acquire image data from the four cameras; Zoom routing steps: Determine the zoom range based on the current zoom ratio set by the user; if it is in the single-camera working range, select a single dominant camera; if it is in the fusion corridor range, calculate the fusion weights of the two participating cameras according to the preset weight function. Automatic ROI extraction and tracking steps: On the image of the dominant camera or a camera with high weight, automatically extract and track ROIs by calculating the salient regions of the image; Field of view cropping and geometric alignment steps: Based on the current zoom ratio and the automatic ROI, calculate the cropping area required by each participating camera to achieve an equivalent field of view, and perform cropping and resampling to achieve geometric alignment of multiple images; Image fusion steps: Within the fusion corridor, the two geometrically aligned images are fused pixel-level according to the fusion weights, and the white balance of the non-dominant camera is gradually adjusted to approximate that of the dominant camera; within the single-camera working area, the through image of that camera is output. Binocular depth recognition steps: Based on the completed geometrically aligned image pairs, stereo matching is performed using multi-baseline combination to calculate the depth information of the scene.

2. The image processing method based on four-camera cross-focal-length continuous zoom fusion according to claim 1, characterized in that, In the zoom routing step, the fusion corridor is set in the overlapping area between the zoom ranges of two adjacent cameras; The weighting function is an S-shaped curve, and a hysteresis mechanism and time-dimensional smoothing filter are introduced to prevent frequent switching or jitter at the corridor boundary.

3. The image processing method based on four-camera cross-focal-length continuous zoom fusion according to claim 2, characterized in that, The width of the fusion corridor is set to be no less than 1.35 times the equivalent focal length span corresponding to the minimum overlapping field of view of the two cameras in that interval.

4. The image processing method based on four-camera cross-focal-length continuous zoom fusion according to claim 1, characterized in that, The automatic ROI extraction and tracking step further includes a lag update mechanism, which satisfies at least one of the following conditions: ROI update is triggered only when the change in the current zoom magnification exceeds a preset threshold, or when the time since the last update exceeds a time threshold. Set a dead zone around the center of the current ROI; ROI drift within this dead zone will not trigger an update. Changes in the center or scale of the ROI will only take effect after a preset dwell time has been maintained. The location of the ROI under rapid motion is predicted and compensated by combining data from the inertial measurement unit.

5. The image processing method based on four-camera cross-focal-length continuous zoom fusion according to claim 1, characterized in that, In the field-of-view cropping and geometric alignment steps, a unified world coordinate system or a reference camera coordinate system is used, and the extrinsic parameters are used to perform micro-compensation on the cropping area to correct parallax caused by different camera optical centers.

6. The image processing method based on four-camera cross-focal-length continuous zoom fusion according to claim 1, characterized in that, In the binocular depth recognition step, the image pairs used come from a combination of cameras with an equivalent field of view, and the multi-baseline combination includes at least one of an ultra-wide-angle-wide-angle combination, a wide-angle-telephoto combination, and a telephoto-periscope telephoto combination.

7. The image processing method based on four-camera cross-focal-length continuous zoom fusion according to claim 1, characterized in that, The stereo matching employs a hybrid approach that combines a semi-global matching algorithm with a lightweight neural network model.

8. An image processing system based on four cameras and continuous zoom fusion across focal lengths, characterized in that, For implementing the image processing method as described in any one of claims 1-7, the image processing system comprises: The quad-camera module includes four optical lenses with different focal lengths and overlapping zoom ranges, as well as an image sensor. The memory is used to store the calibration parameters, fusion corridor parameters, and computer programs of the four-camera module. The processor is configured to execute the computer program to operate the following functional modules: An initialization module is used to load the calibration parameters and configure system parameters; The image acquisition and synchronization module is used to control the four camera modules to acquire image data synchronously; The zoom routing module is used to determine the dominant camera and / or cameras participating in fusion and their fusion weights based on the current zoom level. An automatic ROI processing module is used to extract and track regions of interest on the dominant camera image; The geometric alignment and cropping module is used to perform field-of-view cropping and geometric alignment on the camera images participating in the fusion based on the ROI and the current zoom ratio. An image fusion module is used to perform weighted fusion of aligned images within a fusion corridor and to handle photometric consistency; and The depth calculation module is used to perform stereo matching based on aligned image pairs and output a depth map.

9. The image processing system based on four-camera cross-focal-length continuous zoom fusion according to claim 8, characterized in that, The zoom routing module is configured to: calculate the fusion weight using an S-shaped weight curve within the fusion corridor interval, and introduce a hysteresis mechanism and time-dimensional smoothing filtering.

10. The image processing system based on four-camera cross-focal-length continuous zoom fusion according to claim 8, characterized in that, The automatic ROI processing module is configured to perform a lag update mechanism, including at least one of the following: triggering conditions based on zoom change or time, dead zone setting, dwell time judgment, and motion prediction combined with IMU data.

Images