Zooming methods, devices, chips, terminals, and storage media

Abstract

This application discloses a zoom method, apparatus, chip, terminal, and storage medium, relating to the field of image capture technology. The method includes: acquiring first image data, which is image data output by a first image sensor; when the real-time zoom magnification reaches a zoom magnification threshold, controlling a second image sensor to turn on to output second image data, the zoom magnification threshold being located between a first zoom magnification and a second zoom magnification; determining a first image frame rate of the first image data and a second image frame rate of the second image data, such that the sum of the first image frame rate and the second image frame rate does not exceed the processing frame rate limit of the front-end chip; performing spatial alignment processing on the first image data at the first image frame rate and the second image data at the second image frame rate; and controlling the first image sensor to turn off when the real-time zoom magnification reaches the second zoom magnification.

Description

Technical Field

[0001] This application relates to the field of image capture technology, and in particular to a zoom method, device, chip, terminal, and storage medium. Background Technology

[0002] With the continuous development of terminal technology, more and more terminals are equipped with at least two fixed-focus lenses with different focal lengths. Users can switch image sensors during shooting to achieve zoom and improve shooting effects in different scenarios.

[0003] In related technologies, to avoid image abrupt changes during zooming due to lens switching, the terminal can use Spatial Alignment Transform (SAT) to align the brightness, viewing angle, distortion, and other elements of the two images, thereby achieving smooth image switching during zooming. Summary of the Invention

[0004] This application provides a zoom method, apparatus, chip, terminal, and storage medium that can improve image smoothness during zooming in devices with limited bandwidth.

[0005] On one hand, embodiments of this application provide a zoom method, the method comprising:

[0006] Acquire first image data, which is image data output by a first image sensor, and the first image sensor corresponds to a camera with a first zoom factor;

[0007] When the real-time zoom level reaches the zoom level threshold, the second image sensor is controlled to turn on to output second image data. The camera corresponding to the second image sensor uses the second zoom level, and the zoom level threshold is located between the first zoom level and the second zoom level.

[0008] The first image frame rate of the first image data and the second image frame rate of the second image data are determined such that the sum of the first image frame rate and the second image frame rate does not exceed the processing frame rate limit of the front-end chip. The front-end chip is used to perform image processing on the image data and input the processed image data to the back-end chip.

[0009] Spatial alignment processing is performed on the first image data at the first image frame rate and the second image data at the second image frame rate;

[0010] When the real-time zoom level reaches the second zoom level, the first image sensor is controlled to turn off.

[0011] On the other hand, embodiments of this application provide a zoom device, the device comprising:

[0012] The acquisition module is used to acquire first image data, which is image data output by a first image sensor, and the camera corresponding to the first image sensor adopts a first zoom factor.

[0013] The control module is used to control the second image sensor to turn on and output second image data when the real-time zoom magnification reaches the zoom magnification threshold. The camera corresponding to the second image sensor adopts the second zoom magnification, and the zoom magnification threshold is located between the first zoom magnification and the second zoom magnification.

[0014] The processing module determines the first image frame rate of the first image data and the second image frame rate of the second image data, such that the sum of the first image frame rate and the second image frame rate does not exceed the processing frame rate limit of the front-end chip. The front-end chip is used to perform image processing on the image data and input the processed image data to the back-end chip.

[0015] The processing module is further configured to perform spatial alignment processing on the first image data at the first image frame rate and the second image data at the second image frame rate.

[0016] The control module is also used to control the first image sensor to turn off when the real-time zoom reaches the second zoom.

[0017] On the other hand, embodiments of this application provide a terminal, the terminal including a processor and a memory, the memory storing at least one program, the at least one instruction being loaded and executed by the processor to implement the zoom method as described above.

[0018] On the other hand, embodiments of this application provide a chip that includes programmable logic circuits and / or program instructions, which, when the chip is running, are used to implement the zoom method as described above.

[0019] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one program, wherein the at least one instruction is loaded and executed by a processor to implement the zoom method as described above.

[0020] On the other hand, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the zoom method described above.

[0021] In this embodiment, when the first image sensor is working and the zoom level reaches a zoom level threshold, the terminal controls the second image sensor to start. Based on the processing frame rate limit of the front-end chip, the terminal determines the first and second image frame rates corresponding to the two image data streams, ensuring that the sum of the first and second image frame rates does not exceed the processing frame rate limit. Then, spatial alignment transformation is performed on the two image data streams. When the zoom level reaches the second zoom level corresponding to the second image sensor, the terminal controls the first image sensor to shut down and switches to output the second image data from the second image sensor, completing the entire zoom process. In scenarios where the processing bandwidth of the front-end chip is limited, a threshold is set based on the zoom level change, and the image frame rates of the first and second image sensors are adjusted when the zoom level reaches the threshold. This enables simultaneous operation of dual streams under low bandwidth, which helps to implement SAT-based zoom processes on low-to-mid-range platforms, thereby improving the smoothness of images during zoom. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown;

[0024] Figure 2 A flowchart of a zoom method provided in an exemplary embodiment of this application is shown;

[0025] Figure 3 A flowchart illustrating an exemplary embodiment of the image frame rate dynamic adjustment process provided in this application is shown.

[0026] Figure 4 This illustration shows a schematic diagram of the dynamic frame rate adjustment process provided in an exemplary embodiment of this application;

[0027] Figure 5 A timing diagram illustrating a dynamic frame rate adjustment process is shown in an exemplary embodiment of this application;

[0028] Figure 6 A flowchart of a zoom method provided by another exemplary embodiment of this application is shown;

[0029] Figure 7 A flowchart of a zoom method provided by another exemplary embodiment of this application is shown;

[0030] Figure 8A structural block diagram of a zoom device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0033] For ease of understanding, the terms used in the embodiments of this application will be explained below.

[0034] Spatial Alignment Transform (SALT) is a multi-camera hybrid zoom solution used on mobile devices. It mimics the smooth zoom of traditional cameras through a series of computational photography algorithms, solving the problem of abrupt image changes during multi-camera switching on mobile devices. When a user adjusts the zoom level from the zoom level corresponding to the first image sensor to the zoom level corresponding to the second image sensor, the terminal displays the image output by the first image sensor. The SALT algorithm then smoothly enlarges the image output by the first image sensor digitally, while simultaneously adjusting the angles, spatial positions, and overall brightness of various elements in the image to make the adjusted image closely resemble the image output by the second image sensor.

[0035] Frame rate: This is a measure of the number of frames displayed. The unit of measurement is frames per second (FPS) or hertz (Hz). Using a higher frame rate results in smoother, more realistic animation. Generally, a frame rate of around 30 FPS is acceptable to most users, but increasing the performance to 60 FPS can significantly improve interactivity and realism.

[0036] Frame interpolation algorithm: This is an algorithm that uses image processing techniques to generate a new frame of image. The frame interpolation algorithm calculates a new image based on consecutive preceding and following frames as an intermediate frame, and uses the intermediate frame to supplement (improve FPS) or replace the original frame (which may contain consecutive identical frames).

[0037] Zoom functionality facilitates finding suitable framing ranges during shooting. With the development of mobile phones, multiple cameras have become a common feature, each with a different focal length, allowing users to experience zoom capabilities similar to traditional cameras on their phones. Due to the slim and intelligent nature of mobile phones, most cameras use fixed-focus lenses. The mainstream zoom method employs hybrid zoom, combining digital and optical zoom, while continuous zoom is primarily achieved through algorithms. Spatial alignment algorithms can align brightness, position, and image distortion between two images obtained from lenses with different focal lengths, and magnify the target area through digital zoom to obtain an image at the mid-range focal length of the lens. Spatial alignment algorithms operate on two data streams. However, in low-to-mid-range platforms, the performance and bandwidth of the Image Signal Processor (ISP) are limited, making it impossible to process two high-frame-rate data streams simultaneously, thus hindering the implementation of zoom methods based on spatial alignment algorithms.

[0038] To achieve continuous zoom by applying spatial alignment transformation in mid-to-low-end platform terminals, this application provides a zoom scheme that dynamically adjusts the image output frame rate. This scheme can ensure that dual data streams are output simultaneously without exceeding the device's processing frame rate limit, thereby achieving spatial alignment transformation of the image based on the dual data streams, improving the smoothness of the image during zoom, and enabling users to obtain a better zoom experience on mid-to-low-end platform terminals.

[0039] Please refer to Figure 1 The diagram illustrates a structural block diagram of a terminal provided in an exemplary embodiment of this application. The terminal 100 may include one or more components such as a front-end chip 110, a back-end chip 120, a memory 130, and an image sensor 140.

[0040] The front-end chip 110 is used to preprocess data and transmit the preprocessed data to the back-end chip 120, which helps reduce the data processing pressure on the back-end chip 120. In this embodiment, the front-end chip 110 is equipped with an ISP for preprocessing the image data output by the image sensor 140. Of course, in other possible embodiments, the front-end chip 110 may also be equipped with other types of processors, which are not limited in this embodiment. It should be noted that compared with the back-end chip 120, the performance and processing bandwidth of the front-end chip 110 are smaller, that is, the processing power of the front-end chip 110 is lower than that of the back-end chip 120.

[0041] The back-end chip 120 may include one or more processing cores. The back-end chip 120 connects to various parts within the terminal 100 using various interfaces and lines, and performs various functions and processes data of the terminal 100 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 130, and by calling data stored in the memory 130. Optionally, the back-end chip 120 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The back-end chip 120 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and modem. Specifically, the CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; the NPU is used to implement Artificial Intelligence (AI) functions; and the modem is used to handle wireless communication. It is understandable that the aforementioned modem may not be integrated into the back-end chip 120, but may be implemented using a separate chip.

[0042] The memory 130 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 130 may include a non-transitory computer-readable storage medium. The memory 130 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 130 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc.; the data storage area may store data created based on the use of the terminal 100 (such as audio data, phone book, etc.).

[0043] Image sensor 140 is used to convert image optical signals into electronic signals and transmit image data. In this embodiment, terminal 100 is provided with at least two image sensors 140 (corresponding to two cameras with the same viewing direction), and different two image sensors 140 correspond to cameras with different zoom ratios. For example, terminal 100 is provided with a rear wide-angle lens and a rear ultra-wide-angle lens. In some embodiments, at least two cameras can work independently, that is, each camera outputs image data independently to its corresponding image sensor 140; correspondingly, at least two cameras can also work simultaneously, that is, multiple image sensors 140 simultaneously output multiple streams of image data.

[0044] In addition, those skilled in the art will understand that the structure of the terminal 100 shown in the above figures does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal 100 may also include a display screen, a speaker, an input unit, sensors (such as an accelerometer, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module, etc., which will not be described in detail here.

[0045] Please refer to Figure 2 This document illustrates a flowchart of a zoom method provided in an exemplary embodiment of this application. This embodiment uses this method for... Figure 1 Taking the terminal shown as an example, the method may include the following steps.

[0046] Step 201: Obtain first image data. The first image data is the image data output by the first image sensor, and the camera corresponding to the first image sensor uses the first zoom level.

[0047] The terminal acquires the first image data output by the first image sensor and displays the viewfinder image based on the first image data. In this embodiment, the first image sensor is the image sensor corresponding to the camera being used before zooming, and the camera corresponding to the first image sensor is a fixed-focus lens with a first zoom magnification. Accordingly, the first image sensor outputs the first image data at the first zoom magnification.

[0048] Optionally, the frame rate of the first image data is the target output frame rate, which can be the maximum output frame rate or the default output frame rate used by the image sensor when outputting image data. For example, the target output frame rate is 30 FPS.

[0049] Step 202: When the real-time zoom level reaches the zoom level threshold, control the second image sensor to turn on to output the second image data. The camera corresponding to the second image sensor uses the second zoom level, and the zoom level threshold is located between the first zoom level and the second zoom level.

[0050] In one possible implementation, during shooting using the terminal, when zooming is required, the user can perform a zoom operation by triggering the zoom control displayed on the shooting interface or by using zoom gestures (such as zooming in and out of the viewfinder with a finger). This zoom operation is used to increase or decrease the zoom magnification.

[0051] Optionally, since the camera corresponding to the first image sensor has a fixed-focus lens, during the zoom process, the terminal needs to digitally zoom the first image data according to the first zoom level and the real-time zoom level. For example, the terminal can locally magnify the center of the image represented by the first image data to simulate the effect of increased zoom level.

[0052] To ensure smooth image quality during zooming and avoid image abrupt changes caused by image sensor switching, in this embodiment, the terminal needs to control the second image sensor to be turned on in advance during the gradual switching from the first image sensor to the second image sensor, so as to perform spatial alignment transformation based on the two-channel image data output by the two image sensors.

[0053] Regarding the timing of controlling the activation of the second image sensor, in one possible implementation, the terminal sets a zoom factor threshold between a first zoom factor and a second zoom factor (the zoom factor used by the camera corresponding to the second image sensor). When the real-time zoom factor reaches the zoom factor threshold, the terminal controls the second image sensor to activate; when the real-time zoom factor does not reach the zoom factor threshold, the second image sensor remains deactivated.

[0054] In an illustrative example, when a user adjusts the zoom level of the lens from 0.5× (the zoom level of the camera corresponding to the first image sensor) to 1.0× (the zoom level of the camera corresponding to the first image sensor), only the first image sensor is active when the real-time zoom level is below 0.8×; when the real-time zoom level reaches 0.8×, the terminal controls the second image sensor to turn on, so that both the first and second image sensors are active and output image data.

[0055] Step 203: Determine the first image frame rate of the first image data and the second image frame rate of the second image data, such that the sum of the first image frame rate and the second image frame rate does not exceed the upper limit of the processing frame rate of the front-end chip. The front-end chip is used to perform image processing on the image data and input the processed image data to the back-end chip.

[0056] In this embodiment, the image data output by the image sensor is first preprocessed by the front-end chip, and then the front-end chip transmits the preprocessed image data to the back-end chip for further processing. Due to limitations in processing performance and bandwidth, the front-end chip cannot process multiple high-frame-rate image data streams simultaneously. Therefore, when both the first and second image sensors are activated simultaneously, frame rate control of the image data is necessary to enable the front-end chip to process the image data output by both the first and second image sensors simultaneously.

[0057] In one possible implementation, the terminal determines a first image frame rate for the first image data and a second image frame rate for the second image data based on the processing frame rate limit of the front-end chip, and ensures that the sum of the first and second image frame rates does not exceed the processing frame rate limit. The processing frame rate limit is the maximum frame rate supported by the front-end chip for image processing.

[0058] Optionally, the upper limit of the processing frame rate is greater than or equal to the target output frame rate of the image sensor, but less than twice the target output frame rate. For example, if the target output frame rate of the image sensor is 30 FPS, the upper limit of the processing frame rate of the front-end chip is 40 FPS.

[0059] In an illustrative example, when the processing frame rate limit of the front-end chip is 40 FPS, the sum of the first image frame rate and the second image frame rate does not exceed 40 FPS. For example, the first image frame rate and the second image frame rate are both 20 FPS, or the first image frame rate is 30 FPS and the second image frame rate is 10 FPS.

[0060] Optionally, the first image frame rate and the second image frame rate are determined and configured by the back-end chip of the terminal.

[0061] In some embodiments, the first image frame rate is greater than or equal to the second image frame rate, that is, the image frame rate of the first image data is given priority, so as to ensure that the display frame rate of the viewfinder during zooming is consistent with the first image frame rate.

[0062] Step 204: Spatial alignment processing is performed on the first image data at the first image frame rate and the second image data at the second image frame rate.

[0063] When different cameras on a terminal capture the same target, the target and scene in the images differ significantly in terms of field of view, size, and position. Directly switching between the two images to achieve zoom would be abrupt. Through spatial alignment processing, the first image data undergoes smooth digital center scaling, and its position, distortion, and brightness are adjusted to match the second image data. This processing yields images corresponding to multiple zoom levels between the first and second image data, ensuring smoothness during zooming.

[0064] In this embodiment, the spatial alignment process is executed by a back-end chip (such as an application processor within the back-end chip). The back-end chip uses a spatial alignment algorithm to align the first image data and the second image data. For two images obtained at different focal lengths, the algorithm performs operations such as magnification, translation, and rotation on the first image data based on the second image data, making the position of the first image data closer to the position in the second image data, and adjusting elements such as brightness and distortion of the image.

[0065] Furthermore, after completing the spatial alignment process, the terminal displays the view based on the spatially aligned image data.

[0066] It should be noted that the first image data and the second image data are image data output by the front-end chip after preprocessing. The preprocessing performed by the front-end chip includes black level compensation, lens shading correction, bad pixel correction, color interpolation, noise removal, white balance correction, color correction, gamma correction, color space conversion (RGB to YUV), etc. This embodiment does not limit these processes.

[0067] Step 205: When the real-time zoom reaches the second zoom level, control the first image sensor to turn off.

[0068] When the zoom level reaches the second zoom level, the terminal controls the first image sensor to turn off, that is, the first image sensor stops transmitting image data backward and switches to the second image sensor, that is, the viewfinder is displayed based on the second image data output by the second image sensor.

[0069] In an illustrative example, as the user adjusts the zoom level of the lens from 0.5× (the zoom level corresponding to the first image sensor) to 1.0× (the zoom level corresponding to the first image sensor), when the real-time zoom level reaches 1.0×, the terminal controls the first image sensor to turn off and switches to the second image sensor.

[0070] It should be noted that during the process of changing from the second zoom level to the third zoom level (the zoom level corresponding to the third image sensor), if the real-time zoom level reaches a certain zoom level threshold between the second zoom level and the third zoom level, the terminal controls the third image sensor to turn on and performs image frame rate determination and spatial alignment processing. This will not be elaborated on in this embodiment.

[0071] In summary, in this embodiment, when the first image sensor is working and the zoom level reaches a zoom level threshold, the terminal controls the second image sensor to start. Based on the upper limit of the processing frame rate of the front-end chip, the terminal determines the first and second image frame rates corresponding to the two image data streams, ensuring that the sum of the first and second image frame rates does not exceed the upper limit of the processing frame rate. Then, spatial alignment transformation is performed on the two image data streams. When the zoom level reaches the second zoom level corresponding to the second image sensor, the terminal controls the first image sensor to shut down and switches to output the second image data from the second image sensor, completing the entire zoom process. In scenarios with limited processing bandwidth of the front-end chip, a threshold is set based on the zoom level change, and the image frame rates of the first and second image sensors are adjusted when the zoom level reaches the threshold. This enables simultaneous operation of dual streams under low bandwidth conditions, facilitating the implementation of SAT-based zoom processes on low-to-mid-range platforms and improving the smoothness of images during zooming.

[0072] In low-to-mid-range platform terminals, the processing bandwidth of their front-end chips is limited, making it impossible to process two high-frame-rate image data streams simultaneously. Therefore, the front-end chip has a processing frame rate limit when preprocessing the two image data streams. When the first and second image sensors are simultaneously activated, the terminal's back-end chip (such as an AP) determines the first and second image frame rates corresponding to each of the two image data streams, thereby achieving frame rate control of the image data and enabling the front-end chip to process the two image data streams simultaneously. Regarding the method of determining the first and second image frame rates, in one possible implementation, any of the following methods can be used:

[0073] 1. The first image frame rate and the second image frame rate remain unchanged during zooming.

[0074] In one possible implementation, based on the upper limit of the processing frame rate of the front-end chip, the back-end chip predetermines the first image frame rate and the second image frame rate when the two data streams are transmitted simultaneously. When the second image sensor is turned on, the frame rates of the first image data and the second image data are controlled to remain unchanged according to the first image frame rate and the second image frame rate until the real-time zoom reaches the second zoom.

[0075] Optionally, the back-end chip predetermines that the first image frame rate and the second image frame rate are the same and do not exceed half of the upper limit of the processing frame rate.

[0076] The first image sensor is the image sensor of the camera currently in use before the zoom process begins. Therefore, at the initial stage of zooming, the first image frame rate is the target output frame rate used by the terminal when outputting image data using one image sensor. This target output frame rate is a relatively high frame rate that meets the user's shooting needs. To simultaneously process dual data streams, the sum of the first and second image frame rates cannot exceed the processing frame rate limit. Therefore, the first image sensor cannot maintain a consistently high frame rate. In this embodiment, after triggering the second image sensor to activate, the first image frame rate of the first image data needs to be reduced. Regarding the method of reducing the first image frame rate, the backend chip sets the first and second image frame rates to be the same as the processing frame rate limit, and not exceeding half of the processing frame rate limit.

[0077] In an illustrative example, the maximum processing frame rate is 40 FPS, the initial output frame rate of the first image sensor is 30 FPS (i.e., the initial first image frame rate is 30 FPS), when the real-time zoom reaches the zoom threshold, the first image frame rate is set to 20 FPS, and the second image frame rate is set to 20 FPS. Then, when processing dual data simultaneously, the sum of the image frame rates does not exceed 40 FPS.

[0078] Of course, in other possible implementations, the first image frame rate and the second image frame rate can be different, and the first image frame rate can be greater than the second image frame rate. For example, when the processing frame rate limit is 40 FPS, the first image frame rate can be 25 FPS and the second image data can be 15 FPS, or the first image frame rate can be 30 FPS and the second image data can be 10 FPS.

[0079] 2. The first image frame rate and the second image frame rate are dynamically adjusted during zooming, and the first image frame rate decreases during zooming, while the second image frame rate increases during zooming.

[0080] In the above possible implementation, as the terminal controls the second image sensor to start, the back-end chip is configured to reduce the first image frame rate. Therefore, the display frame rate of the viewfinder is at a low frame rate for a long time, which affects the display quality of the viewfinder.

[0081] In one possible implementation, when the second image sensor is activated, the first image data maintains a high frame rate (e.g., maintains the target image frame rate), while the second image data uses a lower frame rate. As the real-time zoom level gradually approaches the second zoom level, the back-end chip dynamically increases the second image frame rate. Correspondingly, based on the processing frame rate limit of the front-end chip, the back-end chip dynamically decreases the first image frame rate to ensure that the sum of the first and second image frame rates does not exceed the processing frame rate limit. Therefore, the first image data can be output at a high frame rate for a certain period, ensuring the display quality of the viewfinder image.

[0082] In an illustrative example, as the user adjusts the zoom level from 0.5× (the zoom level corresponding to the first image sensor) to 1.0× (the zoom level corresponding to the first image sensor), when the real-time zoom level reaches 0.8× (the zoom level threshold), the back-end chip determines that the first image data will continue to be output at 30 FPS (the target output frame rate), and the second image frame rate is initialized at 10 FPS. When the real-time zoom level reaches 0.9×, the back-end chip determines that the second image frame rate is increased to 15 FPS. Based on the processing frame rate limit of 40 FPS, the back-end chip determines that the second image frame rate is reduced to 25 FPS.

[0083] Regarding the method of dynamically adjusting the first image frame rate and the second image frame rate, in one possible implementation, any of the following methods can be adopted:

[0084] 2.1 The back-end chip adjusts the frame rate of the first image and the frame rate of the second image based on the adjustment strategy. The adjustment strategy includes the timing of frame rate adjustment and the corresponding frame rate adjustment method.

[0085] In one possible implementation, the terminal pre-determines an adjustment strategy for the first and second image frame rates during zooming. When the real-time zoom magnification approaches the first zoom magnification, the lower frame rate of the second image data is sufficient to meet the requirements of the spatial alignment algorithm. However, as the real-time zoom magnification approaches the second zoom magnification, the difference between the image in the first image data and the image displayed on the terminal becomes significant. Therefore, the spatial alignment algorithm needs to increase the number of spatial alignment operations to adjust the difference between the two images, correspondingly requiring a higher frame rate for the second image data. Therefore, in the adjustment strategy, the terminal pre-determines the frame rate adjustment timing based on the real-time zoom magnification, progressively increases the second image frame rate based on the required second image frame rate for different real-time zoom magnifications, and progressively decreases the first image frame rate based on the upper limit of the processing frame rate. During zooming, when the real-time zoom magnification reaches the frame rate adjustment timing, the back-end chip adjusts the first and second image frame rates based on the adjustment strategy corresponding to that frame rate adjustment timing.

[0086] In an illustrative example, the zoom threshold is 0.7×. The terminal is pre-set with an adjustment strategy for the user when adjusting the real-time zoom from 0.5× (the zoom level corresponding to the first image sensor) to 1.0× (the zoom level corresponding to the first image sensor): First gradient frame rate: when the real-time zoom reaches 0.7× (the zoom threshold), the first image frame rate is 30 FPS, and the second image frame rate is 10 FPS; Second gradient frame rate: when the real-time zoom reaches 0.8×, the first image frame rate is 25 FPS, and the second image frame rate is 15 FPS; Third gradient frame rate: when the real-time zoom reaches 0.9×, the first image frame rate is 20 FPS, and the second image frame rate is 20 FPS. The backend chip adjusts the first and second image frame rates based on the above adjustment strategy.

[0087] Optionally, the back-end chip can set different gradient adjustment strategies based on the number of frames of the images already output by the second image sensor.

[0088] In an illustrative example, the terminal pre-sets an adjustment strategy for the user when adjusting the real-time zoom from 0.5 × (the zoom level corresponding to the first image sensor) to 1.0 × (the zoom level corresponding to the first image sensor): First gradient frame rate: when the second image sensor has output 0 frames, the first image frame rate is 30 FPS (target output frame rate), and the second image frame rate is 10 FPS; Second gradient frame rate: when the second image sensor has output 5 frames, the first image frame rate is 25 FPS, and the second image frame rate is 15 FPS; Third gradient frame rate: when the second image sensor has output 10 frames, the first image frame rate is 20 FPS, and the second image frame rate is 20 FPS. The backend chip adjusts the first and second image frame rates based on the above adjustment strategy.

[0089] 2.2 The back-end chip initializes the first image frame rate and the second image frame rate, and adjusts the first image frame rate and the second image frame rate based on the spatial alignment effect.

[0090] Spatial alignment algorithms are based on adjusting first image data using second image data. However, a certain difference still exists between the adjusted image and the second image data. When this difference is too large, image abruptness can occur during zooming. The difference between the adjusted image and the second image data is reflected in the differences in displacement, distortion, brightness, and angular deflection. Spatial alignment algorithms can use these differences to calculate the spatial alignment effect and characterize the similarity between the two images. Since spatial alignment algorithms adjust first image data based on second image data, when the second image data contains many images, the algorithm performs more spatial alignment operations, resulting in a better spatial alignment effect. This means the backend chip needs to gradually increase the frame rate of the second image based on the real-time spatial alignment effect to improve the spatial alignment performance.

[0091] In one possible implementation, when the terminal controls the second image sensor to start, the backend chip initializes the first image frame rate and the second image frame rate. Optionally, the backend chip initializes the first image frame rate to the target output frame rate and the second output frame rate to a low frame rate. After spatial alignment processing based on the two image data, the backend chip obtains spatial alignment effect information by comparing the differences between the image obtained by the alignment algorithm and the second image data. Based on the spatial alignment effect information, it determines the frame rate adjustment parameters and further adjusts the first image frame rate and the second image frame rate based on the frame rate adjustment parameters.

[0092] Spatial alignment effect can serve as a standard for determining how to adjust the frame rate of the second image. When the difference between the image obtained by the spatial alignment algorithm and the second image data is large, it indicates that the current frame rate of the second image is insufficient to meet the requirements of spatial alignment processing. In one possible implementation, the back-end chip quantifies the difference between the image obtained by the spatial alignment algorithm and the second image data using a spatial alignment effect parameter. The back-end chip can then determine the frame rate parameter, including the target adjusted frame rate, based on the quantized value of the difference between the images (i.e., the spatial alignment effect parameter). This will be described below with reference to an exemplary embodiment.

[0093] Please refer to Figure 3 The diagram illustrates a flowchart of an image frame rate dynamic adjustment process provided in an exemplary embodiment of this application.

[0094] Step 310: Initialize the first image frame rate and the second image frame rate.

[0095] In this embodiment, the first image sensor is the image sensor currently in use by the terminal before zooming, outputting images at the target output frame rate. To ensure that the sum of the two data frame rates does not exceed the upper limit of the processing frame rate, the back-end chip initializes the second image frame rate at a lower frame rate. For example, if the upper limit of the processing frame rate is 40 FPS, the first image frame rate (target output frame rate) is 30 FPS, and the second image frame rate is initialized at 5 FPS.

[0096] Step 320: Adjust the frame rate of the first image and the frame rate of the second image based on the spatial alignment effect.

[0097] Spatial alignment effect refers to the alignment performance of the spatial alignment algorithm. The backend chip performs operations such as magnification, translation, and rotation on the first image data based on the second image data by running the spatial alignment algorithm. Correspondingly, the number of spatial alignments per unit time is equal to the number of frames in the second image data. Since the more spatial alignment processes per second, the better the spatial alignment effect, the frame rate of the second image should be increased accordingly to improve the spatial alignment effect.

[0098] In one possible implementation, this step may include the following sub-steps:

[0099] Step 321: Determine the spatial alignment effect parameters. The spatial alignment effect parameters are used to characterize the spatial alignment effect and are determined based on the spatially aligned image data and the second image data.

[0100] The spatial alignment effect parameter is a parameter that characterizes the spatial alignment effect. The spatial alignment algorithm compares and calculates the deviation between the spatially aligned image and the second image data, and calculates the difference value of each element, including brightness difference value, angle deflection value, distortion value, etc. The difference between the two images is quantified from different angles. Then, the algorithm calculates the spatial alignment effect parameter based on each difference value. This parameter comprehensively characterizes the similarity between the image obtained by the algorithm and the second image data. A low similarity indicates a poor spatial alignment effect.

[0101] In one possible implementation, the spatial alignment effect parameter = a1 * brightness difference value + a2 * angle deflection value + a3 * distortion value, where a1, a2, and a3 are the weights of the differences between different elements in calculating the spatial alignment effect parameter, representing the proportion of each element's influence on the spatial alignment effect.

[0102] Step 322: Adjust the frame rate of the second image based on the spatial alignment effect parameters.

[0103] A lower spatial alignment effect parameter indicates that the second image frame rate is lower than the second image frame rate required for real-time zoom. Based on the spatial alignment effect parameter, the back-end chip determines the second image frame rate that can meet the spatial alignment processing requirements, and then configures the second image frame rate to adjust the frame rate.

[0104] Optionally, the terminal establishes a correspondence between spatial alignment effect parameters and image frame rates. Accordingly, the front-end chip determines the second image frame rate to be adjusted based on the real-time spatial alignment effect parameters and this correspondence. Table 1 illustrates the illustrative correspondence between spatial alignment effect parameters and image frame rates.

[0105] Table 1

[0106] Parameter 1 to Parameter 2 5FPS Parameters 2 to Parameters 3 10FPS Parameters 3 to 4 15FPS

[0107] In an illustrative example, as shown in Table 1, the first image has a frame rate of 30 FPS and the second image has a frame rate of 10 FPS. After spatial alignment processing, the back-end chip calculates that the spatial alignment effect parameter of the spatial alignment image is between parameter 3 and parameter 4, thereby determining to adjust the frame rate of the second image to 15 FPS.

[0108] Step 323: Adjust the first image frame rate based on the adjusted second image frame rate.

[0109] As the second image frame rate increases, the sum of the first and second image frame rates approaches the upper limit of the processing frame rate. To ensure that the sum of the first and second image frame rates does not exceed the upper limit of the processing frame rate, i.e., to ensure that the front-end chip can process two image data streams simultaneously, the back-end chip needs to correspondingly reduce the first image frame rate. Regarding the method of adjusting the first image frame rate based on the adjusted second image frame rate, in one possible implementation, the following approach can be adopted:

[0110] 1. If the sum of the adjusted second image frame rate and the first image frame rate does not exceed the upper limit of the processing frame rate, maintain the first image frame rate.

[0111] When dynamically adjusting the frame rate, the premise of adjusting the image frame rate is to maintain the first image frame rate as high as possible. Therefore, if the sum of the adjusted second image frame rate and the first image frame rate does not exceed the upper limit of the processing frame rate, the front-end chip can process two image data simultaneously and transmit them to the back end, so there is no need to adjust the first image frame rate.

[0112] For example, if the maximum processing frame rate is 40 FPS, the first image frame rate is 30 FPS, the initial frame rate of the second image is 5 FPS, and the frame rate of the second image is adjusted to 10 FPS based on spatial alignment, then the sum of the frame rates of the second and first images will not exceed the maximum processing frame rate. Therefore, the frame rate of the first image can be kept unchanged at 30 FPS.

[0113] 2. If the sum of the adjusted second image frame rate and the first image frame rate exceeds the upper limit of the processing frame rate, the first image frame rate is reduced, wherein the sum of the adjusted second image frame rate and the reduced first image frame rate does not exceed the upper limit of the processing frame rate.

[0114] When the second image frame rate is increased significantly, the sum of the adjusted second image frame rate and the first image frame rate exceeds the processing frame rate limit. At this point, the front-end chip cannot process the images from both data streams simultaneously. To ensure smooth spatial alignment processing, the back-end chip needs to accordingly reduce the first image frame rate until the sum of the second image frame rate and the first image frame rate does not exceed the processing frame rate limit.

[0115] For example, if the maximum processing frame rate is 40 FPS, the first image frame rate is 30 FPS, the initial frame rate of the second image is 5 FPS, and the frame rate of the second image is adjusted to 15 FPS based on spatial alignment, then the sum of the second image frame rate and the first image frame rate is 45 FPS, exceeding the maximum processing frame rate. The backend chip then determines to adjust the first image frame rate to 25 FPS, reducing the sum of the second and first image frame rates to 40 FPS, thus not exceeding the maximum processing frame rate. This frame rate adjustment range is merely an example and is not intended to limit the scope of this application.

[0116] In summary, in this embodiment, when the real-time zoom reaches the zoom threshold, the back-end chip initializes the output frame rate of the second image sensor and the first image frame rate, and determines the frame rate parameter for adjusting the second image frame rate based on the spatial alignment effect parameter calculated by the spatial alignment algorithm. Then, based on the increase of the second image frame rate and the upper limit of the processing frame rate, the back-end chip determines whether to maintain or reduce the first image frame rate. In scenarios where the terminal bandwidth of low-end platforms is limited, the frame rates of the first and second image sensors are dynamically adjusted based on the spatial alignment effect, realizing dynamic frame rate adjustment based on the terminal display image quality. This helps to reduce image jumps and improve the smoothness of the image during zoom while enabling simultaneous operation of dual streams under low bandwidth.

[0117] Based on the above method, the first image frame rate and the second image frame rate are determined. The back-end chip further sends instructions to relevant components to adjust the image frame rate, thereby enabling the back-end chip to adjust the frame rate. Regarding the method of implementing frame rate adjustment, in one possible implementation, any of the following methods can be used:

[0118] 1. Send a first frame rate configuration instruction containing a first image frame rate to a first image sensor, and send a second frame rate configuration instruction containing a second image frame rate to a second image sensor. The first image sensor is used to configure a first output frame rate according to the first image frame rate, and the second image sensor is used to configure a second output frame rate according to the second image frame rate.

[0119] In one possible implementation, the back-end chip sends frame rate configuration instructions to the first image sensor and the second image sensor respectively, based on the determined first image frame rate and second image frame rate, instructing the first image sensor and the second image sensor to configure the output frame rate of the image data.

[0120] Based on the determined first image frame rate and second image frame rate, the back-end chip sends a first frame rate configuration instruction to the first image sensor. The first image sensor configures its own output frame rate according to the first image frame rate in the first frame rate configuration instruction, thereby transmitting the first image data at the first image frame rate. Similarly, based on the determined second image frame rate, the back-end chip sends a second frame rate configuration instruction to the second image sensor. The second image sensor configures its own output frame rate according to the second image frame rate in the second frame rate configuration instruction, thereby transmitting the second image data at the determined second image frame rate.

[0121] 2. Send a first frame dropping instruction to the front-end chip. The frame dropping instruction includes a first image frame rate and a second image frame rate. The front-end chip is used to perform frame dropping processing on the image data output by the first image sensor and the second image sensor based on the first image frame rate and the second image frame rate, respectively. The first image sensor and the second image sensor output image data to the front-end chip according to the target output frame rate.

[0122] In Method 1 above, the frame rate adjustment process first requires the back-end chip to send a frame rate configuration command to the image sensor, and then the image sensor needs to configure the output frame rate based on the frame rate configuration command. The whole process takes a long time (especially configuring the output frame rate).

[0123] Frame rate adjustment via frame dropping eliminates the need for an image sensor and is significantly faster. Frame dropping is a process performed on the received image data by a frame dropping module located in the front-end chip. In frame dropping, the module controls the transmission of the corresponding number of frames based on the frame rate parameter in the received frame dropping instruction, while stopping the transmission of the remaining frames.

[0124] For example, when the first image frame rate is 30 FPS, the back-end chip sends a frame dropping command to the front-end chip to adjust the first image frame rate to 25 FPS. At this time, the frame dropping module transmits 25 frames of the obtained 30 frames of the first image data and discards the remaining 5 frames. When the frame rate is adjusted through the frame dropping algorithm, the image sensor can always maintain the target output frame rate or the default output frame rate without modifying the configured output frame rate.

[0125] In one possible implementation, the back-end chip can send a frame-dropping command to the frame-dropping module in the front-end chip to adjust the frame rate using a frame-dropping algorithm. Based on a determined first output frame rate and second output frame rate, the back-end chip sends a first frame-dropping command to the front-end chip, and the frame-dropping module in the front-end chip executes the command. The first frame-dropping command is an instruction that controls the frame-dropping module to drop frames. The frame-dropping module determines the number of image frames to be transmitted based on the first and second output frame rates in the first frame-dropping command, and discards the remaining image frames accordingly.

[0126] In an illustrative example, both the first and second output frame rates are 30 FPS. The backend chip sends a first frame-dropping instruction, which specifies that the first image frame rate should be adjusted to 25 FPS and the second image frame rate should be adjusted to 15 FPS. Based on the first frame-dropping instruction, the frame-dropping module discards 5 frames of the first image data per second and transmits the remaining 25 frames, and discards 15 frames of the second image data and transmits the remaining 15 frames.

[0127] 3. Send a second frame dropping instruction to the front-end chip. The frame dropping instruction contains a first image frame rate. The front-end chip is used to perform frame dropping processing on the image data output by the first image sensor based on the first image frame rate. The first image sensor outputs image data to the front-end chip according to the target output frame rate. Send a second frame rate configuration instruction containing a second image frame rate to the second image sensor. The second image sensor is used to configure a second output frame rate according to the second image frame rate.

[0128] In both Method 1 and Method 2 described above, the back-end chip uses a consistent strategy to adjust the first image frame rate and the second image frame rate. In another possible implementation, the back-end chip can use different strategies described above to adjust the frame rate.

[0129] Since the image output by the first image sensor is ultimately presented as the terminal display image after spatial alignment transformation, the first image data must prioritize image quality during zooming. Based on the fact that frame rate adjustment takes less time, the image frame rate can be adjusted to the target image frame rate in a timely manner. Therefore, the first image frame rate is adjusted by frame dropping strategy.

[0130] For the second image sensor, the back-end chip can send a frame rate configuration command to the second image sensor to instruct it to configure the output frame rate, thereby adjusting the image frame rate.

[0131] In an illustrative example, such as Figure 4 As shown, the front-end chip 410 needs to preprocess the image data output by the image sensor 400 through the Pre-ISP (Image Signal Preprocessing) module 412, and then input the preprocessed image data into the back-end chip 420 through MIPI.

[0132] To ensure the display quality of the viewfinder image during zooming, the back-end chip 420 needs to perform spatial alignment transformation on the two image data through the control alignment module 421, so that the control-aligned image data is input into the upper-level image processing module 422 for further processing, and finally, based on the processed image data, the display screen is controlled to display the viewfinder image.

[0133] After the spatial alignment transformation is performed, the frame rate adjustment module 423 in the back-end chip 420 determines the adjustment parameters of the image frame rate based on the spatial alignment effect, and sends a frame rate configuration command to the image sensor 400 based on the adjustment parameters, instructing the image sensor 400 to adjust the output frame rate.

[0134] In addition, since the front-end chip 410 is also equipped with a frame dropping module 411, the back-end chip 420 can also send a frame dropping command to the front-end chip 410 based on the adjustment parameters. The frame dropping module 411 performs frame dropping processing on the image data transmitted by the image sensor 400 based on the image frame rate indicated by the frame dropping command, so as to meet the processing performance of the Pre-ISP module 412.

[0135] After frame rate adjustment, the two image data streams can be processed simultaneously at the front-end chip and then transmitted to the spatial alignment module at the back-end chip for spatial alignment. Regarding the method of spatial alignment, one possible approach is to use any of the following:

[0136] 1. When the frame rate of the first image is the same as that of the second image, perform frame-by-frame spatial alignment processing on the first image data and the second image data.

[0137] When the back-end chip adjusts the first image frame rate and the second image frame rate to be the same, the spatial alignment algorithm receives two image data with the same number of frames. The algorithm can align the first image data of each frame with the corresponding second image data of the second frame, and each frame of first image data can generate a new image.

[0138] For example, if the backend chip determines the first image frame rate and the second image frame rate to be 20 FPS based on the upper limit of the processing frame rate, then within one second the algorithm will perform frame-by-frame alignment of the corresponding 20 frames of first image data based on 20 frames of second image data.

[0139] 2. When the first image frame rate and the second image frame rate are different, spatial alignment processing is performed on the first image data and the second image data based on the minimum output frame rate between the first image frame rate and the second image frame rate.

[0140] The spatial alignment algorithm is based on processing the first image data using the second image data. In other words, the second image data serves as the comparison standard during processing. When the second image sensor is initialized at a lower frame rate, and both images are transmitted to the spatial alignment processing module, the second image frame rate is lower than the first image frame rate. Therefore, the spatial alignment algorithm can only process a portion of the first image data based on the second image data and then transmit the processed image forward. Specifically, during spatial alignment, it is necessary to perform spatial alignment based on the images in the first and second image data at the same time.

[0141] 3. When the frame rate of the first image is greater than that of the second image, frame interpolation is performed on the second image data. The frame rate of the interpolated second image data is then the frame rate of the first image. Frame-by-frame spatial alignment is then performed on both the first image data and the interpolated second image data. When the frame rate of the first image is greater than that of the second image, direct spatial alignment would require the same number of alignment steps as the second image frame rate. By using frame interpolation to increase the frame rate of the second image, the two frame rates can be kept consistent. This allows for full utilization of both image data streams while simultaneously achieving frame-by-frame spatial alignment, increasing the number of alignment steps and thus improving the spatial alignment effect.

[0142] In one possible implementation, when the first image frame rate is greater than the second image frame rate, the back-end chip performs frame interpolation processing on the second image data using a frame interpolation algorithm, increasing the number of image frames in the second image data to be the same as the number of image frames in the first image data. Frame interpolation processing refers to the frame interpolation algorithm calculating on the second image data after image processing by the front-end chip. The frame interpolation algorithm calculates an intermediate frame based on two consecutive frames in the second image data. This intermediate frame is a new image frame, and it is inserted between the two images, thereby increasing the number of image frames processed.

[0143] It should be noted that the frame interpolation process is performed by a frame interpolation algorithm located in the back-end chip, and the frame interpolation process should be performed before the spatial alignment process.

[0144] In the above embodiments, to ensure the SAT process is implemented under bandwidth constraints, the frame rate of the first image data used as the output image is reduced, which affects the image quality to some extent. At the end of the zoom process, the terminal needs to promptly increase the frame rate of the output image to ensure high image quality. Since this process is essentially a frame rate adjustment process, the method for increasing the output image frame rate should be consistent with the frame rate adjustment method used in the early stage of the zoom process, and may include the following methods:

[0145] 1. Send a third frame rate configuration command containing the target output frame rate to the second image sensor, which is used to configure the second output frame rate as the target output frame rate.

[0146] In the early stages of the zoom process, when the back-end chip adjusts the frame rate of the second image sensor by sending a frame rate configuration command to the second image sensor, the back-end chip adjusts the frame rate by sending a third frame rate configuration command to the second image sensor when it is necessary to increase the frame rate of the second image sensor.

[0147] In one possible implementation, before shutting down the first image sensor, the zoom method prioritizes maintaining the first image frame rate. After switching between the first and second image sensors, the back-end chip sends a third frame rate configuration command to the second image sensor. Based on the third frame rate configuration command, the sensor outputs second image data at the target output frame rate, i.e., the second output frame rate is adjusted to the target output frame rate. Here, the second output frame rate refers to the frame rate of the image output by the second image sensor.

[0148] In an illustrative example, the maximum processing frame rate is 40 FPS, the target output frame rate is 30 FPS, and before switching image sensors, the first image sensor has a first output frame rate of 25 FPS and the second image sensor has a second output frame rate of 15 FPS. When the zoom reaches 1.0× and spatial alignment is completed, the back-end chip controls the first image sensor to turn off and adjusts the output frame rate of the second image sensor to 30 FPS.

[0149] Optionally, before switching between the first and second image sensors, the back-end chip adjusts the second output frame rate to the target output frame rate. After the frame rate is increased, the image sensor switching is performed, and the second output frame rate after the switch is the target output frame rate, without the need for further frame rate adjustment. This avoids low frame rate images appearing in the terminal display after the image sensor switch.

[0150] In an illustrative example, such as Figure 5 As shown, the first image sensor is the ultra-wide-angle image sensor corresponding to the ultra-wide-angle camera, and the second image sensor is the wide-angle image sensor corresponding to the wide-angle camera. The processing frame rate limit of the front-end chip is 40 FPS, and the target output frame rate (the initial output frame rate of the ultra-wide-angle image sensor) is 30 FPS. When the zoom magnification reaches the zoom magnification threshold, the back-end chip initializes the wide-angle image sensor at 10 FPS, controls the wide-angle image sensor to start outputting image data, and uses 10 FPS as the output frame rate. When the zoom magnification reaches the second zoom magnification (the zoom magnification corresponding to the wide-angle image sensor), the back-end chip sends a third frame rate configuration command to the wide-angle image sensor. The wide-angle resource manager adjusts the output frame rate from 10 FPS to 30 FPS based on the third frame rate configuration command. After the ultra-wide-angle image sensor stops transmitting image data, the wide-angle image sensor transmits image data again at 30 FPS based on the third frame rate configuration command, and then the ultra-wide-angle image sensor stops outputting image data. In this example, after the image sensor switching is completed, the wide-angle image sensor continues to output at a frame rate of 30 FPS. The frame rate of the image displayed on the terminal is 30 FPS before and after the image sensor switching, so as to avoid a sudden drop in image quality during the switching.

[0151] 2. Send a first stop frame dropping command to the front-end chip, which is used to stop the frame dropping processing of the image data output by the second image sensor based on the first stop frame dropping command.

[0152] In the early stages of the zoom process, when the back-end chip adjusts the second image frame rate by sending a frame dropping command to the front-end chip, the back-end chip adjusts the frame rate by sending a first stop frame dropping command to the second image sensor when it is necessary to increase the second image frame rate.

[0153] In one possible implementation, after the first image sensor is turned off, the second image sensor is switched to transmit image data backward. The back-end chip sends a first stop frame dropping command to the frame dropping module, and the frame dropping module stops processing the second image data output by the second image sensor according to the command.

[0154] In an illustrative example, the maximum processing frame rate is 40 FPS, the target output frame rate is 30 FPS, and before switching image sensors, the second output frame rate of the second image sensor is 30 FPS. The frame rate of the second image sensor is adjusted to 15 FPS through a frame dropping algorithm. When the zoom reaches 1.0×, the back-end chip controls the first image sensor to turn off and sends a first stop frame dropping command to the front-end chip to stop frame dropping, so that the frame rate of the second image sensor is adjusted to 30 FPS.

[0155] Optionally, before switching between the first and second image sensors, the frame dropping algorithm is disabled to adjust the second image frame rate to the target output frame rate. After the frame rate increase is complete, the image sensor switching is performed, and the second image frame rate after the switch is the target output frame rate, without the need for further frame rate adjustment.

[0156] In an illustrative example, the front-end chip has a processing frame rate cap of 40 FPS and a target output frame rate of 30 FPS. Before switching image sensors, the second output frame rate is 30 FPS. The frame dropping algorithm adjusts the second image frame rate to 15 FPS. When the zoom reaches 1.0×, a first stop frame dropping command is sent to stop the frame dropping algorithm. The second image frame rate is then adjusted to 30 FPS, and the first image sensor is turned off.

[0157] It should be noted that the zoom process does not end solely when the real-time zoom magnification reaches the second zoom magnification. Maintaining a certain real-time zoom magnification for a certain duration can also be considered the end of the zoom process. In this case, it can be assumed that there is no further need for magnification adjustment, and the first image frame rate should be increased to ensure the quality of the image displayed on the terminal. The method for increasing the output image frame rate should be compatible with the frame rate adjustment methods described above. In this case, the frame rate increase can be achieved using any of the following methods:

[0158] 1. When the real-time zoom magnification reaches the zoom magnification threshold and the holding time reaches the duration threshold, a fourth frame rate configuration command containing the target output frame rate is sent to the first image sensor. The first image sensor is used to configure the first output frame rate as the target output frame rate.

[0159] In the early stages of the zoom process, when the back-end chip adjusts the second image frame rate by sending a first frame rate configuration command to the first image sensor, the back-end chip adjusts the frame rate by sending a fourth frame rate configuration command to the first image sensor when it is necessary to increase the first image frame rate.

[0160] The duration threshold refers to the length of time the real-time zoom magnification remains constant. When the duration threshold is reached, it can be determined that there is no longer a need to change the real-time zoom magnification, and the terminal will need to output images at this real-time zoom magnification for an extended period. When the zoom magnification has not reached the second zoom magnification and the duration reaches the duration threshold, the first image sensor continues to operate and serves as the image sensor for transmitting image data. Therefore, the first output frame rate should be promptly increased to the target output frame rate to ensure image quality.

[0161] In an illustrative example, during the process of adjusting the zoom magnification from 0.5× (the zoom magnification corresponding to the first image sensor) to 1.0× (the zoom magnification corresponding to the second image sensor), the zoom magnification threshold is 0.8×, and the duration threshold is 2s. When the real-time zoom magnification reaches 0.9×, the back-end chip has completed the adjustment of the first image frame rate, that is, the back-end chip has sent the first frame rate configuration command to reduce the first image frame rate to 20FPS. When the zoom magnification is maintained at 0.9× for 2s, the back-end chip sends a fourth configuration command to the first image sensor to increase the first output frame rate to the target output frame rate of 30FPS.

[0162] In one possible implementation, the back-end chip adjusts the first image frame rate to decrease based on the real-time zoom magnification. Then, when the terminal remains unchanged at the same or another real-time zoom magnification for a period of time, the back-end chip sends a fourth frame rate configuration command to the first image sensor. The image sensor of the first image sensor transmits the image back at the target output frame rate based on the fourth frame rate configuration command.

[0163] 2. When the real-time zoom level reaches the zoom level threshold and the holding time reaches the holding time threshold, a second stop frame dropping command is sent to the front-end chip. The front-end chip is used to stop the frame dropping processing of the image data output by the first image sensor based on the second stop frame dropping command.

[0164] In the early stages of the zoom process, when the back-end chip adjusts the frame rate of the first image by sending a frame dropping command to the first image sensor, the back-end chip adjusts the frame rate by sending a stop frame dropping command to the first image sensor when it is necessary to increase the frame rate of the first image.

[0165] After the backend chip adjusts and reduces the first image frame rate by sending a frame dropping command, the terminal maintains a certain real-time zoom magnification for a duration threshold. Then, the backend chip sends a second stop frame dropping command to the frame dropping module. This second stop frame dropping command stops the frame dropping module from dropping frames of the first image data. Based on this command, the frame dropping module stops transmitting certain frames of the first image data. Accordingly, in the zoom method where frame rate adjustment is achieved through the frame dropping module, the first transmission frame rate is the target transmission frame rate. That is, after the frontend chip stops dropping frames of the first image data, the first image frame rate is adjusted to the target output frame rate.

[0166] In an illustrative example, during the zoom adjustment from 0.5× (the zoom level corresponding to the first image sensor) to 1.0× (the zoom level corresponding to the second image sensor), the zoom level threshold is 0.8×, and the duration threshold is 2 seconds. When the real-time zoom level reaches 0.9×, the back-end chip sends a frame dropping command to the front-end chip, instructing the front-end chip to drop frames from the first image data output by the first image sensor, reducing the first image frame rate to 20 FPS. When the zoom level is maintained at 0.9× for 2 seconds, the back-end chip sends a stop frame dropping command to the front-end chip, instructing the front-end chip to stop dropping frames from the first image data output by the first image sensor, restoring the first image frame rate to 30 FPS.

[0167] The zoom process will now be described in detail through an exemplary embodiment.

[0168] Please refer to Figure 6 The diagram illustrates a flowchart of a zoom method provided in another exemplary embodiment of this application.

[0169] Step 601: Obtain first image data. The first image data is the image data output by the first image sensor, and the camera corresponding to the first image sensor uses the first zoom factor.

[0170] This step is the same as step 201, and will not be repeated here.

[0171] Step 602: When the real-time zoom level reaches the zoom level threshold, control the second image sensor to turn on. The camera corresponding to the second image sensor uses the second zoom level, and the zoom level threshold is located between the first zoom level and the second zoom level.

[0172] This step is the same as step 202, and will not be repeated here.

[0173] Step 603: Determine the first image frame rate and the second image frame rate based on the upper limit of the processing frame rate. The first image frame rate and the second image frame rate are the same and do not exceed half of the upper limit of the processing frame rate.

[0174] To enable the front-end chip to process dual-channel image data simultaneously, the sum of the first and second image frame rates must not exceed the upper limit of the processing frame rate. In this embodiment, after the terminal controls the second image sensor to start, the back-end chip configures the adjustment of the first and second image frame rates based on pre-determined frame rate parameters. The back-end chip pre-determines the same first and second image frame rates according to the upper limit of the processing frame rate, and neither exceeds half of the upper limit.

[0175] In one illustrative embodiment, the processing frame rate of the front-end chip is capped at 40 FPS. When the real-time zoom reaches the zoom threshold, the back-end chip determines that the first image frame rate is 20 FPS and the second image frame rate is 20 FPS, with the sum of the frame rates not exceeding 40 FPS.

[0176] Step 604: Send a first frame rate configuration instruction containing a first image frame rate to the first image sensor and send a second frame rate configuration instruction containing a second image frame rate to the second image sensor. The first image sensor is used to configure a first output frame rate according to the first image frame rate, and the second image sensor is used to configure a second output frame rate according to the second image frame rate.

[0177] In this embodiment, based on the first image frame rate determined in step 603, the back-end chip sends a first frame rate configuration instruction to the first image sensor. The image sensor of the first image sensor determines the number of updates per second according to the first output frame rate in the first frame rate configuration instruction, and then the first image sensor transmits the image backward at the first output frame rate in the instruction. Similarly, based on the second image frame rate determined in step 603, the back-end chip sends a second frame rate configuration instruction to the second image sensor, and the second image sensor transmits the image backward according to the second output frame rate in the second frame rate configuration instruction.

[0178] Step 605: If the first image frame rate and the second image frame rate are the same, perform frame-by-frame spatial alignment processing on the first image data and the second image data.

[0179] In this embodiment, the back-end chip is pre-set that when the first image sensor and the second image sensor work simultaneously, the first image frame rate and the second image frame rate are the same. Therefore, the spatial alignment algorithm receives two image data with the same frame rate. The algorithm can perform alignment processing on the first image data of each frame according to the second image data of the corresponding frame at the same time. Each frame of first image data can generate a new image frame through spatial alignment transformation processing. That is, spatial alignment processing can be performed frame by frame.

[0180] Step 606: When the real-time zoom reaches the second zoom level, control the first image sensor to turn off.

[0181] This step is the same as step 205, and will not be repeated here.

[0182] Step 607: Send a third frame rate configuration command containing the target output frame rate to the second image sensor, which is used to configure the second output frame rate as the target output frame rate.

[0183] In this embodiment, when the second image data is switched to be displayed on the terminal, the frame rate of the second image remains a low frame rate, not exceeding half of the upper limit of the processing frame rate. To ensure the quality of the image displayed on the terminal, the frame rate of the second image needs to be increased promptly after the image sensor switch is completed. Based on the adjustment of the second image frame rate by the back-end chip in step 604 by sending a first frame rate configuration command, the back-end chip should adjust the frame rate by sending a configuration command to the second image sensor. The back-end chip sends a third frame rate configuration command to the second image sensor to increase the second image frame rate to the target output frame rate.

[0184] In an illustrative example, the front-end chip has a processing frame rate limit of 40 FPS and a target output frame rate of 30 FPS. The back-end chip predetermines the second image frame rate to be 20 FPS. When the zoom level reaches the zoom level threshold, the back-end chip sends a first frame rate configuration command to the second image sensor, and then the image sensor in the second image sensor transmits images at a frame rate of 20 FPS. When the zoom level reaches the second zoom level, the terminal controls the second image sensor to turn off. In order to increase the second image frame rate, it is necessary to adjust the frame rate at which the image sensor of the second image sensor transmits images, that is, the second output frame rate. Therefore, the back-end chip sends a third frame rate configuration command to the second image sensor to increase the second output frame rate to the target output frame rate.

[0185] In the above embodiments, the back-end chip reduces the first image frame rate when the second image sensor is activated. The first image sensor outputs data at a low frame rate for a longer period, meaning the terminal displays a low-quality image for an extended time. In one possible implementation, the terminal sets different frame rate parameters in real time based on spatial alignment effect parameters, prioritizing the first image frame rate during the early stages of zooming to shorten the time the terminal displays the image at a low frame rate. This will be described below with reference to exemplary embodiments.

[0186] Please refer to Figure 7 The diagram illustrates a flowchart of a zoom method provided in another exemplary embodiment of this application.

[0187] Step 701: Obtain first image data. The first image data is the image data output by the first image sensor, and the camera corresponding to the first image sensor uses the first zoom factor.

[0188] This step is the same as step 201, and will not be repeated here.

[0189] Step 702: When the real-time zoom level reaches the zoom level threshold, control the second image sensor to turn on to output the second image data. The camera corresponding to the second image sensor uses the second zoom level, and the zoom level threshold is located between the first zoom level and the second zoom level.

[0190] This step is the same as step 202, and will not be repeated here.

[0191] Step 703: Initialize the first image frame rate and the second image frame rate.

[0192] As the terminal controls the second image sensor to turn on, the terminal initializes the output frame rates of the first and second image sensors. In this embodiment, the terminal determines and adjusts the first and second output frame rates to target output frame rates. Based on the upper limit of the processing frame rate, as the first and second image sensors transmit images backward at the target image frame rate, the back-end chip sends a first frame drop instruction to the front-end chip, adjusting the sum of the first and second image frame rates to be lower than the upper limit of the processing frame rate.

[0193] Optionally, in the first frame dropping instruction, the first image frame rate is kept at the target output frame rate to ensure the image quality of the image displayed on the terminal, and the second image frame rate is a low frame rate.

[0194] Step 704: If the frame rate of the first image is greater than the frame rate of the second image, perform frame interpolation on the second image data. The frame rate of the second image data after frame interpolation is the frame rate of the first image.

[0195] Since the terminal prioritizes the first image frame rate during initialization of the first and second image frame rates in step 703, the first image frame rate is greater than the second image frame rate. When the frame rates of the two data streams are different, in order to improve the spatial alignment effect, the backend chip first performs frame interpolation processing on the second image frame rate, which has a lower frame rate, to make the number of image frames of the two data streams consistent, thereby enabling frame-by-frame spatial alignment processing.

[0196] In an illustrative example, the first image has a frame rate of 25 FPS and the second image has a frame rate of 15 FPS. The frame interpolation algorithm obtains 10 intermediate frames based on the second image data and inserts them into the second image data. Thus, when the two image data are transmitted to the spatial alignment processing module, 30 frames of images are transmitted per second in the second image data, consistent with the first image data.

[0197] Step 705: Perform frame-by-frame spatial alignment processing on the first image data and the second image data after frame interpolation.

[0198] Based on step 704, when the image data is transmitted to the spatial alignment processing module, the first image frame rate and the second image frame rate are the same, so frame-by-frame spatial alignment processing can be performed.

[0199] Step 706: Determine the spatial alignment effect parameters. The spatial alignment effect parameters are used to characterize the spatial alignment effect and are determined based on the spatially aligned image data and the second image data.

[0200] By comparing the processed image obtained in step 705 with the second image data, the spatial alignment algorithm calculates the difference values ​​of each element. Then, the spatial alignment algorithm substitutes these difference values ​​into the spatial alignment effect parameter calculation formula for further calculation.

[0201] Spatial alignment effect parameters = a1 * brightness difference value + a2 * angle deflection value + a3 * distortion value + ...

[0202] This allows us to obtain the spatial alignment effect parameters for this spatial alignment process.

[0203] Step 707: Adjust the second image frame rate based on the spatial alignment effect parameters; adjust the first image frame rate based on the adjusted second image frame rate.

[0204] Based on the spatial alignment effect parameters obtained in step 706, the back-end chip determines the corresponding frame rate parameters, refers to the correspondence table between spatial alignment effect parameters and frame rate parameters (the correspondence table is for illustrative purposes only, and this application does not limit the method for determining the correspondence between spatial alignment effect parameters and frame rate parameters), determines the second image frame rate, and determines the first image frame rate based on the upper limit of the processing frame rate.

[0205] Step 708: Send a first frame dropping instruction to the front-end chip. The frame dropping instruction includes a first image frame rate and a second image frame rate. The front-end chip is used to perform frame dropping processing on the image data output by the first image sensor and the second image sensor based on the first image frame rate and the second image frame rate, respectively. The first image sensor and the second image sensor output image data to the front-end chip according to the target output frame rate.

[0206] Based on the first and second image frame rates determined by the back-end chip in step 707, the back-end chip sends a first frame dropping instruction to the frame dropping module of the front-end chip in real time. The first frame dropping instruction includes a frame rate parameter determined based on the spatial alignment effect parameter. The frame dropping module controls a portion of the first image data and a portion of the second image data to stop being transmitted backward based on the first and second image frame rates in the first frame dropping instruction, thereby achieving frame rate adjustment.

[0207] Step 709: When the real-time zoom reaches the second zoom level, control the first image sensor to turn off.

[0208] Step 710: Send a first stop frame dropping command to the front-end chip. The front-end chip is used to stop the frame dropping processing of the image data output by the second image sensor based on the first stop frame dropping command.

[0209] Please refer to Figure 8 The diagram illustrates a structural block diagram of a zoom device provided in an exemplary embodiment of this application, the device comprising:

[0210] The acquisition module 801 is used to acquire first image data, which is image data output by a first image sensor, and the camera corresponding to the first image sensor adopts a first zoom factor.

[0211] The control module 802 is used to control the second image sensor to turn on to output second image data when the real-time zoom magnification reaches the zoom magnification threshold. The camera corresponding to the second image sensor adopts the second zoom magnification, and the zoom magnification threshold is located between the first zoom magnification and the second zoom magnification.

[0212] Processing module 803 determines the first image frame rate of the first image data and the second image frame rate of the second image data, so that the sum of the first image frame rate and the second image frame rate does not exceed the processing frame rate limit of the front-end chip. The front-end chip is used to perform image processing on the image data and input the processed image data to the back-end chip.

[0213] The processing module 803 is further configured to perform spatial alignment processing on the first image data at the first image frame rate and the second image data at the second image frame rate.

[0214] The control module 802 is also used to control the first image sensor to turn off when the real-time zoom reaches the second zoom.

[0215] Optional,

[0216] The first and second image frame rates remain constant during zooming;

[0217] The first image frame rate and the second image frame rate are dynamically adjusted during zooming, with the first image frame rate decreasing and the second image frame rate increasing during zooming.

[0218] Optionally, the processing module 803 is used for:

[0219] The first image frame rate and the second image frame rate are determined based on the processing frame rate limit, wherein the first image frame rate and the second image frame rate are the same and do not exceed half of the processing frame rate limit;

[0220] The frame rates of the first and second images are adjusted based on an adjustment strategy, which includes the timing of frame rate adjustment and the corresponding frame rate adjustment method.

[0221] Initialize the first image frame rate and the second image frame rate; adjust the first image frame rate and the second image frame rate based on spatial alignment effect.

[0222] Optionally, the processing module 803 is further configured to:

[0223] When the first image frame rate and the second image frame rate are the same, the first image data and the second image data are subjected to frame-by-frame spatial alignment processing.

[0224] When the first image frame rate is different from the second image frame rate, spatial alignment processing is performed on the first image data and the second image data based on the minimum output frame rate between the first image frame rate and the second image frame rate.

[0225] When the frame rate of the first image is greater than the frame rate of the second image, frame interpolation is performed on the second image data, and the frame rate of the second image data after frame interpolation is the frame rate of the first image; frame-by-frame spatial alignment is performed on the first image data and the second image data after frame interpolation.

[0226] Optionally, the control module 802 is used for:

[0227] A third frame rate configuration command containing a target output frame rate is sent to the second image sensor, which is used to configure the second output frame rate as the target output frame rate;

[0228] A first stop frame dropping instruction is sent to the front-end chip, and the front-end chip is used to stop the frame dropping processing of the image data output by the second image sensor based on the first stop frame dropping instruction;

[0229] When the real-time zoom magnification reaches the zoom magnification threshold and the holding time reaches the duration threshold, a fourth frame rate configuration command containing the target output frame rate is sent to the first image sensor, and the first image sensor is used to configure the first output frame rate as the target output frame rate.

[0230] When the real-time zoom level reaches the zoom level threshold and the holding time reaches the holding time threshold, a second stop frame dropping command is sent to the front-end chip. The front-end chip is used to stop the frame dropping processing of the image data output by the second image sensor based on the second stop frame dropping command.

[0231] Optionally, during the process of adjusting the frame rate of the first image and the frame rate of the second image based on spatial alignment effect, the processing module 803 is further configured to:

[0232] Determine spatial alignment effect parameters, which characterize the spatial alignment effect, and are determined based on the spatially aligned image data and the second image data;

[0233] Adjust the frame rate of the second image based on the spatial alignment effect parameters;

[0234] The first image frame rate is adjusted based on the adjusted second image frame rate.

[0235] Optionally, during the process of adjusting the first image frame rate based on the adjusted second image frame rate, the processing module 803 is used to:

[0236] If the sum of the adjusted second image frame rate and the first image frame rate does not exceed the processing frame rate limit, the first image frame rate is maintained.

[0237] If the sum of the adjusted second image frame rate and the first image frame rate exceeds the upper limit of the processing frame rate, the first image frame rate is reduced, wherein the sum of the adjusted second image frame rate and the reduced first image frame rate does not exceed the upper limit of the processing frame rate.

[0238] Optionally, the control module 802 is further configured to:

[0239] Send a first frame rate configuration instruction containing the first image frame rate to the first image sensor, and send a second frame rate configuration instruction containing the second image frame rate to the second image sensor. The first image sensor is configured to configure a first output frame rate according to the first image frame rate, and the second image sensor is configured to configure a second output frame rate according to the second image frame rate.

[0240] A first frame dropping instruction is sent to the front-end chip. The frame dropping instruction includes the first image frame rate and the second image frame rate. The front-end chip is used to perform frame dropping processing on the image data output by the first image sensor and the second image sensor based on the first image frame rate and the second image frame rate, respectively. The first image sensor and the second image sensor output image data to the front-end chip according to the target output frame rate.

[0241] A second frame dropping instruction is sent to the front-end chip, the frame dropping instruction containing the first image frame rate. The front-end chip is used to perform frame dropping processing on the image data output by the first image sensor based on the first image frame rate. The first image sensor outputs image data to the front-end chip according to the target output frame rate. A second frame rate configuration instruction containing the second image frame rate is sent to the second image sensor, the second image sensor is used to configure a second output frame rate according to the second image frame rate.

[0242] In summary, in this embodiment, when the first image sensor is working and the real-time zoom reaches the zoom threshold, the control module controls the second image sensor to start. Based on the upper limit of the processing frame rate of the front-end chip, the control module determines the first and second image frame rates corresponding to the two image data streams, ensuring that the sum of the first and second image frame rates does not exceed the upper limit of the processing frame rate. The processing module then performs spatial alignment transformation on the two image data streams. When the zoom reaches the second zoom level corresponding to the second image sensor, the control module controls the first image sensor to shut down and switches to output the second image data from the second image sensor, completing the entire zoom process. In scenarios with limited processing bandwidth of the front-end chip, setting a threshold based on the zoom level change and adjusting the image frame rates of the first and second image sensors when the zoom reaches the threshold enables simultaneous operation of dual streams under low bandwidth conditions. This helps to implement SAT-based zoom processes on low-to-mid-range platforms, thereby improving the smoothness of images during zoom.

[0243] This application also provides a chip, which includes programmable logic circuits and / or program instructions, and when the chip is running, it is used to implement the zoom method as described in the above embodiments.

[0244] This application also provides a computer-readable storage medium storing at least one program that is executed by a processor to implement the zoom method as described in the above embodiments.

[0245] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the zoom method provided in the above embodiments.

[0246] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0247] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A zoom method, characterized in that, The method includes: Acquire first image data, which is image data output by a first image sensor, and the camera corresponding to the first image sensor uses a first zoom factor; When the real-time zoom level reaches the zoom level threshold, the second image sensor is controlled to turn on to output second image data. The camera corresponding to the second image sensor uses the second zoom level, and the zoom level threshold is located between the first zoom level and the second zoom level. The first image frame rate of the first image data and the second image frame rate of the second image data are determined such that the sum of the first image frame rate and the second image frame rate does not exceed the processing frame rate limit of the front-end chip. The front-end chip is used to perform image processing on the image data and input the processed image data to the back-end chip. Spatial alignment processing is performed on the first image data at the first image frame rate and the second image data at the second image frame rate; When the real-time zoom level reaches the second zoom level, the first image sensor is controlled to turn off.

2. The method according to claim 1, characterized in that, The first image frame rate and the second image frame rate remain constant during zooming; or, The first image frame rate and the second image frame rate are dynamically adjusted during zooming, and the first image frame rate decreases and the second image frame rate increases during zooming.

3. The method according to claim 2, characterized in that, The first image frame rate and the second image frame rate remain constant during zooming; Determining the first image frame rate of the first image data and the second image frame rate of the second image data includes: The first image frame rate and the second image frame rate are determined based on the processing frame rate upper limit. The first image frame rate and the second image frame rate are the same and do not exceed half of the processing frame rate upper limit.

4. The method according to claim 2, characterized in that, The first image frame rate and the second image frame rate are dynamically adjusted during zooming. Determining the first image frame rate of the first image data and the second image frame rate of the second image data includes: The frame rates of the first and second images are adjusted based on an adjustment strategy, which includes the timing of frame rate adjustment and the corresponding frame rate adjustment method. or, Initialize the first image frame rate and the second image frame rate; adjust the first image frame rate and the second image frame rate based on spatial alignment effect.

5. The method according to claim 4, characterized in that, The adjustment of the first image frame rate and the second image frame rate based on spatial alignment includes: Determine spatial alignment effect parameters, which characterize the spatial alignment effect, and are determined based on the spatially aligned image data and the second image data; Adjust the frame rate of the second image based on the spatial alignment effect parameters; The first image frame rate is adjusted based on the adjusted second image frame rate.

6. The method according to claim 5, characterized in that, The adjustment of the first image frame rate based on the adjusted second image frame rate includes: If the sum of the adjusted second image frame rate and the first image frame rate does not exceed the processing frame rate limit, the first image frame rate is maintained. or, If the sum of the adjusted second image frame rate and the first image frame rate exceeds the upper limit of the processing frame rate, the first image frame rate is reduced, wherein the sum of the adjusted second image frame rate and the reduced first image frame rate does not exceed the upper limit of the processing frame rate.

7. The method according to any one of claims 1 to 6, characterized in that, The spatial alignment process for the first image data at the first image frame rate and the second image data at the second image frame rate includes: When the first image frame rate and the second image frame rate are the same, the first image data and the second image data are subjected to frame-by-frame spatial alignment processing. or, When the first image frame rate is different from the second image frame rate, spatial alignment processing is performed on the first image data and the second image data based on the minimum output frame rate between the first image frame rate and the second image frame rate. or, When the first image frame rate is greater than the second image frame rate, the second image data is subjected to frame interpolation, and the frame rate of the second image data after frame interpolation is the first image frame rate; the first image data and the second image data after frame interpolation are subjected to frame-by-frame spatial alignment.

8. The method according to any one of claims 1 to 6, characterized in that, After determining the first image frame rate of the first image data and the second image frame rate of the second image data, the method further includes: Send a first frame rate configuration instruction containing the first image frame rate to the first image sensor, and send a second frame rate configuration instruction containing the second image frame rate to the second image sensor. The first image sensor is configured to configure a first output frame rate according to the first image frame rate, and the second image sensor is configured to configure a second output frame rate according to the second image frame rate. or, A first frame dropping instruction is sent to the front-end chip. The frame dropping instruction includes the first image frame rate and the second image frame rate. The front-end chip is used to perform frame dropping processing on the image data output by the first image sensor and the second image sensor based on the first image frame rate and the second image frame rate, respectively. The first image sensor and the second image sensor output image data to the front-end chip according to the target output frame rate. or, A second frame dropping instruction is sent to the front-end chip, the frame dropping instruction containing the first image frame rate. The front-end chip is used to perform frame dropping processing on the image data output by the first image sensor based on the first image frame rate. The first image sensor outputs image data to the front-end chip according to the target output frame rate. A second frame rate configuration instruction containing the second image frame rate is sent to the second image sensor, the second image sensor is used to configure a second output frame rate according to the second image frame rate.

9. The method according to any one of claims 1 to 6, characterized in that, The method further includes: A third frame rate configuration command containing a target output frame rate is sent to the second image sensor, the second image sensor being configured to set the second output frame rate to the target output frame rate, the second output frame rate referring to the frame rate of the image output by the second image sensor; or, A first stop frame dropping instruction is sent to the front-end chip, and the front-end chip is used to stop the frame dropping processing of the image data output by the second image sensor based on the first stop frame dropping instruction.

10. The method according to any one of claims 1 to 6, characterized in that, The method further includes: When the real-time zoom reaches the zoom threshold and the holding time reaches the duration threshold, a fourth frame rate configuration command containing the target output frame rate is sent to the first image sensor. The first image sensor is used to configure the first output frame rate to the target output frame rate. The first output frame rate refers to the frame rate of the image output by the first image sensor. or, When the real-time zoom level reaches the zoom level threshold and the holding time reaches the holding time threshold, a second stop frame dropping command is sent to the front-end chip. The front-end chip is used to stop the frame dropping processing of the image data output by the second image sensor based on the second stop frame dropping command.

11. A zoom device, characterized in that, The device includes: The acquisition module is used to acquire first image data, which is image data output by a first image sensor, and the camera corresponding to the first image sensor adopts a first zoom factor. The control module is used to control the second image sensor to turn on and output second image data when the real-time zoom magnification reaches the zoom magnification threshold. The camera corresponding to the second image sensor adopts the second zoom magnification, and the zoom magnification threshold is located between the first zoom magnification and the second zoom magnification. The processing module determines the first image frame rate of the first image data and the second image frame rate of the second image data, such that the sum of the first image frame rate and the second image frame rate does not exceed the processing frame rate limit of the front-end chip. The front-end chip is used to perform image processing on the image data and input the processed image data to the back-end chip. The processing module is further configured to perform spatial alignment processing on the first image data at the first image frame rate and the second image data at the second image frame rate. The control module is also used to control the first image sensor to turn off when the real-time zoom reaches the second zoom.

12. A chip, characterized in that, The chip includes programmable logic circuitry and / or program instructions, which, when the chip is running, are used to implement the zoom method as described in any one of claims 1 to 10.

13. A terminal, characterized in that, The terminal includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the zoom method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the zoom method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the zoom method as described in any one of claims 1 to 10.

Images