Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: comparing a current zoom ratio with a target optical variation point zoom ratio to obtain a comparison result; the target light change point zoom magnification is greater than the original light change point zoom magnification; under the condition that the comparison result shows that the first camera is the main camera, fusing first image data collected by the first camera and second image data collected by the second camera to obtain fused image data; obtaining a target image according to the current zoom ratio and the fused image data; and under the condition that the comparison result shows that the second camera is the main camera, obtaining a target image according to the current zoom ratio and the second image data. By adopting the method, picture jumping can be avoided.

Description

Technical Field

[0001] This application relates to the field of image technology, and in particular to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of photography technology, people have increasingly higher requirements for the quality of captured images. To achieve better shooting results, multi-camera designs have become increasingly popular. Different cameras are designed for different shooting scenarios, and different shooting needs can be met by switching between cameras. Zooming is a common scenario for smartphones. For multi-camera platforms, due to parallax and camera configuration, some zooming scenarios suffer from image jumps, resulting in relatively poor image smoothness during zooming. Summary of the Invention

[0003] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium that can ensure smoother image switching.

[0004] In a first aspect, this application provides an image processing method, comprising:

[0005] The current zoom ratio is compared with the target optical zoom point zoom ratio to obtain the comparison result; the target optical zoom point zoom ratio is greater than the original optical zoom point zoom ratio.

[0006] If the comparison result indicates that the first camera is the main camera, the first image data captured by the first camera and the second image data captured by the second camera are fused to obtain fused image data; the target image is obtained based on the current zoom level and the fused image data.

[0007] If the comparison result indicates that the second camera is the main camera, the target image is obtained based on the current zoom level and the second image data.

[0008] Secondly, this application also provides an image processing apparatus, comprising:

[0009] The comparison module is used to compare the current zoom ratio with the target optical zoom point zoom ratio to obtain a comparison result; the target optical zoom point zoom ratio is greater than the original optical zoom point zoom ratio.

[0010] The dual-camera module is used to fuse first image data acquired by the first camera and second image data acquired by the second camera when the comparison result indicates that the first camera is the main camera, to obtain fused image data; and to obtain a target image based on the current zoom level and the fused image data.

[0011] A switching module is used to obtain a target image based on the current zoom level and the second image data when the comparison result indicates that the second camera is the main camera.

[0012] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the image processing method of the first aspect.

[0013] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method of the first aspect.

[0014] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method of the first aspect.

[0015] The aforementioned image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product compare the current zoom ratio with the target optical zoom ratio to obtain a comparison result, which forms at least two cases. Since the target optical zoom ratio is greater than the original optical zoom ratio, when switching the main camera using the target optical zoom ratio, the moment when switching from a focal length with a smaller zoom ratio to a focal length with a larger zoom ratio is delayed. Therefore, the situation where the comparison result indicates the first camera is the main camera is prolonged, allowing the first camera to capture more images. The system acquires one image data and another image data from the second camera. By fusing these two types of image data, more fused image data can be obtained. This allows for the generation of more target images based on the current zoom level and the fused image data, thereby increasing the smoothness of the image during the main camera switching process. Then, if the comparison result indicates that the second camera is the main camera, the target image is obtained based on the current zoom level and the second image data, thus completing the main camera switching process. This achieves smooth changes in the image during the zoom process and ensures the smoothness of the image during the main camera switching process. Attached Figure Description

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0018] Figure 2 This is a schematic diagram of the target image of an image processing method in one embodiment;

[0019] Figure 3 This is a flowchart illustrating an image processing method in one embodiment;

[0020] Figure 4 This is a schematic diagram of the process for obtaining fused image data in one embodiment;

[0021] Figure 5 This is a schematic diagram illustrating the current zoom ratio changing from 0.6x to 6x in one embodiment;

[0022] Figure 6 This is a software architecture diagram of an image processing method in one embodiment;

[0023] Figure 7 This is a schematic diagram illustrating the specific process of an image processing method in one embodiment;

[0024] Figure 8 A flowchart illustrating the predicted values ​​of the current frame in one embodiment;

[0025] Figure 9 This is a structural block diagram of an image processing device in one embodiment;

[0026] Figure 10 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0028] Zooming is a common scenario for multi-camera platforms, and the smoothness and responsiveness of zooming are key optimization areas to consider. Some electronic devices may have multiple different types of cameras. For example, the camera operation page of a multi-camera phone may display focal length selections such as 0.6X, 1X, 3X, or 6X. The smaller the number, the wider the field of view and the farther away the subject, roughly corresponding to the focal lengths of ultra-wide (UW), wide (W), and telephoto (T) cameras. Figure 1The electronic device shown includes three cameras: an ultra-wide-angle camera, a wide-angle camera, and a telephoto camera. The zoom ratio at which the three cameras switch is the optical zoom point. Specifically, the original optical zoom ratio between the ultra-wide-angle camera and the wide-angle camera is 1X, which is as follows... Figure 2 As shown in (a); the wide-angle camera captures images at a 2X zoom ratio, as shown in... Figure 2 As shown in (b); the original optical zoom ratio between the wide-angle camera and the telephoto camera is 3X, which is as follows: Figure 2 As shown in (c); the telephoto camera captures images at a 5x zoom ratio, as shown in... Figure 2 As shown in (d) in the figure.

[0029] For multi-camera platforms, some zoom scenarios present challenges in optimizing transitions. For example, in a 0.6x to 6x zoom transition, when the transition zoom ratio determined by the corresponding point-cut curve reaches the original optical zoom ratio, the secondary camera needs to support corresponding image data acquisition. However, the secondary camera's activation time is insufficient, resulting in a delay in the process from power-on to image data output. This leads to probabilistic abnormal image shifts or jitter during zooming. For instance, with a 3-camera platform, the secondary camera is activated only after receiving the point-cut signal, thus requiring the main and secondary cameras to output image data together around the 5th or 6th frame in the transition frame. In some point-cut curves, the original optical zoom ratio between the two cameras is 3x, and the zoom ratio of the 7th frame image data is greater than 3x, requiring the main camera to switch from the ultra-wide-angle lens to the telephoto lens. Therefore, before switching to the main camera, algorithms for multi-frame fusion scenarios such as spatial alignment only use 1-2 frames for alignment or fusion. If there is an alignment error, the error may not converge within the two frames, resulting in a sudden image change and a poor user experience. Therefore, ensuring smooth lens switching and minimizing image jumps during zooming has become a key consideration for current cameras. Smoothness refers to the fluidity of the image frames displayed on the screen during zooming, i.e., continuous flow; while minimizing image jumps is achieved through spatial alignment processing to ensure consistent image quality before and after the switch.

[0030] The image processing method provided in this application can be applied to electronic devices. These electronic devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The electronic device can be a terminal or a server.

[0031] In some exemplary embodiments, such as Figure 3 As shown, an image processing method is provided, which is applied to an electronic device. The method includes steps 302 to 306, wherein:

[0032] Step 302: Compare the current zoom ratio with the target optical zoom point zoom ratio to obtain the comparison result; the target optical zoom point zoom ratio is greater than the original optical zoom point zoom ratio.

[0033] The current zoom ratio is the zoom ratio of the image frame that the electronic device needs to display at the current moment; there is a current frame at the current moment, which is the image frame that needs to be previewed, photographed, or recorded; correspondingly, the zoom operation can be a zoom operation in the camera preview scene, the photo shooting scene, or the recording scene.

[0034] The target optical zoom point zoom ratio is greater than the original optical zoom point zoom ratio, and both the target and original optical zoom point zoom ratios are optical zoom point zoom ratios. The optical zoom point zoom ratio refers to the zoom ratio threshold for switching or activating a new camera. This ratio is used to switch the main camera, turning the original secondary camera into the main camera and vice versa. The switched secondary camera can be turned off. For example, the target optical zoom point zoom ratio can be set for both the ultra-wide-angle and telephoto cameras. In this case, there is no need to control the activation of the wide-angle camera between the ultra-wide-angle and telephoto cameras, allowing for the decision to activate all three cameras simultaneously as needed, thereby reducing power consumption when not activating all three cameras at the same time. For example, the first camera is an ultra-wide-angle camera, and the second camera is a telephoto camera. In a scenario where the zoom ratio gradually changes from the focal length of the ultra-wide-angle camera to the focal length of the telephoto camera, and it is a point-to-point zoom, the focal length of the ultra-wide-angle camera can be 0.6-3.5, and the focal length of the second camera as the main camera can be 3.5-6. In this case, the wide-angle camera can be turned off to save power.

[0035] The comparison result is a numerical comparison of the zoom ratio, which is used to determine the main camera among the cameras. The comparison result may include a first comparison result and a second comparison result, which are different. The first comparison result indicates that the first camera is the main camera, and the second comparison result indicates that the second camera is the main camera. The first comparison result can be represented by a first identifier, and the second comparison result can be represented by a second identifier.

[0036] For example, in response to a zoom operation, during the focal length switching process from 0.6X to 6X, the current zoom ratio is continuously updated using a point-cut curve, and then each updated current zoom ratio is compared with the target optical zoom point zoom ratio. If the current zoom ratio is less than the target optical zoom point zoom ratio, a first comparison result is obtained; if the current zoom ratio is greater than or equal to the target optical zoom point zoom ratio, a second comparison result is obtained. It is understood that steps 302-306 can be applied not only to the aforementioned 0.6X to 6X focal length switching process, but also to the focal length switching process from 1X to 6X.

[0037] Step 304: If the comparison result indicates that the first camera is the main camera, the first image data acquired by the first camera and the second image data acquired by the second camera are fused to obtain fused image data; the target image is obtained based on the current zoom level and the fused image data.

[0038] The first camera can be activated before the second camera. The main camera is the camera corresponding to the focal length of the current zoom level, and at least the main camera is required for image acquisition. Before the camera switching process involving the zoom level of the target optical zoom point, the first camera is the main camera, and the second camera is required as a secondary camera for image acquisition to ensure a smooth switching process through image data fusion. After the camera switching process involving the zoom level of the target optical zoom point, the second camera is required as the main camera for image acquisition, and it can be used independently for image acquisition.

[0039] The first image data is the image data captured by the first camera, and the second image data is the image data captured by the second camera. Since the zoom level changes gradually during the zoom process, the camera activated at different zoom levels has different possibilities. Therefore, the first camera, which is activated first, acts as the main camera and captures the first image data. After the second camera is activated, the first camera continues to capture the first image data, and the second camera captures the second image data. The first and second image data are then fused to obtain the fused image data.

[0040] The fused image data is used to avoid smoothness issues in the image caused by directly switching the main camera. Because the first and second cameras are located in different positions on the electronic device, there is a positional deviation in the image data captured by these two cameras. Therefore, through spatial alignment and other algorithms in the image fusion process, the positional deviation between the fused image data and the second image data is gradually reduced, so as to ensure the smoothness of the image during the switching of the main camera.

[0041] The target image is at least one frame of image that needs to be displayed by the electronic device. The target image is obtained by adjusting image data under various conditions based on the current zoom level. Therefore, different image data is adaptively selected to generate the target image under different conditions to ensure smoothness of the image during the switching of the main camera. The conditions in which the target image is located include at least the case where the first camera is the main camera and the case where the second camera is the main camera; the case where the first camera is the main camera can be further refined to include the case where the first camera is the main camera and the second camera is not activated, and the case where the first camera is the main camera and the second camera is activated, so that the processing method of the target image can be refined, further increasing the smoothness of the image.

[0042] For example, the field of view can be determined based on the current zoom level, and the fused image data can be cropped according to the field of view to obtain cropped fused image data; the target image can then be generated based on the cropped fused image data. Alternatively, the fused image data can be scaled, cropped, or otherwise processed based on the current zoom level to obtain processed fused image data; the target image can then be generated based on the processed fused image data.

[0043] Step 306: If the comparison result indicates that the second camera is the main camera, obtain the target image based on the current zoom level and the second image data.

[0044] Because the zoom ratio changes gradually during the zoom process, at the first moment, the zoom ratio is within the focal range of the first camera, which is the primary camera. Image acquisition requires both the primary and secondary cameras, which are then activated and used for image fusion to obtain the fused image data. Then, at the second moment, when the zoom ratio is within the focal range of the second camera, the primary camera switching process is complete. The secondary camera can then be used independently for image acquisition, and the target image is obtained and displayed using the corresponding secondary image data. Simultaneously, after the primary camera switching process is complete, both the primary and secondary cameras can continue to acquire image data.

[0045] For example, the field of view can be determined based on the current zoom ratio, and the second image data can be cropped according to the field of view to obtain cropped second image data; the target image can then be generated based on the cropped second image data. Alternatively, the second image data can be scaled, cropped, or otherwise processed based on the current zoom ratio to obtain processed second image data; the target image can then be generated based on the processed second image data.

[0046] In the image processing method described above, the current zoom ratio is compared with the target optical zoom point zoom ratio to obtain a comparison result, which forms at least two scenarios. Since the target optical zoom point zoom ratio is greater than the original optical zoom point zoom ratio, when switching the main camera using the target optical zoom point zoom ratio, the moment when switching from a focal length with a smaller zoom ratio to a focal length with a larger zoom ratio is delayed. Therefore, the scenario where the comparison result indicates the first camera is the main camera is extended, allowing the first camera to acquire more first image data and the second camera to acquire more second image data. These two types of image data are then fused to obtain more fused image data. This results in more target images being obtained based on the current zoom ratio and the fused image data, thus increasing the smoothness of the image during the main camera switching process. Then, when the comparison result indicates the second camera is the main camera, the target image is obtained based on the current zoom ratio and the second image data, thereby completing the main camera switching process. Therefore, even in the event of probabilistic abnormal shifts or jitter, the smoothness of the image during the main camera switching process can be guaranteed.

[0047] Therefore, differentiated control optimizations were performed on the camera switching control strategy and the alignment and filtering strategies within the algorithm to achieve optimized switching transition control for this scenario. In this case, there is no need to excessively increase responsiveness, power consumption is not affected by activating too many cameras, zoom time is not increased, and the second camera in multi-camera platform switching is activated promptly with superior clarity. Thus, superior clarity ensures no image abruptness during switching, further enhancing the user's zoom experience.

[0048] In some embodiments, before comparing the current zoom ratio with the target optical zoom point to obtain the comparison result, the method further includes: in response to the zoom operation, obtaining the initial zoom ratio and the final zoom ratio; when the initial zoom ratio is at the focal length of the first camera and the final zoom ratio is at the focal length of the second camera, sequentially obtaining zoom ratios as the current zoom ratio from a zoom ratio sequence consisting of a preset number of initial zoom ratios and intermediate zoom ratios between the initial zoom ratio and the final zoom ratio.

[0049] Zooming refers to changing the focal length. Zooming can be a fixed-point zoom operation, meaning the user has a clearly defined final zoom level during the zoom process. Zooming can also be achieved through zoom controls on the camera's interface, which can be numbers, text, or icons representing multiple preset zoom levels. In this case, the zooming operation is a point-to-point operation.

[0050] Zooming can also refer to indeterminate continuous zooming, meaning there is no clear termination zoom level during the zooming process. By adjusting the zoom level and providing the zoomed image, the user can obtain a better preview or recording. Zooming can also be a zoom level that is slid to a specific target. Zooming can be implemented through touch or sliding zoom controls, which can also be implemented through zoom controls on the camera application interface. Zoom controls on the camera application interface can be discs, bars, or other shapes. In some possible implementations, zooming can also be achieved by pressing a button, sliding, or rotating a hardware device, such as configuring a rotatable device on an electronic device, and adjusting the zoom level by rotating this rotatable device; the rotatable device is a component capable of rotation. In this case, the zooming operation is a sliding operation.

[0051] The initial zoom ratio is the zoom ratio at which the zoom operation is triggered. The final zoom ratio is the zoom ratio at which the zoom process ends, as indicated by the zoom operation. For example, the zoom ratio at the time of detecting a zoom operation is used as the initial zoom ratio, while the zoom ratio indicated by the zoom operation is the final zoom ratio. During a zoom operation, the electronic device can reach the final zoom ratio.

[0052] When the initial zoom ratio is at the focal length of the first camera and the final zoom ratio is at the focal length of the second camera, the zoom operation involves a camera switching application scenario. A zoom ratio sequence can be constructed for this application scenario to obtain the current zoom ratio acquisition method, thereby ensuring the smoothness of the image switching.

[0053] A zoom ratio sequence is a set of zoom ratios arranged sequentially. This sequence is a collection of zoom ratio data, which can be a queue, stack, or other ordered data structure. The first zoom ratio output in the sequence is a preset number of initial zoom ratios. Because this preset number controls the delay in the zoom process, it allows the electronic device more time to activate the second camera, increasing the time when the first and second cameras are simultaneously active. The preset number can be two frames, three frames, or other values.

[0054] Intermediate zoom ratios are gradually changing transitional zoom ratios. Intermediate zoom ratios can be determined using zoom curves. Zoom curves are pre-configured zoom processing methods designed for smoothness during the zoom process. They can be expressed using different zoom curve functions or mapping tables, and these functions can be linear, power, exponential, etc.

[0055] Electronic devices can first acquire the zoom curve under point-cutting operation, and then use the zoom curve to obtain the intermediate zoom ratio between the starting zoom ratio and the ending zoom ratio, so as to realize the gradual change of the field of view of the image data, avoid the sudden change of the field of view (FOV), and make the image change smoother.

[0056] For example, in response to a zoom operation, the zoom ratio of the electronic device at the moment the zoom operation is triggered is obtained as the starting zoom ratio, and the zoom ratio indicated by the zoom operation is used as the ending zoom ratio. When the starting zoom ratio is in the focal length of the ultra-wide-angle camera and the ending zoom ratio is in the focal length of the telephoto camera, from the zoom ratio sequence consisting of the starting zoom ratio of the two frames and the intermediate zoom ratios of the multiple frames between the starting zoom ratio and the ending zoom ratio, the starting zoom ratio of the two frames is first obtained as the current zoom ratio, and after obtaining the starting zoom ratio of the two frames, the intermediate zoom ratios of the multiple frames are obtained sequentially.

[0057] In this embodiment, when the starting zoom ratio and the ending zoom ratio are directly obtained through zoom operation, the focal length of the camera at the two zoom ratios obtained directly through zoom operation is used to precisely indicate that the application scenario of this zoom operation involves camera switching. Therefore, a zoom ratio sequence is first constructed by using a preset number of starting zoom ratios and corresponding intermediate zoom ratios, and then the zoom ratios are obtained sequentially as the current zoom ratio. Thus, the preset number of starting zoom ratios are used as the current zoom ratios first, so that the change process of the current zoom ratio is delayed, thereby increasing the time when multiple cameras are activated, and thus increasing the smoothness of the image during the main camera switching process.

[0058] In some embodiments, the method further includes: generating an activation signal for the second camera when it is determined that the initial zoom ratio is at the focal length of the first camera and the final zoom ratio is at the focal length of the second camera; and controlling the second camera to be in an activated state according to the activation signal.

[0059] The activation signal is used to indicate that the second camera is turned on. The activation signal can be in the form of a voltage level, a voltage level change, or a signal representing data. The generation time of the activation signal, the formation time of the zoom ratio sequence, and the sequential acquisition of zoom ratios as the current zoom ratio do not necessarily have a sequential order.

[0060] For example, the ultra-wide-angle camera has a focal length of 0.6x and the telephoto camera has a focal length of 6x; when the starting zoom ratio is detected to be 0.6x and the ending zoom ratio is 6x, an activation signal for the telephoto camera is generated; based on the activation signal for the telephoto camera, the telephoto camera is controlled to perform power-on and startup processing until the startup process of the telephoto camera is completed, and the telephoto camera is kept in the image acquisition state to acquire second image data through the telephoto camera.

[0061] In this embodiment, the focal lengths of the cameras at the two zoom magnifications are efficiently obtained through zoom operation to determine the situation where the zoom operation involves camera switching; thus, the second camera activation signal is generated more promptly, and the second camera is activated and put into the on state by controlling the activation signal, so that the moment when the first camera and the second camera are activated simultaneously arrives quickly, thereby increasing the time when multiple cameras are activated, and thus more target images are generated simultaneously based on the image data after the fusion of the first image data and the second image data, thereby increasing the smoothness of the image during the main camera switching process.

[0062] In some embodiments, before fusing the first image data acquired by the first camera and the second image data acquired by the second camera to obtain the fused image data, the method further includes: obtaining a camera on / off status identifier; and determining that the first camera and the second camera are in an on / off state based on the camera on / off status indicated by the on / off status identifier.

[0063] The on / off status indicator is a parameter that represents whether a camera is in the on / off state. The on / off status indicator can be an active map; the camera's on / off state indicated by the indicator can be represented by the value of the active map. That is, the value of the active map represents whether the camera is in the zoom-in state. Therefore, different values ​​of the same parameter represent the on / off states of various cameras, ensuring processing efficiency.

[0064] The camera's on / off status can also be represented in binary, reducing the number of ways to express this status and thus improving processing efficiency. From least significant bit to most significant bit, the binary representations are: the ultra-wide-angle and wide-angle cameras on simultaneously; the wide-angle camera on alone; the telephoto camera on alone; and the ultra-wide-angle and telephoto cameras on simultaneously (UW, W, Tele, UTee). For example, an active map value of 3 corresponds to hexadecimal 0x0011, indicating that both the ultra-wide-angle and wide-angle cameras are on.

[0065] In this embodiment, the switch status indicator accurately determines whether the first camera and the second camera are in the on state, so as to more precisely control the timing when multiple cameras are turned on at the same time. Furthermore, by setting the concept of switch status indicator, the state of different cameras can be qualitatively analyzed in a targeted manner, thereby further refining the situation of the first camera as the main camera, forming the situation before the second camera is turned on and the situation after the second camera is turned on.

[0066] In some embodiments, before fusing the first image data acquired by the first camera and the second image data acquired by the second camera, the method further includes: when the second camera is not turned on, determining a pre-alignment offset based on the calibration parameters and focus distance between the first camera and the second camera; performing a position offset on the first image data acquired by the first camera according to the pre-alignment offset to obtain offset first image data; and obtaining a target image based on the current zoom ratio and the offset first image data.

[0067] The calibration parameters are parameters set for the first and second cameras. The calibration parameters include at least the internal parameters of the first and second cameras, as well as the external parameters for image acquisition by the first and second cameras. The calibration parameters may also include distortion parameters of the first and second cameras. For example, an internal parameter could be the focal length of the camera, and an external parameter could be the baseline length between the first and second cameras, where the baseline length is the distance between the optical centers of the first and second cameras.

[0068] The focusing distance is the distance between an electronic device and the subject being photographed. The electronic device can determine the focusing distance using first image data captured by a first camera. For example, the first image data can be substituted into a contrast detection method or a phase detection method to obtain the focusing distance.

[0069] Since the positional difference between the first and second cameras can be determined by calibration parameters such as baseline length, and the focal lengths of the first and second cameras are known and there is a focusing distance, the parallax of the shooting object between the first and second cameras can be determined according to the binocular ranging principle. This parallax is used to obtain the pre-alignment offset.

[0070] The pre-alignment offset is the offset when the second camera is not activated. The pre-alignment offset is used at least to offset the position of the first image data acquired by the first camera, so that the position of the subject in the first image data approximates its position in the fused data. The pre-alignment offset can also be used to adjust filter parameters to increase their accuracy, thereby ensuring image smoothness.

[0071] In one example, the first camera is an ultra-wide-angle camera, and the second camera is a telephoto camera. The pre-alignment offset is calculated to approximate the offset when image alignment is not possible. This allows the target image captured separately by the ultra-wide-angle camera to be magnified and offset simultaneously, ensuring a smooth transition with the fused image data. Without this magnification and offset, there would be significant discrepancies between the image obtained by centering and magnifying the target image captured separately by the ultra-wide-angle camera before the telephoto camera is activated, and the target image generated from the fused image data after the telephoto camera is activated.

[0072] In this embodiment, when the second camera is not turned on, the pre-alignment offset is determined by calibration parameters and focus distance. The pre-alignment offset can reflect the expected deviation between the first image data and the second image data. Therefore, before the second image data is acquired, the first image data is offset by the pre-alignment offset, so that the offset first image data is closer to the expected second image data. Therefore, when the second camera is turned on, the deviation between the offset first image data and the fused image data is relatively small, thereby making the target image when the second camera is not turned on have less positional deviation than the target image when the second camera is turned on, thus ensuring the smoothness of the image.

[0073] In some embodiments, the first image data acquired by the first camera is offset according to a pre-alignment offset to obtain offset first image data, including: determining the predicted value of the current frame according to a preset offset; fusing the predicted value of the current frame and the pre-alignment offset according to the Kalman filter gain corresponding to the current frame to obtain an adjusted pre-alignment offset; and offsetting the first image data acquired by the first camera according to the adjusted pre-alignment offset to obtain offset first image data.

[0074] The preset offset is at least one pre-set offset. The preset offset can be 0, in which case it is not used to adjust the pre-alignment offset. The preset offset can also be other values, used to fine-tune the pre-alignment offset of the current frame.

[0075] The current frame is the image frame that needs to be previewed, captured, or recorded for display. When the second camera is not turned on, the image data of the current frame includes the first image data of the current frame and the first image data after offset. When the second camera is turned on, the image data of the current frame includes the first image data and the second image data of the current frame, and may also include the fused image data of the current frame.

[0076] For example, when the second camera is not turned on, the i-th Kalman gain and the i-th covariance are obtained; based on the conversion relationship between the covariance and the Kalman gain, and the i-th covariance, the (i+1)-th Kalman gain is determined; based on the i-th Kalman gain, the k-th pre-alignment offset and the preset offset are adjusted to obtain the (i+1)-th adjusted pre-alignment offset; where i is a positive integer; and the first Kalman gain and the first covariance can be preset or adjusted in real time. Thus, by continuously adjusting the Kalman gain, the accuracy of the gain is increased, thereby ensuring image smoothness.

[0077] After obtaining the (i+1)th adjusted pre-alignment offset, the i-th covariance can be adjusted according to the (i+1)th Kalman gain to obtain the (i+1)th covariance, thereby determining the (i+1)th Kalman gain and the (i+2)th adjusted pre-alignment offset. The Kalman gain is adjusted cyclically in this way to increase the accuracy of the gain and thus ensure the smoothness of the image.

[0078] In this embodiment, on the one hand, Kalman filtering is applied to the pre-alignment offset to increase the smoothness of the current image when the second camera is not activated; on the other hand, the Kalman gain is adjusted by the pre-alignment offset to increase the accuracy of the filter parameters, thereby ensuring the smoothness of subsequent images.

[0079] In some embodiments, before obtaining the target image based on the current zoom ratio and the second image data, the method further includes: determining the predicted value of the current frame based on the initial feature offset; fusing the predicted value of the current frame and the initial feature offset based on the Kalman filter gain corresponding to the current frame to obtain the feature offset of the current frame; and performing a position offset on the second image data based on the feature offset of the current frame to obtain the offset second image data.

[0080] Correspondingly, the target image is obtained based on the current zoom ratio and the second image data, including: obtaining the target image based on the current zoom ratio and the offset second image data.

[0081] The initial feature offset is the offset obtained by image registration in the current frame. Since the initial feature offset is obtained by feature matching based on the first and second image data of the current frame, both the first and second cameras are turned on when the initial feature offset is obtained.

[0082] For example, with the second camera as the main camera and the first camera not turned off, the j-th Kalman gain and the j-th covariance are obtained; based on the conversion relationship between the covariance and the Kalman gain, and the j-th covariance, the (j+1)-th Kalman gain is determined; the k-th initial offset is used as the prediction value of the current frame; based on the j-th Kalman gain, the k-th initial offset and the prediction value of the current frame are adjusted to obtain the (j+1)-th adjusted initial offset; where j is a positive integer; and the first Kalman gain and the first covariance can be preset or adjusted in real time. Thus, by continuously adjusting the Kalman gain, the accuracy of the gain is increased, thereby ensuring image smoothness.

[0083] After obtaining the (j+1)th adjusted initial offset, the jth covariance can be adjusted according to the (j+1)th Kalman gain to obtain the (j+1)th covariance, thereby determining the (j+1)th Kalman gain and the (j+2)th adjusted initial offset. This process is repeated to adjust the Kalman gain, thereby increasing the accuracy of the gain and ensuring the smoothness of the image.

[0084] In this embodiment, with the second camera as the main camera and both the first and second cameras turned on, image registration is performed on the current frame to obtain the initial feature offset, clarify the feature deviation between the first image data and the second image data, and adjust the second image data accordingly. Therefore, after the first camera is turned off, the deviation between the fused image data and the offset second image data is relatively small, thereby ensuring the smoothness of the image.

[0085] In some embodiments, comparing the current zoom ratio with the target optical zoom point zoom ratio to obtain a comparison result includes: if the current zoom ratio is less than the target optical zoom point zoom ratio, determining the main camera identifier as a first main camera identifier value, the first main camera identifier value being used to indicate that the first camera is the main camera; if the current zoom ratio is greater than or equal to the target optical zoom point zoom ratio, determining the main camera identifier as a second main camera identifier value, the second main camera identifier value being used to indicate that the second camera is the main camera.

[0086] The main camera identifier is a parameter of the main camera, which can be represented by the master camera ID. The first main camera identifier value and the second main camera identifier value are two different parameter values ​​for the main camera identifier, and these two parameter values ​​can be represented by different numbers, letters, or characters. For example, a first main camera identifier value of true indicates that the first camera is the main camera, and a second main camera identifier value of false indicates that the second camera is the main camera; or, a first main camera identifier value of 1 indicates that the first camera is the main camera, and a second main camera identifier value of 2 indicates that the second camera is the main camera.

[0087] In this embodiment, under different comparisons between the current zoom ratio and the target optical zoom point zoom ratio, the different comparison results are quantified into values ​​for the main camera identifier. Then, using the first and second main camera identifier values, one of the first and second cameras is selected as the main camera. Since the focal length of the larger current zoom ratio is longer, a relatively larger target optical zoom point zoom factor is used to delay the switching time from the main camera to the second camera during the transition from a smaller to a larger zoom ratio, thus ensuring smooth image quality.

[0088] In some embodiments, as Figure 4 As shown, the first image data captured by the first camera and the second image data captured by the second camera are fused to obtain fused image data, including:

[0089] Step 402: Determine the first image data captured by the first camera and the second image data captured by the second camera.

[0090] For example, based on the target time, first image data acquired by the first camera at the target time and second image data acquired by the second camera at the target time can be determined, so as to determine the first image data and the second image data from the image data in the cache. Alternatively, based on the current time, first image data acquired by the first camera at the current time and second image data acquired by the second camera at the current time can be determined, so as to determine the first image data and the second image data from the image data acquired in real time.

[0091] Step 404: Determine the feature offset of the current frame based on the feature offset between the first image data and the second image data.

[0092] Feature offset is the positional deviation of the same object in the first image data and the second image data. The feature offset is determined for the first and second image data of the current frame. The feature offset can be a feature offset itself or it can be obtained based on feature matching of the same object in the first and second image data. This feature can be a feature contained in multiple pixels, multiple image regions, or multiple parts of the object. For example, the feature offset includes the feature deviation of the image on the x-axis and the feature deviation of the image on the y-axis, i.e., offset_x and offset_y.

[0093] The current frame feature offset is the feature offset applied to the current moment. The current frame feature offset is determined at least based on the feature offset, and can also be determined based on the offset at the current moment and offsets prior to the current moment. Alignment can be performed based on the feature offsets of multiple frames of first and second image data, and the results of multiple alignments can be filtered to reduce the error of the current frame feature offset, thereby more accurately reducing the deviation between the first and second image data and ensuring smoothness.

[0094] For example, feature matching is performed on the first image data and the second image data of the current frame to obtain a second feature that matches the first feature of the current frame; the first feature is a feature in the first image data of the current frame, and the second feature is a feature in the second image data of the current frame; based on the pixel offset between the first feature and the second feature, an initial feature offset between the first image data and the second image data of the current frame is determined; based on the initial feature offset, the feature offset of the current frame is determined.

[0095] Step 406: Based on the current frame feature offset, fuse the first image data and the second image data to obtain the fused image data.

[0096] In one example, the pixel data in the first image data and the second image data are adjusted according to the current frame feature offset to obtain the adjusted first image data and the adjusted second image data; for each coordinate position, the adjusted first image data and the adjusted second image data are fused to obtain the fused image data.

[0097] In this embodiment, the feature offset of the current frame is determined based on the feature offset between the first image data and the second image data, so that the feature offset of the current frame is related to each frame itself, forming an effect similar to offset value measurement; then, the first image data and the second image data are fused according to the feature offset of the current frame to obtain fused image data, so that the difference between the fused image data and the second image data is reduced.

[0098] In some embodiments, determining the feature offset of the current frame based on the feature offset between the first image data and the second image data includes: determining the initial feature offset of the current frame based on the feature offset between the first image data and the second image data; converting the initial feature offset based on the offset conditions satisfied by the reference frame and the current frame to obtain the feature offset of the current frame; the reference frame is at least one frame of image data processed before the current frame.

[0099] The initial feature offset is the offset obtained by image registration in the current frame. Since the initial feature offset is obtained by feature matching based on the first and second image data of the current frame, both the first and second cameras are in the active state when the initial feature offset is obtained. The initial feature offset can be the feature offset between the first and second image data of the current frame, or it can be obtained by fine-tuning the feature offset between the first and second image data of the current frame.

[0100] A reference frame is image data whose offset is determined before the current frame and adjusted based on the offset of the reference frame. The reference frame can be the frame preceding the current frame; that is, the reference frame is a frame of image data from the previous moment, and the current frame is a frame of image data from the current moment. A preset frame interval can exist between the reference frame and the current frame, which can be 0 frames or 1 frame. The offset of the reference frame has at least two types: one is a pre-aligned offset, and the other is a type that does not use pre-aligned offsets. The type that does not use pre-aligned offsets includes the reference frame feature offset. The reference frame feature offset can be the feature offset between the first and second image data of the reference frame, or it can be the result of adjusting the feature offset between the first and second image data of the reference frame. In this case, the offset type of the reference frame forms a dimension of the offset condition, allowing for a more detailed classification of different conversion methods.

[0101] Offset conditions are based on at least two dimensions: one is the offset type of the reference frame, and the other is the offset type to be used in the current frame. Offset conditions determine the transformation method for the initial feature offset. For the offset conditions satisfied by the reference frame and the current frame, the corresponding transformation method for the initial feature offset can be determined.

[0102] In this embodiment, when both the first and second cameras are turned on, image registration is first performed to obtain the initial feature offset. Then, the conversion method of the initial feature offset is adaptively controlled through the current frame and the reference frames before it, so as to more accurately determine the feature offset of the current frame in different refined cases.

[0103] In some embodiments, the initial feature offset is transformed based on the offset conditions satisfied by the reference frame and the current frame to obtain the feature offset of the current frame, including: when the reference frame is offset based on the pre-aligned offset and the current frame is not offset based on the pre-aligned offset, the feature offset of the current frame is obtained based on the initial feature offset.

[0104] When the reference frame is offset based on a pre-alignment offset, and the current frame is not offset based on a pre-alignment offset, the reference frame is based solely on image data captured by the first camera, while the current frame is composed of image data captured by both the first and second cameras. In this case, since the pre-alignment offset does not involve the concept of an initial feature offset, directly using the pre-alignment offset to adjust the Kalman gain may cause ghosting issues in the target image of the current frame. Ghosting refers to the problem of multiple overlapping effects on the same subject. Therefore, the feature offset of the current frame obtained based on the initial feature offset is more accurate in this situation.

[0105] The reference frame offset based on the pre-alignment offset means that, when the second camera is not turned on, the pre-alignment offset is determined based on the calibration parameters and focus distance between the first and second cameras; according to the pre-alignment offset, the position of the first image data acquired by the first camera is offset to obtain the offset first image data; when the offset first image data is in the same zoom process as the current frame, the offset first image data is the reference frame of the current frame.

[0106] The current frame not being offset based on the pre-aligned offset means that both the first and second cameras are on, capable of acquiring the first and second image data. In this case, not directly using the Kalman gain adjusted by the reference frame to process the initial feature offset can actually increase accuracy, and when the current frame is used as the reference frame for subsequent frames, the filtering effect of subsequent frames is more accurate.

[0107] In one example, when the reference frame is offset based on the pre-aligned offset and the current frame is not offset based on the pre-aligned offset, the initial feature offset can be used as the feature offset of the current frame. Alternatively, the initial feature offset can be adjusted by preset parameters to obtain the feature offset of the current frame.

[0108] In this embodiment, when the reference frame is offset based on the pre-aligned offset and the current frame is not offset based on the pre-aligned offset, the initial feature offset is not processed directly using the Kalman gain adjusted by the reference frame. This can increase accuracy, avoid the problem of multiple overlapping effects of the same subject in the current frame, and when the current frame is used as the reference frame for subsequent frames, the filtering effect of subsequent frames is more accurate.

[0109] In some embodiments, the initial feature offset is transformed based on the offset conditions satisfied by the reference frame and the current frame to obtain the feature offset of the current frame, including: when neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame, the feature offset of the current frame is obtained based on the initial feature offset.

[0110] The reference optical zoom ratio is at least one optical zoom ratio between the focal lengths of different cameras. The reference optical zoom ratio is less than or equal to the target optical zoom ratio. During a single zoom cycle, each time the reference optical zoom ratio is passed, the current frame feature offset is obtained based on the initial feature offset. When the first camera and the second camera are each a single camera, the reference optical zoom ratio is the target optical zoom ratio. When multiple cameras exist (at least one of the first and second cameras), the reference optical zoom ratio includes both the target optical zoom ratio and the zoom ratios between the multiple cameras.

[0111] When neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame, the reference frame and the current frame are based on image data collected by different cameras. Since the positions of different cameras are different, the original filtering method will cause the current frame to be inaccurate. Therefore, the feature offset of the current frame obtained based on the initial feature offset is more accurate.

[0112] The fact that neither the reference frame nor the current frame is offset based on the pre-alignment offset means that the fused image data and the current frame are in the same zoom process, and there is a reference zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame. In this case, the reference frame has its own fused image data, and the current frame may have its own fused image data, or it may have a second image data captured by the second camera alone.

[0113] In this embodiment, when neither the reference frame nor the current frame is offset based on the pre-alignment offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame, the feature offset of the current frame can be obtained directly based on the initial feature offset, which can increase the accuracy and avoid the problem of multiple overlapping effects of the same shooting object in the current frame. Furthermore, when the current frame is used as the reference frame for subsequent frames, the filtering effect of subsequent frames is more accurate.

[0114] In other embodiments, the initial feature offset is transformed based on the offset conditions satisfied by the reference frame and the current frame to obtain the feature offset of the current frame, including: determining the predicted value of the current frame based on the offset corresponding to the offset conditions satisfied by the reference frame and the current frame; and fusing the predicted value of the current frame and the initial feature offset according to the Kalman filter gain corresponding to the current frame to obtain the feature offset of the current frame between the first image data and the second image data.

[0115] The predicted value is the offset between the first image data and the second image data; the predicted value is a dimension used to determine the feature offset of the current frame. This dimension is adaptively adjusted based on the reference frame and the current frame. For the zoom situation that is refined according to the offset conditions satisfied by these two frames, the predicted value under the refined zoom situation can be determined by the offset corresponding to the offset conditions, so as to more accurately determine the corresponding feature offset of the current frame.

[0116] The Kalman filter gain is the fusion weight corresponding to the current frame. The Kalman filter gain can be determined based on the prior error of the current frame in the reference frame, or it can be preset. In some cases, determining the Kalman filter gain of the current frame based on the prior error of the reference frame allows for adaptive adjustment of the Kalman filter gain, resulting in higher accuracy. In other cases, a preset Kalman filter gain can prevent the Kalman filter gain from being affected by the external environment, thus leading to higher accuracy.

[0117] The method for determining the predicted value of the current frame may include: under the condition that the offset conditions satisfied by the reference frame and the current frame represent normal filtering, converting the feature offset of the previous frame into the predicted value of the current frame based on the state prediction equation of the Kalman filter; the image data of the previous frame belongs to one of the reference frames; the feature offset of the previous frame is determined based on the predicted value of the previous frame and the initial feature offset of the previous frame, and the initial feature offset of the previous frame is obtained by image registration based on the first image data and the second image data of the previous frame.

[0118] The process of fusing the predicted value and initial feature offset of the current frame based on the Kalman filter gain corresponding to the current frame includes: determining the difference between the predicted value and the initial feature offset of the current frame to obtain a difference deviation; adjusting the difference deviation based on the Kalman filter gain corresponding to the current frame to obtain an adjusted difference deviation; and fusing the predicted value of the current frame with the adjusted difference deviation. The difference deviation can be the difference between the predicted value and the initial feature offset of the current frame, the result of adjusting the predicted value of the current frame by the observer matrix, or the difference between the predicted values ​​of the current frame.

[0119] In this embodiment, the predicted value of the current frame in the Kalman filtering process is first determined by the offset corresponding to the offset condition that has been met. Then, the Kalman filter gain corresponding to the current frame is used to perform weighted fusion on the predicted value and the initial feature offset obtained by image registration, so as to obtain the feature offset of the current frame after Kalman filtering. After weighted fusion using the Kalman filter, the feature offset of the current frame can be made relatively accurate, thereby ensuring the smoothness of the image.

[0120] In some embodiments, determining the predicted value of the current frame based on the offset corresponding to the offset conditions satisfied by the reference frame and the current frame includes: determining the predicted value of the current frame based on the initial feature offset when the reference frame and the current frame meet the feature offset assignment conditions.

[0121] The feature offset assignment condition is a type of offset condition, and the offset corresponding to the feature offset assignment condition is the initial feature offset. When the feature offset assignment condition is met, the predicted value of the current frame is determined based on the initial feature offset. For example, the initial feature offset can be used as the predicted value of the current frame. Alternatively, the conditions can be further refined, and the initial feature offset can be adjusted according to the parameters corresponding to the refined conditions to obtain the predicted value of the current frame.

[0122] In this embodiment, the predicted value of the current frame is determined based on the initial feature offset, thus adjusting the normal filtering process of the Kalman filter. In this case, the predicted value of the current frame is not determined based on the offset finally estimated from the reference frame, but rather based on the initial feature offset. Therefore, under the condition of meeting the feature offset assignment conditions, the accuracy of Kalman gain, covariance, or other filter parameters can be guaranteed, thereby ensuring the smoothness of subsequent images.

[0123] In some embodiments, the feature offset assignment condition includes offsetting one of the reference frame and the current frame based on a pre-aligned offset.

[0124] The assignment conditions for feature offsets include two cases: one is that the reference frame is offset based on the pre-aligned offset, and the current frame does not need to be offset based on the pre-aligned offset; the other is that the reference frame is not offset based on the pre-aligned offset, and the current frame needs to be offset based on the pre-aligned offset.

[0125] When the reference frame is not offset based on the pre-alignment offset, and the current frame needs to be offset based on the pre-alignment offset, the steps performed by the electronic device include: when the second camera is not turned on, determining the predicted value of the current frame according to the preset offset; fusing the predicted value of the current frame and the pre-alignment offset according to the Kalman filter gain corresponding to the current frame to obtain the adjusted pre-alignment offset; offsetting the position of the first image data acquired by the first camera according to the adjusted pre-alignment offset to obtain the offset first image data; and obtaining the target image according to the current zoom ratio and the offset first image data.

[0126] When the reference frame is not offset based on the pre-aligned offset, and the current frame needs to be offset based on the pre-aligned offset, the steps performed by the electronic device include: with the second camera as the main camera and the first camera remaining on, determining the predicted value of the current frame based on the initial feature offset; fusing the predicted value of the current frame and the initial feature offset based on the Kalman filter gain corresponding to the current frame to obtain the feature offset of the current frame; offsetting the position of the second image data based on the feature offset of the current frame to obtain the offset second image data; correspondingly, obtaining the target image based on the current zoom ratio and the second image data includes: obtaining the target image based on the current zoom ratio and the offset second image data.

[0127] When offsetting is based on pre-aligned offsets, and the current frame does not require offsetting based on pre-aligned offsets, the two initial feature offsets of the current frame are fused according to the Kalman filter gain corresponding to the current frame to obtain the current frame feature offset between the first image data and the second image data. For example: determine the difference between the initial feature offsets and the initial feature offsets of the current frame to obtain the difference deviation, which can be 0; adjust the difference deviation according to the Kalman filter gain corresponding to the current frame to obtain the adjusted difference deviation, which can also be 0; and fuse the predicted value of the current frame with the adjusted difference deviation.

[0128] In this embodiment, when the reference frame is offset based on the pre-alignment offset and the current frame does not need to be offset based on the pre-alignment offset, and when the reference frame is not offset based on the pre-alignment offset and the current frame needs to be offset based on the pre-alignment offset, the prediction value of the current frame is determined based on the initial feature offset, which can make the smoothness of the image higher.

[0129] In some embodiments, the feature offset assignment conditions include that neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame.

[0130] The condition for assigning the feature offset can also be that neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame.

[0131] When neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame, the reference frame and the current frame are based on image data collected by different cameras. Since the positions of different cameras are different, the original filtering method may cause inaccuracies in the current frame. Therefore, obtaining the predicted value of the current frame based on the initial feature offset can make the predicted value of the current frame more accurate, and also make the feature offset of the current frame more accurate.

[0132] In this embodiment, when neither the reference frame nor the current frame is offset based on the pre-alignment offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame, the predicted value of the current frame can be obtained directly based on the initial feature offset. This can increase accuracy, avoid the problem of multiple overlapping effects of the same shooting object in the current frame, and when the current frame is used as the reference frame for subsequent frames, the filtering effect of subsequent frames is more accurate, resulting in higher smoothness of the image.

[0133] In some embodiments, determining the predicted value of the current frame based on the offset corresponding to the offset conditions satisfied by the reference frame and the current frame includes: determining the predicted value of the current frame according to the preset offset when the reference frame and the current frame meet the preset value assignment conditions.

[0134] The preset value assignment condition is a type of offset condition, and the offset corresponding to the preset value assignment condition is the preset offset. When the preset value assignment condition is met, the predicted value of the current frame is determined based on the preset offset. For example, the preset offset can be used as the predicted value of the current frame, or the corresponding conditions can be further refined, and the preset offset can be adjusted according to the parameters corresponding to the refined conditions to obtain the predicted value of the current frame.

[0135] In this embodiment, the predicted value of the current frame is determined based on a preset feature offset, thus adjusting the normal filtering process of the Kalman filter. In this case, the predicted value of the current frame is not determined based on the offset finally estimated from the reference frame, but rather based on the preset feature offset. Therefore, under the condition of meeting the preset value assignment conditions, the accuracy of Kalman gain, covariance, or other filter parameters can be guaranteed, thereby ensuring the smoothness of subsequent images.

[0136] In some embodiments, the preset value assignment conditions include that neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is no reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame.

[0137] When neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is no reference optical zoom point between the zoom magnification of the reference frame and the zoom magnification of the current frame, the reference frame and the current frame are based on image data captured by the same camera. However, the original filtering method may cause inaccuracies in the current frame. Therefore, obtaining the predicted value of the current frame based on the preset offset can make the predicted value of the current frame more accurate and the feature offset of the current frame more accurate.

[0138] In this embodiment, when neither the reference frame nor the current frame is offset based on the pre-alignment offset, and there is no reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame, the predicted value of the current frame can be determined directly based on the preset offset, which can increase the accuracy and make the image smoother.

[0139] In one exemplary embodiment, such as Figure 5 As shown, in the case of a point-to-curve, in response to the zoom operation, the current zoom ratio gradually changes from 0.6x to 6x according to the zoom curve. At this time, the first camera is an ultra-wide-angle camera (UW), and the second camera is a telephoto camera (Tele). The target optical zoom point zoom ratio and the reference optical zoom point zoom ratio of both are the same value, which is 3.5x. There are three stages at this time: In the first stage 501, that is, the interval from frame 1 to frame 5, the main camera is the ultra-wide-angle camera, and the telephoto camera is in an off state; In the second stage 502, that is, the interval from frame 5 to frame 10, both the ultra-wide-angle camera and the telephoto camera are on, and the current zoom ratio is less than the target optical zoom point zoom ratio, so step 304 is executed; Next, in the third stage 503, that is, the interval from frame 10 to frame 20, the second camera becomes the main camera, so image data is acquired at least based on the second camera, and the first camera can also be used as an auxiliary camera for image data acquisition. Figure 5 The current zoom levels are shown in Table 1. The frame indices are arranged in chronological order, as shown in Table 1 below:

[0140] Table 1

[0141] Frame Index Current zoom ratio 1 0.6 2 0.6 3 0.6 4 0.76 5 1.03 6 1.42 7 1.87 8 2.32 9 2.78 10 3.49 11 4.06 12 4.51 13 4.87 14 5.16 15 5.39 16 5.57 17 5.72 18 5.83 19 5.91

[0142] In one exemplary embodiment, such as Figure 6As shown, the method in this embodiment is divided into camera control at the Hardware Abstraction Layer (HAL) and alignment and filtering strategies in the algorithm. The Algorithm Processing Layer (APS) issues point-cutting instructions, as well as the starting and ending magnifications of the point cuts, based on the acquired scene information. The HAL issues intermediate magnifications and switches cameras based on the instructions issued by the upper layer. The SAT algorithm performs field-of-view cropping and alignment offset calculations based on the real-time magnification, active map status indicator, and master camera information issued by the HAL, achieving a smooth transition of the image when switching the master camera during zooming.

[0143] In this implementation, the camera application (APP) sends a request to the framework layer, which then forwards the request to the hardware abstraction layer (HAL). The HAL controls the camera drivers, which in turn turn the cameras on and off. The cameras include Sensor1, Sensor2, and Sensor3. Simultaneously, the HAL executes the main camera switching process and sends the current zoom ratio, active map, and masterID to the spatial alignment module (SAT) to perform image fusion and obtain target images under various conditions. Furthermore, the HAL can interact with the algorithm processing layer.

[0144] In one embodiment, firstly, a camera start / stop and switching control scheme is formed through steps 302-306 above. This scheme is mainly for camera control mechanisms in zoom scenarios, and its essential purpose is to ensure the smoothness and fluidity of image changes during zooming, minimizing image jumps during zooming. This embodiment, from the perspective of ensuring the number of frames in dual-camera operation (i.e., the number of frames with both main and secondary cameras operating simultaneously), designs the camera start / stop timing and the optical zoom point magnification for switching cameras in the control strategy. This ensures sufficient dual-camera operation for the spatial alignment algorithm to calculate the alignment deviation and convergence deviation errors, thus ensuring a smoother target image. Secondly, through steps 402-406 above and corresponding embodiments, the improved Kalman filtering strategy ensures that the deviation calculated by the spatial alignment algorithm is not affected by previous alignment during continuous point cutting, ensuring the accuracy of the current alignment and solving the ghosting problem caused by inaccurate filtering of the deviation.

[0145] In one exemplary embodiment, such as Figure 7 As shown, it includes steps 601-607, as well as steps for connecting other methods:

[0146] Step 601: First, it is necessary to identify whether the current scene is a zoom scene with a starting magnification of 0.6x and a ending magnification of 6x. If it is such a zoom scene, then this strategy mechanism will be activated.

[0147] Step 602: The hardware abstraction layer sends two frames of 0.6x as the current zoom ratio, and then sends the changing zoom ratio, which can be as follows: Figure 5 The zoom ratio shown.

[0148] Step 603: Simultaneously, the Hardware Abstraction Layer (HAL) sends a camera activation signal to activate the telephoto camera. It's important to note that the camera activation process takes approximately five frames; the HAL must not delay sending the camera activation signal, ensuring the telephoto camera activates on the 5th frame of the transition frame.

[0149] Step 604: The spatial alignment algorithm receives the current zoom ratio, main camera identifier, and on / off status identifier from the hardware abstraction layer, processes them accordingly, and outputs the current zoom ratio. Specifically, Step 604 includes: Step 6041: Determine if the telephoto camera is activated. If not activated, proceed to Step 6042: Calculate the pre-alignment deviation based on the calibration parameters and the focus distance from the hardware abstraction layer. If activated, proceed to Step 6043: Perform feature matching using the images from the ultra-wide-angle camera and the telephoto camera to calculate the alignment deviation and obtain the current frame feature offset.

[0150] Step 605: The hardware abstraction layer determines whether the current zoom ratio reaches the target optical zoom ratio (e.g., 3.50x). If not, it continues to send the current zoom ratio and the unchanged main camera identifier to the spatial alignment algorithm to continue executing step 604. If the current zoom ratio is greater than the optical zoom ratio of 3.50x, it switches the main camera to the telephoto camera, sends the information to the spatial alignment algorithm again, and continues zooming. Thus, from frame 5 to frame 10 in the transition frames, 6 frames are used for spatial alignment by the algorithm, converging the alignment deviation. Step 605 corresponds to steps 302-306 above.

[0151] Step 607: After the spatial alignment judgment of the main camera switches to the telephoto camera, stop the feature matching between UW and the telephoto camera, and only calculate the cropping of the telephoto camera matching the current zoom ratio. When the hardware abstraction layer judges that the current zoom ratio has reached the termination ratio of 6.0, it sends the current zoom ratio = 6.0 and keeps it unchanged, ending the zoom process.

[0152] In one embodiment, steps 402-406 are further defined as involving improvements to the Kalman filter. The Kalman filter principle includes a motion equation and a measurement equation. The motion equation is used to determine the predicted value at the current time based on the final estimated value at the previous time. The measurement matrix is ​​used to obtain the measurement value. In the above embodiment, the initial feature offset is the measurement value in the Kalman filter, that is, the result of image registration is the measurement value in the Kalman filter.

[0153] The equation of motion for the Kalman filter is: x k =A k x k-1 +B k u k +ω k

[0154] The measurement equation for a Kalman filter is: Z k =C k x k +v k

[0155] Where, matrix A k Let matrix B be the transition matrix. k =I is the control matrix, and matrix C k This is the observation matrix. I is an identity matrix consisting of 1s; u k It is the control input vector, ω k This is noise, which is assumed to be 0. The motion equations and measurement equations of the Kalman filter form a set of formulas. In this set of formulas, besides the predicted value x at the current moment... k Compared with the predicted value x at the previous time step k-1 All other parameters are preset; and parameters such as covariance in other formulas are updated in real time.

[0156] In this system, all states and noise follow a Gaussian distribution. Process noise and measurement noise are included in the measurement data.

[0157] The expression for the Gaussian distribution of process noise is: ω k ~N(0,Q) k )

[0158] The expression for the Gaussian distribution of the measured noise is: v k ~N(0, R) k )

[0159] In this embodiment, the above-mentioned Kalman filter principle is adjusted and normal filtering is performed: estimating the state of a linear Gaussian system using a Kalman filter involves two steps:

[0160] The first step is the prediction step, and its expression is as follows:

[0161]

[0162] First, based on the system's prediction method, the final estimated value of the (k-1)th frame is substituted into the formula to calculate the predicted value of the kth frame. Predicted value of frame k This is used to obtain the final estimated value of the k-th frame in conjunction with the corresponding initial feature offset, i.e., the feature vector of the current frame after Kalman filtering of the k-th frame; and since the error is unknown, it is assumed to be 0; in addition, based on the covariance of the previous state... Calculate prior error: covariance

[0163] The second step is to calculate the Kalman gain k. k The final estimated value of the k-th frame is determined, which is the feature offset of the current frame and the updated covariance. The steps are expressed as follows:

[0164]

[0165] Among them, H k It is a linear transformation, R k This is measurement noise. The final estimate is updated based on the predicted and measured values. In the formula z k This is the measured value, i.e., the initial feature offset. For a Kalman filter, if the initial feature offset of the current frame is used as the prediction value, then the current measured value is the final estimated value; if a preset offset, such as 0, is used as the prediction value of the current frame, then the filter performs normal filtering.

[0166] In one example, the spatial alignment algorithm calculates the alignment offset based on image feature point matching. This offset includes an x-axis offset (offset_x) and a y-axis offset (offset_y). Since this is calculated for each frame, the algorithm applies Kalman filtering to the offsets to ensure a smooth transition between the time-sequential frames. Specifically, as shown... Figure 8 As shown, the calculation method for the predicted value of the current frame is further refined for different cases of the reference frame and the current frame:

[0167] Determine whether the pre-alignment deviation is applied in the current frame and the previous frame. If the pre-alignment deviation is applied in both frames, it indicates that there is no double opening. Set the prediction value to 0 and input it into the Kalman filter. That is, perform Kalman filtering based on the current pre-alignment deviation. That is, execute step 801.

[0168] If the previous frame applied a pre-alignment deviation, the current frame does not need a pre-alignment deviation. Instead, the alignment deviation (measured value) obtained by matching image feature points is calculated, indicating that the current frame is the first frame of dual camera operation. The measured value is then used as the predicted value and input into the Kalman filter. The image offset is directly performed using the feature point matching deviation, i.e., step 802 is executed.

[0169] If the pre-alignment deviation was not applied in the previous frame, but is applied in the current frame, it indicates that the double opening is over. The image offset is then directly performed using the pre-alignment deviation, i.e., step 802 is executed.

[0170] If no pre-alignment deviation is applied in either the previous or current frame, it indicates that the image is in a double-open state. However, it's possible that the image is in a point-cutting state where no deviation was applied in the previous frame (i.e., no image offset was made), and no pre-alignment deviation is applied in the current frame. This indicates that the image feature point matching deviation is being calculated for the first time after the light change point, and it is then used as the predicted value input to the Kalman filter, i.e., step 802 is executed. If the conditions are not met, normal filtering is performed, i.e., step 801 is executed.

[0171] If no pre-alignment offset is applied in either the previous or current frame, it indicates that the frame is in a double-open state. However, it may be in an undefined point-cut state, such as a 1x point cut followed by a 0.6x point cut and then immediately a 6x point cut. If Kalman filtering is continuously applied during these two point cuts, the 0.6x point cut to 6x will be affected by the 1x point cut to 0.6x, resulting in misalignment and ghosting. Therefore, the predicted value is also set to 0 and input into the Kalman filter, i.e., step 802 is executed.

[0172] In one embodiment, the above steps are a transition optimization scheme for a 6X zoom transition scenario on a 3-camera platform. Because this scenario is characterized by limited intermediate zoom magnification, time required for the secondary camera to activate, and the need for sharpness convergence, a differentiated control optimization is performed on the camera switching control strategy and the alignment and filtering strategies within the algorithm to achieve optimized transition control for this scenario. Compared to other optimization schemes, this embodiment does not require significantly increasing responsiveness, does not require activating multiple cameras to avoid impacting power consumption, and does not increase zoom duration. It effectively optimizes the 6X zoom transition based on current zoom basic strategies and schemes, solving the problem of delayed Tele activation or poor sharpness causing transitions in a 6X zoom transition scenario on a 3-camera platform, further improving the user's zoom experience.

[0173] In this scenario, to address the issue of insufficient double-frame output during a 0.6x point-to-6x zoom transition, the changed zoom ratio is sent after a two-frame delay. Furthermore, the original 3x optical zoom ratio at the optical zoom point is delayed to between 3x and 3.5x, thus increasing the double-frame output by at least 2-3 frames. This significantly improves the smoothness of the image during the 0.6x point-to-6x transition without noticeable lag in responsiveness. Simultaneously, to address the issue of inaccurate spatial alignment filtering due to previous deviations in consecutive 0.6x double-frame transitions, the spatial alignment filtering algorithm's filtering strategy is optimized. When the current zoom ratio is 0.6x and no alignment deviation is applied to the current frame, the initial feature deviation calculated based on the current frame's image features is used as the predicted value input to the Kalman filter for filtering. This effectively eliminates ghosting in the 0.6x point-to-6x transition scenario.

[0174] It's understandable that the aforementioned control strategy and optimized filtering scheme can also be applied to triple-camera smartphones with a 1x-to-6x switching mode, where the first camera is a wide-angle lens and the second is a telephoto lens. However, for the 1x-to-3x switching process, although there's no cross-camera focal length switching between main cameras, when the main camera is a wide-angle lens, the telephoto lens won't be activated to save power. Therefore, during the 1x-to-3x switching process, when the telephoto lens isn't activated, the algorithm can only utilize the pre-alignment deviation. Furthermore, the telephoto lens may activate slowly, leading to insufficient frame rates for both the wide-angle and telephoto lenses, resulting in ghosting or abrupt changes in the image. Applying the proposed solution can stably increase the frame rate for dual-camera setups, ensuring the alignment quality of the spatial alignment algorithm.

[0175] It is understandable that the improved Kalman filtering strategy proposed in this application can be applied to continuous point cuts at all magnifications. During continuous point cuts, the camera remains on, so the feature alignment deviations calculated in multiple different point cuts will affect each other in the Kalman filter, resulting in ghosting problems due to inaccurate deviations. Applying the proposed improved Kalman filtering strategy can avoid the influence of previous alignments during continuous point cuts, ensuring the accuracy of the current alignment and solving the ghosting problem caused by inaccurate deviation filtering.

[0176] Furthermore, based on the optimized scenario targeted in this embodiment (e.g., 0.6X point-to-6X), multiple point-to-point zoom operations are performed on the rear triple-camera platform, and screen recordings are recorded. Further analysis of the screen recordings clarifies the frame output of different cameras during multi-camera switching. Frame output can be reflected by clarity, the number of frames before switching, etc., and can be preliminarily distinguished and judged using fusion frames. The moment when the current zoom ratio is greater than the target zoom ratio (e.g., 3.5) can be determined, based on the ratio of the field of view of the switching frame to the base field of view. Compared to the moment when the current zoom ratio is greater than the original zoom ratio (e.g., 3X), the switching moment in this embodiment is delayed. If the above conforms to the description of this embodiment, it can be determined that differentiated zoom control and processing have been performed for the 0.6X point-to-6X scenario. Further analysis of image details can infer whether the filtering strategy inside the algorithm is consistent with this solution, such as using the initial feature offset as the current frame offset feature.

[0177] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0178] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0179] In one exemplary embodiment, such as Figure 9 As shown, an image processing apparatus is provided, comprising:

[0180] Comparison module 902 is used to compare the current zoom ratio with the target optical zoom point zoom ratio to obtain a comparison result; the target optical zoom point zoom ratio is greater than the original optical zoom point zoom ratio.

[0181] Before switching, module 904 is used to fuse the first image data acquired by the first camera and the second image data acquired by the second camera when the comparison result indicates that the first camera is the main camera, to obtain fused image data; and to obtain the target image based on the current zoom ratio and the fused image data.

[0182] The switched module 906 is used to obtain a target image based on the current zoom ratio and the second image data when the comparison result indicates that the second camera is the main camera.

[0183] In one embodiment, the pre-switching module 904 is configured to:

[0184] Determine the first image data captured by the first camera and the second image data captured by the second camera;

[0185] The feature offset of the current frame is determined based on the feature offset between the first image data and the second image data;

[0186] Based on the current frame feature offset, the first image data and the second image data are fused to obtain fused image data.

[0187] In one embodiment, the pre-switching module 904 is configured to:

[0188] Based on the feature offset between the first image data and the second image data, the initial feature offset of the current frame is determined;

[0189] Based on the offset conditions satisfied by the reference frame and the current frame, the initial feature offset is transformed to obtain the feature offset of the current frame; the reference frame is at least one frame of image data processed before the current frame.

[0190] In one embodiment, the pre-switching module 904 is configured to:

[0191] When the reference frame is offset based on the pre-aligned offset and the current frame is not offset based on the pre-aligned offset, the feature offset of the current frame is obtained based on the initial feature offset.

[0192] In one embodiment, the pre-switching module 904 is configured to:

[0193] When neither the reference frame nor the current frame is offset based on the pre-alignment offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame, the feature offset of the current frame is obtained based on the initial feature offset.

[0194] In one embodiment, the pre-switching module 904 is configured to:

[0195] The predicted value of the current frame is determined based on the offset corresponding to the offset conditions satisfied by the reference frame and the current frame.

[0196] Based on the Kalman filter gain corresponding to the current frame, the predicted value of the current frame and the initial feature offset are fused to obtain the current frame feature offset between the first image data and the second image data.

[0197] In one embodiment, the pre-switching module 904 is configured to:

[0198] If the reference frame and the current frame meet the feature offset assignment conditions, the predicted value of the current frame is determined based on the initial feature offset.

[0199] In one embodiment, the feature offset assignment condition includes offsetting one of the reference frame and the current frame based on a pre-aligned offset.

[0200] In one embodiment, the feature offset assignment condition includes that neither the reference frame nor the current frame is offset based on the pre-aligned offset, and there is a reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame; the feature offset of the second reference frame is determined based on the feature offset between the first image data of the second reference frame and the second image data of the second reference frame.

[0201] In one embodiment, the pre-switching module 904 is configured to:

[0202] If the reference frame and the current frame meet the preset value assignment conditions, the predicted value of the current frame is determined according to the preset offset.

[0203] In one embodiment, the preset value assignment condition includes that neither the reference frame nor the current frame is offset based on the pre-alignment offset, and there is no reference optical zoom point zoom ratio between the zoom ratio of the reference frame and the zoom ratio of the current frame.

[0204] In one embodiment, before comparing the current zoom ratio with the target optical zoom point zoom ratio to obtain a comparison result, the comparison module 902 is configured to:

[0205] In response to zoom operation, obtain the initial zoom ratio and the final zoom ratio;

[0206] When the initial zoom ratio is at the focal length of the first camera and the final zoom ratio is at the focal length of the second camera, zoom ratios are sequentially obtained from a preset number of the initial zoom ratios and the intermediate zoom ratios between the initial zoom ratio and the final zoom ratio as the current zoom ratio.

[0207] In one embodiment, the comparison module 902 is configured to:

[0208] When it is determined that the initial zoom ratio is within the focal length of the first camera and the final zoom ratio is within the focal length of the second camera, an activation signal for the second camera is generated.

[0209] Based on the activation signal, the second camera is controlled to be in the activated state.

[0210] In one embodiment, before fusing the first image data captured by the first camera and the second image data captured by the second camera to obtain the fused image data, the pre-switching module 904 is configured to:

[0211] Obtain the camera's on / off status indicator;

[0212] Based on the camera on / off status indicated by the switch status identifier, it is determined that the first camera and the second camera are in the on / off state.

[0213] In one embodiment, before fusing the first image data acquired by the first camera and the second image data acquired by the second camera, the pre-switching module 904 is configured to:

[0214] When the second camera is not turned on, the pre-alignment offset is determined based on the calibration parameters and focus distance between the first camera and the second camera.

[0215] Based on the pre-alignment offset, the position of the first image data acquired by the first camera is offset to obtain the offset first image data;

[0216] The target image is obtained based on the current zoom level and the first image data after the offset.

[0217] In one embodiment, the pre-switching module 904 is configured to:

[0218] The predicted value of the current frame is determined according to a preset offset;

[0219] Based on the Kalman filter gain corresponding to the current frame, the predicted value of the current frame and the pre-alignment offset are fused to obtain the adjusted pre-alignment offset;

[0220] Based on the adjusted pre-alignment offset, the position of the first image data acquired by the first camera is offset to obtain the offset first image data.

[0221] In one embodiment, before obtaining the target image based on the current zoom ratio and the second image data, the switching module 906 is configured to:

[0222] The predicted value of the current frame is determined based on the initial feature offset.

[0223] Based on the Kalman filter gain corresponding to the current frame, the predicted value of the current frame and the initial feature offset are fused to obtain the feature offset of the current frame;

[0224] Based on the current frame feature offset, the second image data is offset to obtain the offset second image data;

[0225] The target image is obtained based on the current zoom level and the offset second image data.

[0226] In one embodiment, the comparison module 902 is configured to:

[0227] When the current zoom ratio is less than the zoom ratio of the target optical zoom point, the main camera identifier is determined to be the first main camera identifier value, and the first main camera identifier value is used to indicate that the first camera is the main camera;

[0228] If the current zoom ratio is greater than or equal to the target optical zoom point zoom ratio, the main camera identifier is determined to be the second main camera identifier value, which is used to indicate that the second camera is the main camera.

[0229] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0230] In one exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0231] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0232] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0233] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0234] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0235] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0236] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0237] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0238] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Comparing the current zoom ratio with the target light changing point zoom ratio to obtain a comparison result; the target light changing point zoom ratio is greater than the original light changing point zoom ratio; When the comparison result indicates that the first camera is the main camera, first image data captured by the first camera and second image data captured by the second camera are fused to obtain fused image data; and a target image is obtained according to the current zoom ratio and the fused image data; When the comparison result indicates that the second camera is the main camera, a target image is obtained according to the current zoom factor and the second image data.

2. The method according to claim 1, characterized in that The fusing the first image data collected by the first camera and the second image data collected by the second camera to obtain fused image data includes: Determine first image data captured by the first camera and second image data captured by the second camera; Determining a current frame feature offset based on a feature offset between the first image data and the second image data; The first image data and the second image data are fused according to the current frame feature offset to obtain fused image data.

3. The method according to claim 2, characterized in that The determining the current frame feature offset based on the feature offset between the first image data and the second image data comprises: Determining an initial feature offset of a current frame based on a feature offset between the first image data and the second image data; Based on the offset conditions satisfied by the reference frame and the current frame, the initial feature offset is converted to obtain the current frame feature offset; the reference frame is at least one frame of image data processed before the current frame.

4. The method according to claim 3, characterized in that The converting the initial feature offset based on the offset condition satisfied by the reference frame and the current frame to obtain the feature offset of the current frame includes: When the reference frame is offset based on the pre-alignment offset and the current frame is not offset based on the pre-alignment offset, the current frame feature offset is obtained based on the initial feature offset.

5. The method according to claim 3, characterized in that: The converting the initial feature offset based on the offset condition satisfied by the reference frame and the current frame to obtain the feature offset of the current frame includes: When neither the reference frame nor the current frame is offset based on the pre-alignment offset, and a reference light changing point zoom factor exists between the zoom factors of the reference frame and the current frame, a current frame feature offset is obtained based on the initial feature offset.

6. The method according to claim 3, characterized in that The converting the initial feature offset based on the offset condition satisfied by the reference frame and the current frame to obtain the feature offset of the current frame includes: Determine a prediction value of the current frame based on an offset corresponding to an offset condition satisfied by a reference frame and the current frame; According to the Kalman filter gain corresponding to the current frame, the predicted value of the current frame and the initial feature offset are fused to obtain the current frame feature offset between the first image data and the second image data.

7. The method according to claim 6, characterized in that The step of determining the prediction value of the current frame based on an offset corresponding to an offset condition satisfied by a reference frame and the current frame comprises: In the case where the reference frame and the current frame meet the feature offset assignment conditions, the prediction value of the current frame is determined according to the initial feature offset.

8. The method according to claim 7, characterized in that The characteristic offset assignment condition includes offsetting one of the reference frame and the current frame based on a pre-alignment offset.

9. The method according to claim 7, characterized in that: The characteristic offset assignment conditions include that neither the reference frame nor the current frame is offset based on the pre-alignment offset, and there is a reference light change point zoom magnification between the zoom magnification of the reference frame and the zoom magnification of the current frame; the characteristic offset of the second reference frame is determined based on the characteristic offset between the first image data of the second reference frame and the second image data of the second reference frame.

10. The method according to claim 6, characterized in that The step of determining the prediction value of the current frame based on an offset corresponding to an offset condition satisfied by a reference frame and the current frame comprises: In the case where the reference frame and the current frame meet the preset value assignment conditions, the prediction value of the current frame is determined according to the preset offset.

11. The method according to claim 10, characterized in that The preset value assignment condition includes that neither the reference frame nor the current frame is offset based on the pre-alignment offset, and there is no reference light changing point zoom factor between the zoom factor of the reference frame and the zoom factor of the current frame.

12. The method according to claim 1, characterized in that Before comparing the current zoom factor with the target light changing point zoom factor to obtain a comparison result, the method further includes: In response to a zoom operation, acquiring a starting zoom magnification and an ending zoom magnification; When the starting zoom ratio is within the focal length of the first camera and the ending zoom ratio is within the focal length of the second camera, zoom ratios are sequentially acquired as the current zoom ratio from a zoom ratio sequence consisting of a preset number of starting zoom ratios and intermediate zoom ratios between the starting zoom ratio and the ending zoom ratio.

13. The method according to claim 12, characterized in that The method further comprises: When it is determined that the starting zoom ratio is within the focal length of the first camera and the ending zoom ratio is within the focal length of the second camera, generating a start signal for the second camera; According to the start-up signal, the second camera is controlled to be in an on state.

14. The method according to claim 1, characterized in that Before fusing the first image data collected by the first camera and the second image data collected by the second camera to obtain the fused image data, the method further includes: Get the switch status flag of the camera; According to the camera on state indicated by the switch state identifier, it is determined that the first camera and the second camera are in the on state.

15. The method according to claim 1, characterized in that Before fusing the first image data collected by the first camera and the second image data collected by the second camera, the method further includes: When the second camera is not turned on, determining a pre-alignment offset based on calibration parameters and a focus distance between the first camera and the second camera; Performing a position shift on the first image data collected by the first camera according to the pre-alignment offset to obtain the shifted first image data; A target image is obtained according to the current zoom factor and the shifted first image data.

16. The method according to claim 15, characterized in that The step of performing position shifting on the first image data collected by the first camera according to the pre-alignment offset to obtain the shifted first image data includes: Determine the predicted value of the current frame according to the preset offset; According to the Kalman filter gain corresponding to the current frame, the predicted value of the current frame and the pre-alignment offset are merged to obtain an adjusted pre-alignment offset; According to the adjusted pre-alignment offset, the first image data collected by the first camera is positionally offset to obtain the offset first image data.

17. The method according to claim 1, characterized in that Before obtaining the target image according to the current zoom factor and the second image data, the method further includes: Determining a prediction value of the current frame according to the initial feature offset; According to the Kalman filter gain corresponding to the current frame, the predicted value of the current frame and the initial feature offset are fused to obtain the feature offset of the current frame; Performing position shifting on the second image data according to the current frame feature offset to obtain shifted second image data; The step of obtaining a target image according to the current zoom factor and the second image data includes: A target image is obtained according to the current zoom factor and the shifted second image data.

18. The method according to claim 1, characterized in that The comparing the current zoom factor with the zoom factor of the target light changing point to obtain a comparison result includes: When the current zoom ratio is less than the target light changing point zoom ratio, determining that the main camera identifier is a first main camera identifier value, where the first main camera identifier value is used to indicate that the first camera is the main camera; When the current zoom ratio is greater than or equal to the target light changing point zoom ratio, the main camera identifier is determined to be a second main camera identifier value, where the second main camera identifier value is used to indicate that the second camera is the main camera.

19. An image processing device, characterized in that: The device comprises: A comparison module, used for comparing the current zoom ratio with the target light change point zoom ratio to obtain a comparison result; the target light change point zoom ratio is greater than the original light change point zoom ratio; a pre-switching module, configured to fuse first image data captured by the first camera and second image data captured by the second camera to obtain fused image data when the comparison result indicates that the first camera is the main camera; and obtain a target image according to the current zoom ratio and the fused image data; The post-switching module is used to obtain a target image according to the current zoom ratio and the second image data when the comparison result indicates that the second camera is the main camera.

20. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 18 are implemented.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

Images