Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method, and the method comprises the steps: determining an accumulated value of a beat counter in response to a switching instruction from a first frame rate mode to a second frame rate mode; wherein the first frame rate mode drives parallax calculation based on a first frame sampling period; the second frame rate mode drives parallax calculation based on a second frame sampling period; the accumulated numerical value of the beat counter is the accumulated frame number after the last parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period; switching from the first frame rate mode to a second frame rate mode based on an accumulated value of the beat counter and a second frame sampling period; performing parallax calculation based on the second frame sampling period, the first preview image and the second preview image to obtain a first depth image; wherein the first preview image is acquired through the first camera, and the second preview image is acquired through the second camera.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an image processing method, apparatus, device, and storage medium. Background Art

[0002] With the continuous progress and development of terminal technologies, especially in portrait photography and bokeh effects, a dedicated portrait mode is usually equipped; in high-load scenarios, the terminal usually uses a frame-drop mode to achieve background bokeh through the combination of software algorithms and hardware.

[0003] However, when the terminal switches between the frame-drop mode and the normal frame rate mode, data loss or processing interruption may occur due to untimely switching of the beat signal, which cannot ensure the continuity and consistency of the data stream and affects the image processing effect. Summary of the Invention

[0004] Embodiments of the present application provide an image processing method, apparatus, device, and storage medium, which can ensure the consistency and continuity of the data stream during the switching process between different frame rate modes and improve the image processing effect.

[0005] The technical solution of the embodiments of the present application is implemented as follows:

[0006] In a first aspect, embodiments of the present application provide an image processing method. The image processing method is applied to an image processing apparatus configured with a first camera and a second camera. The method includes:

[0007] In response to a switching instruction from a first frame rate mode to a second frame rate mode, determining the cumulative value of a beat counter; wherein, the first frame rate mode drives parallax calculation based on a first frame sampling period; the second frame rate mode drives parallax calculation based on a second frame sampling period; the cumulative value of the beat counter is the cumulative number of frames after the previous parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period;

[0008] Switching from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period;

[0009] Performing parallax calculation based on the second frame sampling period, a first preview image, and a second preview image to obtain a first depth image; wherein, the first preview image is captured by the first camera, and the second preview image is captured by the second camera. In a second aspect, embodiments of the present application provide an image processing apparatus. The image processing apparatus includes:

[0010] A determination unit, configured to determine the accumulated value of a beat counter in response to a switching instruction from a first frame rate mode to a second frame rate mode; wherein, the first frame rate mode drives parallax calculation based on a first frame sampling period; the second frame rate mode drives parallax calculation based on a second frame sampling period; the accumulated value of the beat counter is the accumulated number of frames after the last parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period;

[0011] A switching unit, configured to switch from the first frame rate mode to the second frame rate mode based on the accumulated value of the beat counter and the second frame sampling period;

[0012] A calculation unit, configured to perform parallax calculation based on the second frame sampling period, a first preview image, and a second preview image to obtain a first depth image; wherein, the first preview image is captured by the first camera, and the second preview image is captured by the second camera.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory storing processor-executable instructions. When the instructions are executed by the processor, the method according to the first aspect is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the method according to the first aspect is implemented.

[0015] An embodiment of the present application provides an image processing method, apparatus, device, and storage medium. The image processing method is applied to an image processing device configured with a first camera and a second camera. The method includes: in response to a switching instruction from a first frame rate mode to a second frame rate mode, determining the cumulative value of a beat counter; wherein, the first frame rate mode drives parallax calculation based on a first frame sampling period; the second frame rate mode drives parallax calculation based on a second frame sampling period; the cumulative value of the beat counter is the cumulative number of frames after the previous parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period; switching from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period; performing parallax calculation based on the second frame sampling period, a first preview image, and a second preview image to obtain a first depth image; wherein, the first preview image is captured by the first camera, and the second preview image is captured by the second camera. That is to say, in the embodiment of the present application, in the first frame rate mode, the image processing device continuously records the cumulative value of the beat counter. In response to a switching instruction from the first frame rate mode to the second frame rate mode, the image processing device performs collaborative control based on the cumulative value and the second frame sampling period to ensure the synchronization of the beat signal, thereby avoiding data loss or processing interruption caused by timing misalignment, effectively preventing problems such as stuttering of the preview screen or unnatural virtualization effects; ensuring the consistency and continuity of the data stream during the switching process between different frame rate modes, thereby improving the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic flowchart of the implementation of the image processing method provided by the embodiment of the present application;

[0017] Figure 2 Schematic diagram of the first frame sampling period provided by the embodiment of the present application;

[0018] Figure 3 Schematic diagram of the second frame sampling period provided by the embodiment of the present application;

[0019] Figure 4 Schematic diagram of a frame rate mode switching provided by the embodiment of the present application;

[0020] Figure 5 Schematic diagram of a frame rate mode switching provided by the embodiment of the present application;

[0021] Figure 6 Schematic diagram of driving parallax calculation in the second frame rate mode provided by the embodiment of the present application;

[0022] Figure 7 Schematic diagram of driving parallax calculation in the first frame rate mode provided by the embodiment of the present application;

[0023] Figure 8 Schematic diagram of driving calibration alignment in the first frame rate mode provided by the embodiments of the present application;

[0024] Figure 9 Schematic diagram of driving calibration alignment in the second frame rate mode provided by the embodiments of the present application;

[0025] Figure 10 Schematic diagram of driving calibration alignment in the first frame rate mode provided by the embodiments of the present application;

[0026] Figure 11 Schematic diagram of the implementation framework for binocular portrait blurring preview provided by the embodiments of the present application;

[0027] Figure 12 Schematic diagram of the trigger mechanism process for parallax calculation logic when the frame rate mode is switched provided by the embodiments of the present application;

[0028] Figure 13 Schematic diagram of the trigger mechanism process for calibration alignment logic when the frame rate mode is switched provided by the embodiments of the present application;

[0029] Figure 14 Schematic diagram of the process for multi-logic unit collaboration and beat fine-tuning provided by the embodiments of the present application;

[0030] Figure 15 Schematic diagram of the composition structure of the image processing device provided by the embodiments of the present application;

[0031] Figure 16 Schematic diagram of the composition structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0032] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the sake of description, only the parts related to the present application are shown in the drawings.

[0033] Portrait blurring is a common technique in photography. Usually, the background or foreground is blurred to highlight the photographed subject, making the people / objects in the photo more eye-catching. In traditional cameras, portrait blurring is usually achieved by using a large aperture (low f value, such as f / 1.8, f / 2.8), which can obtain a shallow depth of field effect. A large aperture means more light enters the lens, and at the same time, it will reduce the depth of field, making the areas before and after the focus quickly become blurred. However, this blurring method relying on optical hardware is limited by physical conditions and is difficult to directly implement in small devices (such as smartphones).

[0034] In recent years, with the rapid development of computational photography technology, especially the popularization of smartphone camera systems, mobile phone camera computational photography has made great progress in recent years, especially in portrait photography and bokeh effects. Usually equipped with a dedicated portrait mode, it combines software algorithms and hardware to achieve background bokeh. This technology helps to highlight the information of the subject and reduce background interference, similar to the effect of shooting with a large-aperture DSLR camera. Different technical solutions can be adopted according to different costs and technical capabilities. Common mobile phone portrait bokeh technologies include dual-camera systems, multi-camera systems, single-camera + software algorithms, Time of Flight (ToF) measurement sensors, laser-assisted focusing, depth sensors, 3D structured light, etc.

[0035] Most rear cameras of smartphones are generally equipped with two or more cameras, and a dual-camera bokeh system can be achieved without adding additional components, reducing cost expenditures. Therefore, the dual-camera system portrait bokeh solution has been widely applied, and its working principle is as follows:

[0036] (1) Dual-camera system: When the dual-camera system works, it usually includes a main camera and an auxiliary camera. With the two images obtained by these cameras, the mobile phone can calculate the depth information (i.e., the distance from the camera) of each pixel.

[0037] (2) Depth perception and calculation: Using the two slightly different images taken by the dual-camera, the mobile phone camera calculates the depth information in the scene according to the parallax (the slight difference between the two images), thereby generating a depth map to separate the foreground from the background. The depth map is a grayscale image, where the grayscale value of each pixel represents its distance from the camera. Through the depth map, the system can accurately distinguish the foreground from the background and provide precise spatial information for subsequent bokeh processing. To obtain a more accurate depth map in complex or single scenes, processes such as image calibration, Optical Image Stabilizer (OIS) compensation, and Artificial Intelligence (AI) refinement are also added.

[0038] (3) Real-time bokeh algorithm: Based on the depth information provided by the depth map, the mobile phone camera can accurately identify the foreground objects (such as faces or bodies) in the photo and clearly retain them. At the same time, the background area is blurred according to the depth information. An efficient image segmentation algorithm can be used to ensure that the boundary between the subject and the background is processed naturally and smoothly. This processing process is usually presented on the screen in real time, allowing users to preview the accurate bokeh effect before taking a photo.

[0039] The two main functions of the camera are preview and taking pictures. The preview function enables users to view the upcoming scene in real time before taking a picture. The taking picture function is the operation of finally capturing the preview image as a static image, and the final image is determined by pressing the shutter button. Compared with many other algorithms in the previous output-output mode, the real-time bokeh (RTB) algorithm based on the dual-camera system is an extremely complex solution. The traditional RTB general solution couples complex business logics with algorithms, with intricate logical judgments and data dependencies intertwined, and there is a strong coupling relationship among multiple data modules.

[0040] To cope with high-load scenarios or save power, the mobile phone camera will enable the frame rate reduction mode. This mode reduces the number of frames captured per second actively, reducing the processor burden and power consumption. For example, during long-term shooting, background operation, or when multiple cameras work simultaneously, the system will reduce the frame rate to reduce the amount of data processing, thereby extending the battery life and alleviating the processor pressure. However, the frame rate reduction mode also has side effects: the decrease in frame rate will lead to a lower data update frequency, which may cause stuttering in moving images and affect the shooting experience. Although the dual-camera bokeh technology has achieved remarkable results in improving the imaging quality, it still faces challenges in the frame rate reduction mode. Due to limited system resources, the data processing ability drops significantly, affecting the timeliness and stability of key links such as depth calculation, subject recognition, and bokeh processing. Traditional real-time processing algorithms are difficult to ensure the continuity and consistency of the data stream in the frame rate reduction mode, which may lead to stuttering in the preview image, unnatural bokeh effects, and even affect the final imaging quality.

[0041] Specifically, the following problems mainly exist in the frame rate reduction mode:

[0042] (1) When switching between the frame rate reduction mode and the normal frame rate mode, due to the untimely switching of the beat signal, data loss or processing interruption may occur, resulting in stuttering in the preview image and unnatural bokeh effects.

[0043] (2) When multiple logical units (such as calibration alignment logic, stereo calculation logic, AI enhancement logic) work together, the frame rate reduction mode may cause the logical beats to be out of sync, thereby affecting the overall processing efficiency.

[0044] (3) Due to the reduction of the frame rate, the system cannot complete complex depth calculations and bokeh processing within a limited time, resulting in data update delays.

[0045] That is to say, common portrait bokeh solutions may cause data loss or processing interruption due to untimely switching of the beat signal in the frame rate reduction mode and when switching between the frame rate reduction mode and the normal frame rate mode, and cannot ensure the continuity and consistency of the data stream.

[0046] To solve the above problems, the embodiments of the present application provide an image processing method, apparatus, device, and storage medium. In the first frame rate mode, the image processing apparatus continuously records the cumulative value of the beat counter. In response to a switching instruction from the first frame rate mode to the second frame rate mode, the image processing apparatus performs collaborative control based on the cumulative value and the second frame sampling period to ensure the synchronization of the beat signal, thereby avoiding data loss or processing interruption caused by timing misalignment, effectively preventing problems such as stuttering of the preview screen or unnatural blurring effects; and ensuring the consistency and continuity of the data stream during the switching process between different frame rate modes.

[0047] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application.

[0048] An embodiment of the present application provides an image processing method, which can be applied to an image processing apparatus or an electronic device, and can also be applied to any terminal including an image processing apparatus or an electronic device.

[0049] It can be understood that the image processing method proposed in the embodiments of the present application may include a portrait blurring solution in a multi-camera scenario.

[0050] Furthermore, in the embodiments of the present application, the image processing method proposed in the embodiments of the present application can be applied to a scenario where multi-camera shooting is performed and portrait blurring is carried out. That is to say, the image processing method of the present application can be used for portrait blurring processing of a multi-camera system.

[0051] Of course, the application scenarios of the image processing method proposed in the embodiments of the present application are not limited to multi-camera scenarios, and can also be applied to other scenarios such as multiple independent terminal devices respectively performing image processing. The present application does not make specific limitations.

[0052] Next, taking the image processing apparatus as an example, the image processing method proposed in the embodiments of the present application will be described by way of example. Among them, the image processing apparatus is configured with a first camera and a second camera. The first camera in the embodiments of the present application can be the main camera, and the second camera can be the secondary camera. The present application does not make any limitations in this regard.

[0053] Furthermore, in the embodiments of the present application, Figure 1 is a schematic flowchart of the implementation process of the image processing method provided in the embodiments of the present application. As Figure 1 shown, the image processing method may include the following steps:

[0054] Step 101, in response to a switching instruction from the first frame rate mode to the second frame rate mode, determine the cumulative value of the beat counter; wherein, the first frame rate mode drives parallax calculation based on the first frame sampling period; the second frame rate mode drives parallax calculation based on the second frame sampling period; the first frame sampling period is greater than or equal to the second frame sampling period.

[0055] In an embodiment of the present application, after the image processing device receives a switching instruction from the first frame rate mode to the second frame rate mode, it can respond to the switching instruction to determine the cumulative value of the beat counter.

[0056] In an embodiment of the present application, the first frame rate mode refers to a mode in which the image processing device performs parallax calculation at a lower frame rate. That is to say, the image processing device drives parallax calculation based on the first frame sampling period (i.e., the update frequency is lower).

[0057] It should be noted that the first frame rate mode is applicable to scenarios with low requirements for real-time performance, where it is necessary to save computing resources or the load is relatively high.

[0058] In some embodiments, the first frame rate mode can be that the frame rate mode of the second camera in the image processing device is a frame reduction mode. That is to say, the image data collected by the second camera may be periodically missing due to the frame reduction strategy. For example, only 1 frame of image data is valid for every 3 frames of image data.

[0059] Exemplarily, the first frame rate mode can be used in different scenarios. For example, in scenarios such as regular camera preview, event triggers without mode switching / zooming, etc., the image processing device can reduce the hardware load through the frame reduction strategy.

[0060] For example, when the mobile terminal is in the standby state, the preset frame reduction ratio can be set to 3:1, that is, parallax calculation is triggered every 3 frames to save resources. Another example is that when the load of the mobile terminal is high, the preset frame reduction ratio can be set to 6:1, and parallax calculation is triggered every 6 frames. The present application does not specifically limit the frame reduction ratio.

[0061] In an embodiment of the present application, the first frame sampling period is the frame interval period in which the second camera in the first frame rate mode periodically skips some frame data in order to reduce power consumption or data volume.

[0062] That is to say, the first frame sampling period refers to the number of frames skipped periodically by the image processing device when the second camera operates in the first frame rate mode.

[0063] For example, if the sampling period of the first frame is A1, it means that the second camera outputs a frame of valid data every A1 frames, and the remaining frames are discarded or suppressed. At this time, the trigger frequency of the parallax calculation is synchronized with the sampling period of the first frame, that is, the parallax calculation is performed every A1 frames. It should be noted that the sampling period of the first frame is an integer greater than or equal to 1.

[0064] It should be understood that the frame reduction ratio in the embodiments of the present application corresponds to the sampling period of the first frame. Exemplarily, if the frame reduction ratio is 3:1, the sampling period of the first frame is 3; if the frame reduction ratio is 5:1, the sampling period of the first frame is 5.

[0065] In some embodiments, the frame reduction ratio can be dynamically adjusted according to the real-time scene. Next, this embodiment introduces the process of dynamically adjusting the frame reduction ratio.

[0066] (1) Motion detection module. This module evaluates the scene dynamics in real time through optical flow analysis technology. The optical flow algorithm (such as the Farneback dense optical flow) calculates the pixel motion vectors between adjacent frames and statistically analyzes the global motion intensity and distribution. The image processing device can set a threshold to divide the scene into low dynamics (such as static scenery), medium dynamics (such as slowly moving objects), and high dynamics (such as fast motion). For example, if the displacement of 80% of the pixels < 2 pixels / frame, it is low dynamics; if the displacement is between 2 - 5 pixels / frame, it is medium dynamics; if it exceeds 5 pixels / frame, it is high dynamics. This classification result will be used as the core basis for adjusting the frame reduction ratio to ensure that more key frames are retained in dynamic scenes, while static scenes are significantly reduced in frames to save resources.

[0067] (2) Resource decision engine. The core logic unit for dynamically adjusting the frame reduction ratio based on the motion level. It adopts a three-level mapping mechanism: the low-dynamics scene triggers the highest frame reduction (such as 12:1) because the static content has low requirements for frame-to-frame continuity; the medium-dynamics scene adopts medium frame reduction (such as 5:1) to balance efficiency and quality; the high-dynamics scene maintains low frame reduction (such as 2:1) to prevent motion blur. The decision engine can integrate an adaptive threshold adjustment algorithm to fine-tune the frame reduction ratio according to real-time parameters such as device temperature and memory occupancy, forming a closed-loop optimization system. By dynamically matching the scene requirements, it avoids the problem of "resource waste in static scenes" caused by a fixed frame reduction ratio.

[0068] (3) Temporal interpolation compensation. An image quality enhancement module designed for extreme frame reduction scenarios (such as above 5:1). When the frame sampling rate is too low, resulting in a risk of stuttering, first extract the depth information of the historical frames (obtained through a binocular camera or monocular depth estimation) to construct a scene geometric model. Subsequently, use an AI prediction network (such as an LSTM time series model) to estimate the object motion trajectory and texture changes of the intermediate frames and generate high-confidence interpolation frames. Exemplarily, if the Nth frame and the (N + 5)th frame are retained, the system can generate frames from the (N + 1)th to the (N + 4)th frame through motion estimation to maintain visual coherence.

[0069] Exemplarily, in some embodiments, Figure 2 This is a schematic diagram of the first frame sampling period provided by the embodiments of the present application; as can be seen from the figure, the image data collected by the first camera is output at full frame rate, that is, there is image data for each frame. The first frame sampling period is 6; that is to say, the image data collected by the second camera is periodically missing, and only 1 frame of image data is valid for every 6 frames of image data. Therefore, the image processing device can perform a parallax calculation once at the Nth frame based on the first frame sampling period, and then perform a parallax calculation at the (N + 6)th frame and at the (N + 12)th frame.

[0070] In the embodiments of the present application, the second frame rate mode refers to a mode in which the image processing device runs the parallax calculation at a higher frame rate, that is, the image processing device drives the parallax calculation based on the second frame sampling period (i.e., the update frequency is higher). It should be noted that the second frame rate mode is applicable to scenarios that require fast response or high precision.

[0071] In some embodiments, the second frame rate mode can be that the frame rate mode of the second camera in the image processing device is the normal frame rate mode. That is to say, the image data collected by the second camera is complete and not missing, that is, each frame of image data is a valid data frame.

[0072] Exemplarily, the second frame rate mode can also be used in different scenarios. For example, during complex scenario switches such as camera mode switching, lens switching, resolution switching, fallback mechanism trigger (Fallback), and optical / digital zoom adjustment (ZOOM zoom), it is necessary to ensure that the defocus effect converges quickly and maintain the preview frame rate stability of the second camera. Therefore, the second frame rate mode can be adopted.

[0073] In the embodiments of the present application, the second frame sampling period (which can also be referred to as the parallax calculation beat period) can refer to the frame interval period when the second camera in the image processing device is in the second frame rate mode and triggers the parallax calculation. It should be noted that the second frame sampling period is an integer greater than or equal to 1. That is to say, the second frame sampling period refers to the number of frames skipped periodically by the image processing device when the second camera runs in the second frame rate mode.

[0074] Exemplarily, if the second frame sampling period is A2 (A2 is less than or equal to A1), it means that the image processing device performs a parallax calculation every A2 frames of data. For example, when A2 = 3, the image processing device triggers a parallax calculation for every 3 frames of images, and the remaining frames are only used for data caching or skipped calculations. Another example is when A2 = 5, the image processing device triggers a parallax calculation for every 5 frames of images.

[0075] In some embodiments, in scenarios with high real-time performance or high precision (such as fast movement, high-precision navigation), a smaller value of A2 (such as A2 = 1 or 2) can increase the frequency of the parallax calculation output. At this time, the second camera usually outputs data at the full frame rate without periodic missing.

[0076] It should be noted that the configuration of the second frame sampling period can be dynamically adjusted according to the actual requirements of the system. Specifically, the selection of the value of A2 needs to comprehensively consider factors such as hardware computing power, power consumption constraints, and scene complexity: in low-load scenarios, the value of A2 can be appropriately reduced to increase the parallax calculation frequency, thereby improving the defocusing accuracy; while in high-load scenarios or energy-efficiency sensitive scenarios, the value of A2 can be correspondingly increased to reduce the calculation frequency and achieve an optimal balance between computing power and power consumption. The present application does not make specific limitations.

[0077] Exemplarily, in some embodiments, Figure 2 is a schematic diagram of the second frame sampling period provided by the embodiments of the present application; as can be seen from the figure, the image data collected by the first camera and the second camera are both output at the full frame rate, that is, there is image data for each frame. The second frame sampling period is 3; that is to say, the image processing device triggers a parallax calculation every 3 frames of images, and the remaining frames are only used for data caching or skipped calculations. Therefore, the image processing device can perform a parallax calculation on the Nth frame based on the second frame sampling period, and then perform a parallax calculation on the (N + 3)th frame, a parallax calculation on the (N + 6)th frame; a parallax calculation on the (N + 9)th frame, a parallax calculation on the (N + 12)th frame, and a parallax calculation on the (N + 15)th frame.

[0078] In the embodiments of the present application, the cumulative value of the beat counter is used to record the cumulative number of frames (or time beats) since the last parallax calculation, and its function can be dynamically adjusted according to the working mode. For example, in the second frame rate mode, the system continuously monitors this cumulative value and triggers a parallax calculation when the second frame sampling period is reached, and then clears the counter and starts counting again; while in the first frame rate mode, the counter keeps accumulating but no longer serves as a calculation trigger condition, only for numerical recording. Another example is that when switching back from the first frame rate mode to the second frame rate mode, the system immediately resumes using the current cumulative value for condition judgment. If the threshold is met, a parallax calculation is performed, and then the system continues to run according to the logic of the second frame rate mode, so as to achieve seamless connection between different modes and coherent maintenance of the counting state.

[0079] In some embodiments, the image processing device can determine the cumulative value of the beat counter through a hardware counter. Exemplarily, the image processing device can be automatically incremented every time a frame is rendered through the frame synchronization signal built into the graphics processor.

[0080] In some embodiments, the image processing device can also determine the cumulative value of the beat counter through a software timer, which is used to determine the cumulative number of frames since the last parallax calculation.

[0081] It should be noted that the beat counter can also be referred to as the mCurrentGap counter.

[0082] In the embodiments of the present application, the switching instruction is used to trigger the dynamic adjustment of the frame rate mode. It should be noted that the switching instruction can be triggered by different methods, and the present application does not limit this.

[0083] In some embodiments, the switching instruction can be triggered based on user interaction. Exemplarily, the terminal device where the user is located can directly trigger it through a voice command (such as "enable high frame rate mode").

[0084] In some embodiments, the switching instruction can also be triggered by a status identifier.

[0085] Exemplarily, the image processing device can obtain the skip frame logic state and set the skip frame logic identifier (which can also be expressed as isSkipLogicEnable) according to the skip frame logic state. If the image processing device determines that the skip frame logic state is the active state, it can set the skip frame logic identifier to isSkipLogicEnable = true, which means the skip frame logic state is the active state. That is to say, at this time, the image processing device can switch from the second frame rate mode to the first frame rate mode. If the image processing device determines that the skip frame logic state is the closed state, it can set the skip frame logic identifier to isSkipLogicEnable = false, which means the skip frame logic state is the closed state. That is to say, at this time, the image processing device can switch from the first frame rate mode to the second frame rate mode.

[0086] Exemplarily, when the image processing device responds to a change in the system load (such as switching from low load to high load), the image processing device can control the second camera to switch from the second frame rate mode to the first frame rate mode; in another example, when the image processing device responds to scene dynamics (such as changing from a static scene to a fast-moving scene), the image processing device can control the second camera to switch from the first frame rate mode to the second frame rate mode.

[0087] Step 102, switch from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period.

[0088] In an embodiment of the present application, after the image processing device determines the cumulative value of the beat counter in response to a switching instruction from the first frame rate mode to the second frame rate mode, the image processing device may switch from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period.

[0089] In some embodiments, considering that there may be no image data in the preview frame of the secondary camera in the first frame rate mode, therefore, during the process of switching from the first frame rate mode to the second frame rate mode, the image processing device may first determine whether there is image data in the preview frame of the secondary camera that needs to perform parallax calculation. If there is data, it can directly switch from the first frame rate mode to the second frame rate mode. If there is no data, it needs to wait.

[0090] In an embodiment of the present application, during the process of the image processing device switching from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period, when the cumulative value of the beat counter reaches an integer multiple of the second frame sampling period, the image processing device may control the second camera to switch from the first frame rate mode to the second frame rate mode.

[0091] In some embodiments, after the image processing device determines that the frame skipping logic is turned off, that is, the frame skipping logic status identifier is isSkipLogicEnable = false, it may determine the cumulative value of the beat counter (i.e., the cumulative number of frames of the counter mCurrentGap). After that, if the image processing device determines that the cumulative value of the beat counter reaches the second frame sampling period, it can switch from the first frame rate mode to the second frame rate mode and perform parallax calculation according to the second frame sampling period.

[0092] In some embodiments, by way of example, Figure 4 is a schematic diagram of frame rate mode switching provided for an embodiment of the present application, as Figure 4 shown. Assume that the first frame sampling period A1 = 3 in the first frame rate mode and the second frame sampling period A2 = 3 in the second frame rate mode. The first camera outputs image data at full frame rate, that is, there is image data in each frame; the second camera starts using the first frame rate mode from the Nth frame, that is, the second camera outputs one frame of image data every 3 frames according to the first frame sampling period.

[0093] As can be seen from the figure, at the Nth frame, the image processing device can set the frame skipping logic identifier to isSkipLogicEnable = true. After determining that both the main and secondary data of the current frame are valid at the Nth frame, the image processing device performs disparity calculation; resets the beat counter (i.e., sets the cumulative value of the mCurrentGap counter to 0), and sets the isDoStereoDepth signal (i.e., sets the disparity calculation status identifier to isDoStereoDepth = true).

[0094] And so on, the image processing device performs disparity calculation at the (N + 3)th frame; performs disparity calculation at the (N + 6)th frame; performs disparity calculation at the (N + 9)th frame.

[0095] At the (N + 9)th frame, the image processing device detects that the frame skipping logic is turned off and sets the frame skipping logic identifier to isSkipLogicEnable = false. In this case, as can be seen from the figure, the image data corresponding to the second camera arrives normally at the (N + 9)th frame. As Figure 4 shown, the cumulative value of the beat counter is 3, that is, the cumulative value of the beat counter is an integer multiple of the second frame sampling period. Then the image processing device can switch from the first frame rate mode to the second frame rate mode. After that, the second camera outputs image data at full frame rate, and the image processing device performs disparity calculation according to the second frame sampling period, that is, the image processing device performs disparity calculation at the (N + 12)th frame; performs disparity calculation at the (N + 15)th frame.

[0096] In the embodiments of the present application, during the process of the image processing device switching from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period, when the cumulative value of the beat counter has not reached an integer multiple of the second frame sampling period, the image processing device can update the cumulative value of the beat counter; until the updated cumulative value of the beat counter reaches an integer multiple of the second frame sampling period, switch from the first frame rate mode to the second frame rate mode, and perform disparity calculation.

[0097] In some embodiments, after the image processing device determines that the frame skipping logic is turned off, that is, the status of the frame skipping logic identifier is isSkipLogicEnable = false, it can determine the cumulative value of the beat counter. Then, if the image processing device determines that the cumulative value of the beat counter has not reached the second frame sampling period or an integer multiple of the second frame sampling period, it can first switch from the first frame rate mode to the second frame rate mode, but at this time, no disparity calculation is performed, and the cumulative value of the beat counter is obtained and updated in real time until the updated cumulative value of the beat counter reaches the second frame sampling period or an integer multiple of the second frame sampling period, and then disparity calculation is performed according to the second frame sampling period M.

[0098] In some embodiments, by way of example, Figure 5 FIG. is a schematic diagram of frame rate mode switching provided by an embodiment of the present application. As Figure 5 shown, the first camera outputs image data at full frame rate, that is, there is image data for each frame; the second camera starts from the Nth frame and adopts the first frame rate mode, that is, the second camera outputs one frame of image data every 3 frames according to the first frame sampling period.

[0099] It can be seen from the figure that at the Nth frame, the image processing device can set the frame skipping logic identifier to isSkipLogicEnable = true. After the image processing device determines that both the main and secondary data of the current frame are valid at the Nth frame, it performs disparity calculation; and resets the beat counter (that is, sets the accumulated value of the mCurrentGap counter to 0), and sets the isDoStereoDepth signal (that is, sets the disparity calculation status identifier to isDoStereoDepth = true).

[0100] And so on, the image processing device performs disparity calculation at the N + 3th frame; performs disparity calculation at the N + 6th frame.

[0101] At the N + 8th frame, the image processing device detects that the frame skipping logic is turned off, and sets the frame skipping logic identifier to isSkipLogicEnable = false. After that, the second camera outputs image data at full frame rate. In this case, it can be seen from the figure that the image data corresponding to the second camera at the N + 8th frame does not reach normally due to the frame skipping logic. As Figure 5 shown, the accumulated value of the beat counter is 2, that is, the accumulated value of the beat counter does not reach an integer multiple of the second frame sampling period. Then the image processing device can switch from the first frame rate mode to the second frame rate mode, but does not perform disparity calculation at this time, and obtains and updates the accumulated value of the beat counter in real time until the updated accumulated value of the beat counter reaches an integer multiple of the second frame sampling period, and then performs disparity calculation according to the second frame sampling period, that is, performs disparity calculation at the N + 9th frame and the N + 12th frame.

[0102] In some embodiments, after the image processing device determines that the frame skipping logic identifier is isSkipLogicEnable = false, it can determine the accumulated value of the beat counter. Then, if the image processing device determines that the accumulated value of the beat counter does not reach the second frame sampling period or an integer multiple of the second frame sampling period, it can obtain and update the accumulated value of the beat counter in real time until the updated accumulated value of the beat counter reaches the second frame sampling period or an integer multiple of the second frame sampling period, and can switch from the first frame rate mode to the second frame rate mode, and perform disparity calculation according to the second frame sampling period.

[0103] Step 103: Based on the second frame sampling period, the first preview image, and the second preview image, perform parallax calculation to obtain a first depth image. The first preview image is captured by the first camera, and the second preview image is captured by the second camera.

[0104] In an embodiment of the present application, after the image processing device switches from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period, the image processing device can perform parallax calculation based on the second frame sampling period, the first preview image, and the second preview image to obtain a first depth image.

[0105] In an embodiment of the present application, the first preview image may include a sequence of multiple frames of image data continuously captured by the first camera. The second preview image may include a sequence of multiple frames of image data continuously captured by the second camera.

[0106] That is to say, the first preview image is not a single-frame image, but a sequence of multiple frames of images corresponding to a time series continuously captured by the first camera, where each frame of image corresponds to a specific timestamp. Similarly, the second preview image is a sequence of multiple frames of images corresponding to a time series continuously captured by the second camera, where each frame of image corresponds to a specific timestamp. The time series of the first preview image and the second preview image are the same and in one-to-one correspondence.

[0107] In an embodiment of the present application, the image processing device captures the current scene through the first camera and the second camera to obtain the first preview image and the second preview image.

[0108] Exemplarily, in an implementation scenario, the image processing device is configured with multiple different cameras, where the number of cameras configured by the image processing device may be greater than or equal to 2. That is to say, in the present application, the image processing device is a multi-camera device.

[0109] In some embodiments, multiple different cameras usually have different functions and characteristics in design to meet the shooting requirements in different scenarios. The performance and types of multiple different cameras are not specifically limited in the present application.

[0110] In some embodiments, the types of the first camera and the second camera can be flexibly selected according to the application scenario and hardware configuration.

[0111] Exemplarily, in some embodiments, the first camera and the second camera may include, but are not limited to, at least two of cameras such as wide-angle cameras, telephoto cameras, macro cameras, and depth cameras.

[0112] Among them, the wide-angle camera has a wider viewing angle than the main camera, can accommodate more scenes, and is suitable for shooting scenes such as landscapes and buildings. The wide-angle camera brings a stronger visual impact and can show a broader field of view. In addition, the wide-angle camera can also perform excellently when shooting in narrow spaces, making the photos more three-dimensional and stereoscopic.

[0113] The telephoto camera is usually used to achieve optical zoom, can zoom in on distant scenes for shooting, and is suitable for shooting distant views, portrait close-ups, etc. The telephoto camera has a shallower depth of field, can highlight the subject and blur the background, creating a more professional shooting effect. The telephoto cameras of some high-end mobile phones also support high-power optical zoom, and can achieve a longer shooting distance while maintaining clarity.

[0114] The macro camera supports close focusing and can capture wonderful details of the microscopic world, such as flowers and insects. The macro camera usually has a high magnification ratio and good focusing performance, and can present a delicate and clear microscopic world.

[0115] The depth camera is mainly used to enhance the blurring effect of photos, making the subject more prominent and the background more blurred. Through algorithm processing, the depth camera can achieve a more natural and soft blurring effect, enhancing the artistic sense of the photos.

[0116] Of course, in addition to the above common camera types, multiple different cameras can also include some special cameras, such as ToF lenses, movie lenses, etc. Among them, the ToF lens is mainly used to achieve three-dimensional perception and depth measurement, and can be used for functions such as augmented reality and face recognition. The movie lens usually has a high pixel count and excellent color reproduction ability, and is suitable for shooting high-quality videos.

[0117] In the embodiments of the present application, parallax calculation refers to the process of calculating the pixel displacement (parallax) in the horizontal direction between the first preview image and the corresponding feature points in the second preview image, and then converting it into depth information according to camera parameters (such as focal length, baseline distance). The first depth image refers to the image data generated through parallax calculation and containing the depth values of each pixel point in the scene, and the depth information can be represented in the form of grayscale values or matrices.

[0118] In the embodiments of the present application, in the process that the image processing device performs parallax calculation based on the second frame sampling period, the first preview image and the second preview image to obtain the first depth image, the image processing device can determine the first image in the first preview image based on the second frame sampling period; at the same time, determine the second image in the second preview image based on the second frame sampling period; further, the image processing device performs parallax calculation based on the first image and the second image to obtain the first depth image.

[0119] In an embodiment of the present application, the first image refers to a specific frame image selected from a sequence of multiple frames (first preview images) continuously captured by the first camera according to the interval rule of the second frame sampling period. Specifically, when the second frame sampling period A2 = K, the system selects one frame every K frames from the first preview image sequence as the first image (for example, selecting the 1st, K + 1st, 2K + 1st... frames), and these selected frames will be used as the input data for disparity calculation.

[0120] In an embodiment of the present application, the second image refers to a specific frame image selected from the second preview images continuously captured by the second camera according to the interval rule of the second frame sampling period.

[0121] It should be understood that the second image is the corresponding frame image that is strictly time - synchronized with the first image. The image processing device can ensure that the second image and the first image are captured at the same moment (or meet the fixed - delay requirements of the binocular system) through timestamp alignment or hardware synchronization signals, so as to ensure that the scene content of the two images is the same during disparity calculation.

[0122] In an embodiment of the present application, the image processing device can determine the state of disparity calculation and set a disparity - calculation identifier (which can be represented as isDoStereoDetph) according to the state of disparity calculation. If the received state of disparity calculation by the image processing device is the enabled state, the disparity - calculation identifier can be set to isDoStereoDetph = true, indicating that the disparity calculation is in the enabled state. That is to say, at this time, the image processing device can perform disparity calculation on the current frame. If the received state of disparity calculation by the image processing device is the disabled state, the disparity - calculation identifier can be set to isDoStereoDetph = false, indicating that the disparity calculation is in the disabled state. That is to say, at this time, the image processing device cannot perform disparity calculation on the current frame.

[0123] In some embodiments, the image signal processors (ISPs) of the first camera and the second camera can output a sequence of multiple frames of raw image data (i.e., the first preview image and the second preview image) continuously captured to the system memory. Further, the image processing device can periodically select valid data frames that meet the requirements from the memory based on a preset second frame sampling period. Specifically, the system extracts time - synchronized target frames (i.e., the first image and the second image) from the first preview image and the second preview image respectively according to the interval rule (such as every M frames); finally, the first image and the second image are input into the disparity - calculation module, and the disparity - calculation status identifier is set to isDoStereoDetph = true to perform disparity calculation.

[0124] In some embodiments, by way of example, Figure 6Schematic diagram of driving parallax calculation in the second frame rate mode provided by the embodiments of the present application. As Figure 6 shown, the second frame sampling period A2 = 3. In the second frame rate mode, both the first camera and the second camera output image data at full frame rate, that is, each frame has image data. The image processing device can perform parallax calculation according to the second frame sampling period.

[0125] After the image processing device receives the first preview image and the second preview image, if both the first preview image and the second preview image are complete, that is, neither the first camera (Master) nor the second camera (Slave) has skipped frames, the beat counter is started to determine the cumulative value of the beat counter since the last parallax calculation. Assume that the beat counter reaches the second frame sampling period at the Nth frame. Then the image processing device determines the first image in the first preview image and the second image in the second preview image at the Nth frame. After that, the image processing device resets the beat counter (that is, sets the cumulative value of the mCurrentGap counter to 0) and sets the isDoStereoDepth signal (that is, sets the parallax calculation status identifier to isDoStereoDepth = true), and performs parallax calculation based on the first image and the second image to obtain the first depth image.

[0126] Further, if the beat counter reaches the second frame sampling period at the (N + 3)th frame, the image processing device determines the first image in the first preview image and the second image in the second preview image at the (N + 3)th frame. After that, the image processing device sets the cumulative value of the beat counter to 0 and sets the parallax calculation status identifier to isDoStereoDepth = true, and performs parallax calculation based on the first image and the second image to obtain the first depth image. And so on, the image processing device determines the first image in the first preview image and the second image in the second preview image at the (N + 6)th frame, the (N + 9)th frame, the (N + 12)th frame, and the (N + 15)th frame respectively, and performs parallax calculation based on the first image and the second image to obtain the first depth image.

[0127] In the embodiments of the present application, before the image processing device determines the cumulative value of the beat counter in response to the switching instruction from the first frame rate mode to the second frame rate mode, when the frame rate mode of the second camera is the first frame rate mode, the current frame data of the second camera is valid, and the current state of the second camera is non-static, the image processing device can perform parallax calculation based on the first frame sampling period, the first preview image, and the second preview image to obtain the second depth image; and at the same time record the cumulative value of the beat counter.

[0128] In an embodiment of the present application, whether the current frame data of the second camera is valid can be marked by the isCurSlaveExist identifier. That is to say, if isCurSlaveExist = true, it indicates that the current frame data of the second camera is valid; if isCurSlaveExist = false, it indicates that the current frame data of the second camera is invalid.

[0129] In an embodiment of the present application, the current state of the second camera can be marked by the isStable identifier. That is to say, if isStable = true, it indicates that the current state of the second camera is static; if isStable = false, it indicates that the current state of the second camera is non-static.

[0130] In some embodiments, before the image processing device determines the cumulative value of the beat counter in response to a switching instruction from the first frame rate mode to the second frame rate mode, it determines that the frame rate mode of the second camera is the first frame rate mode (i.e., isSkipLogicEnable = true), the current frame data of the second camera is valid (i.e., isCurSlaveExist = true), and the current state of the second camera is non-static (i.e., isStable = false). After that, the image processing device can perform parallax calculation based on the first frame sampling period, the first preview image, and the second preview image to obtain a second depth image; and simultaneously record the cumulative value of the beat counter.

[0131] In an embodiment of the present application, in the process of the image processing device performing parallax calculation based on the first frame sampling period, the first preview image, and the second preview image to obtain a second depth image, the image processing device can determine a third image in the first preview image based on the first frame sampling period; determine a fourth image in the second preview image based on the first frame sampling period; further, the image processing device can perform parallax calculation based on the third image and the fourth image to obtain a second depth image.

[0132] Among them, the third image is any frame image in the first preview image before the first image; the third image is any frame image in the first preview image before the first image.

[0133] In an embodiment of the present application, the third image refers to a certain frame image selected from a multi-frame image sequence collected by the first camera according to the interval rule of the first frame sampling period and located before the current first image in time sequence.

[0134] In an embodiment of the present application, the fourth image refers to a certain frame image selected from a multi-frame image sequence collected by the second camera according to the interval rule of the first frame sampling period and located before the current second image in time sequence.

[0135] In some embodiments, the image signal processors of the first camera and the second camera can output multiple frames of raw image data (i.e., the first preview image and the second preview image) continuously acquired to the system memory. Further, the image processing device can periodically select valid data frames that meet the requirements from the memory based on a preset first frame sampling period. Specifically, the system extracts target frames (i.e., the third image and the fourth image) with time synchronization from the first preview image and the second preview image respectively according to the interval rule; finally, the third image and the fourth image are input into the disparity calculation module, and the disparity calculation status identifier is set to isDoStereoDetph = true to perform disparity calculation.

[0136] In some embodiments, by way of example, Figure 7 FIG. is a schematic diagram of driving disparity calculation in the first frame rate mode provided by the embodiments of the present application. The first camera outputs image data at full frame rate, that is, there is image data in each frame; the second camera starts to adopt the first frame rate mode from the Nth frame, that is, the second camera outputs one frame of image data every 3 frames according to the first frame sampling period. That is to say, after the image processing device receives the first preview image and the second preview image, the second preview image is periodically missing due to the frame dropping strategy, that is, only 1 frame is valid every 3 frames.

[0137] When the image processing device detects that the frame skipping logic is activated at the Nth frame, it can set the frame skipping logic status identifier to isSkipLogicEnable = true; when the image processing device starts to perform disparity calculation in the first frame rate mode from the Nth frame in the second camera, it determines whether the current frame slave camera data is valid. If the current frame slave camera data is valid, it sets the identifier isCurSlaveExist = true to indicate that the current slave camera data is valid. Further, the image processing device determines whether the current state of the second camera is non-static. If the current state is non-static, it sets the identifier isStable = false.

[0138] As can be seen from the figure, at the Nth frame, the image processing device can set the frame skipping logic identifier to isSkipLogicEnable = true. After the image processing device determines that both the current frame master and slave data are valid at the Nth frame, it performs disparity calculation based on the third image and the fourth image to obtain the second depth image. And reset the beat counter (that is, set the accumulated value of the mCurrentGap counter to 0), and set the isDoStereoDepth signal (that is, set the disparity calculation status identifier to isDoStereoDepth = true).

[0139] By analogy, the image processing device performs parallax calculation at the (N + 3)-th frame; performs parallax calculation at the (N + 6)-th frame; performs parallax calculation at the (N + 9)-th frame; performs parallax calculation at the (N + 12)-th frame; performs parallax calculation at the (N + 15)-th frame.

[0140] It should be noted that in the state where isSkipLogicEnable = true, mCurrentGap is only used as a statistical indicator and does not participate in the threshold determination, which can ensure that the core logic in the frame reduction mode is not interfered by redundant counting.

[0141] In an embodiment of the present application, before the image processing device determines the cumulative value of the beat counter in response to a switching instruction from the first frame rate mode to the second frame rate mode, when the frame rate mode of the second camera is the first frame rate mode, if the current frame data of the second camera is invalid or the current state of the second camera is static, the image processing device determines the cumulative value of the beat counter; when the cumulative value of the beat counter reaches the second frame sampling period, the image processing device can perform parallax calculation based on the second frame sampling period, the first preview image, and the second preview image to obtain a third depth image.

[0142] In an embodiment of the present application, the current frame data of the second camera is invalid, that is, isCurSlaveExist = false. The current state of the second camera is static; that is, isStable = true.

[0143] In some embodiments, before the image processing device determines the cumulative value of the beat counter in response to a switching instruction from the first frame rate mode to the second frame rate mode, when it is determined that the frame rate mode of the second camera is the first frame rate mode (i.e., isSkipLogicEnable = true), if the image processing device determines that the current frame data of the second camera is invalid (i.e., isCurSlaveExist = false) or the current state of the second camera is static (i.e., isStable = true), the image processing device determines the cumulative value of the beat counter. After that, when the image processing device determines that the cumulative value of the beat counter reaches the second frame sampling period, parallax calculation can be performed based on the second frame sampling period, the first preview image, and the second preview image to obtain a third depth image.

[0144] In some embodiments, when the image processing device calculates the parallax based on the second frame sampling period, the first preview image, and the second preview image to obtain the third depth image, the image processing device may determine a fifth image in the first preview image based on the second frame sampling period; at the same time, determine a sixth image in the second preview image based on the second frame sampling period; further, the image processing device calculates the parallax based on the fifth image and the sixth image to obtain the third depth image.

[0145] It should be understood that when the frame rate mode of the second camera is the first frame rate mode, if the current frame data of the second camera is invalid, the image processing device needs to determine the cumulative value of the beat counter. If the current state of the second camera is static, the image processing device needs to determine the cumulative value of the beat counter. If the current frame data of the second camera is invalid and the current state of the second camera is static, the image processing device needs to determine the cumulative value of the beat counter, which is not limited in this embodiment of the present application.

[0146] In the embodiments of the present application, before the image processing device calculates the parallax based on the third image and the fourth image to obtain the second depth image, it may determine the acquisition time of the data to be aligned based on the data alignment period; further, when there is no image data corresponding to the acquisition time in the second preview image, the image processing device determines a target frame in the second preview image; where the position of the target frame is before the acquisition time; and data preparation is performed based on the target frame for the data to be aligned.

[0147] That is to say, the target frame is the image data frame corresponding to the nearest time before the acquisition time of the data to be aligned.

[0148] In the embodiments of the present application, the data alignment period may also be referred to as the calibration alignment beat period. The data alignment period refers to the frame interval period for performing the calibration alignment operation on the preview image streams of the binocular cameras (the first camera and the second camera).

[0149] Exemplarily, if the calibration alignment beat period is A3 (e.g., A3 = 8), the system performs the calibration alignment operation on the dual-channel images every 8 frames.

[0150] It should be noted that the data alignment period (A3 value) can be dynamically adjusted according to the system load and real-time requirements. The A3 value is increased in high-load scenarios to reduce the calculation frequency and ensure balanced resource allocation.

[0151] In the embodiments of the present application, the acquisition time of the data to be aligned refers to the specific frame time determined according to the data alignment period (calibration alignment beat period) for performing the image calibration alignment operation of the binocular cameras.

[0152] It should be understood that the acquisition time of the data to be aligned is the acquisition time before the second depth image.

[0153] In some embodiments, the image processing device can determine whether the first preview image and the second preview image are completely matched through the IsPrepareRectify identifier. Specifically, the IsPrepareRectify identifier is a boolean status variable dynamically calculated by the needDoRectify() function, and is used to identify whether the current frame data meets the following verification conditions: (1) Temporal integrity: The timestamp deviation between the main and secondary images is within the threshold range (e.g., <1ms); (2) Data validity: No key areas are lost in the dual-channel images (e.g., the ISP does not discard frames); (3) Geometric alignment ready: Preprocessing such as loading distortion correction parameters has been completed.

[0154] Exemplarily, for each frame of data of the first preview image collected by the first camera and the second preview image collected by the second camera, if the current frame data of the first preview image and the second preview image are completely matched, the image processing device can set the IsPrepareRectify identifier to true.

[0155] In some embodiments, the image processing device can determine whether the current frame can perform a calibration alignment operation through the isDoRectify identifier. If the current frame can perform a calibration alignment operation, the image processing device can set the identifier to isDoRectify = true; if the current frame cannot perform a calibration alignment operation, the image processing device can set the identifier to isDoRectify = false.

[0156] In some embodiments, the image processing device can determine the acquisition time of the data to be aligned based on the data alignment period; further, if there is image data corresponding to the acquisition time in the second preview image, the image processing device directly uses this data to perform a calibration alignment operation.

[0157] In some embodiments, the image processing device performs a calibration alignment operation (i.e., determines the acquisition time) according to the data alignment period (e.g., every 8 frames). If the frame data corresponding to the predetermined acquisition time is missing (such as frame loss or delay), and the current processing progress has not reached this time, the image processing device can determine the target frame at the nearest time before the acquisition time of the data to be aligned in the second preview image as the data preparation for the data to be aligned.

[0158] Exemplarily, Figure 8 is a schematic diagram of driving calibration alignment in the first frame rate mode provided by the embodiments of the present application, such as Figure 8As shown, the first camera outputs image data at full frame rate, that is, there is image data for each frame; the second camera starts from the Nth frame and adopts the first frame rate mode, that is, the second camera outputs one frame of image data every 3 frames according to the first frame sampling period. The first frame sampling period A1 = 3 and the data alignment period A3 = 8 in the first frame rate mode.

[0159] As can be seen from the figure, at the Nth frame, the image processing device can set the frame skipping logic identifier to isSkipLogicEnable = true. After the image processing device determines that the main and secondary data of the current frame are both valid at the Nth frame, it performs parallax calculation; and resets the beat counter (that is, sets the accumulated value of the mCurrentGap counter to 0), and sets the isDoStereoDepth signal (that is, sets the parallax calculation status identifier to isDoStereoDepth = true).

[0160] And so on, the image processing device performs parallax calculation at the N + 3th frame; and performs parallax calculation at the N + 6th frame.

[0161] Assume that the image processing device performs parallax calculation based on the third image and the fourth image at the N + 9th frame, as Figure 8 shown. In the case of frame skipping of the second camera, the image processing device can determine that the acquisition time of the data to be aligned is the N + 8th frame based on the data alignment period A3 = 8; however, since there is no secondary camera preview image data arriving at the N + 8th frame due to frame skipping in the second camera preview, the N + 8th frame cannot perform the calibration alignment operation, so data for its processing needs to be prepared in advance. Further, when there is no image data corresponding to the acquisition time in the second preview image, the image processing device determines that the N + 6th frame in the second preview image is before the N + 8th frame, and the N + 6th frame is the target frame with image data in the second preview image. Therefore, the image processing device can prepare data for the data to be aligned at the N + 6th frame.

[0162] In the embodiment of the present application, when the second camera is in the second frame rate mode, the image processing device can determine the accumulated value of the calibration alignment counter; when the accumulated value of the calibration alignment counter reaches the data alignment period, it performs the calibration alignment operation based on the data alignment period, the first preview image, and the second preview image.

[0163] In some embodiments, during the process of performing a calibration alignment operation based on a data alignment period, a first preview image, and a second preview image, the image processing device may determine a first image to be aligned in the first preview image based on the data alignment period, and at the same time determine a second image to be aligned in the second preview image based on the data alignment period; thereafter, the image processing device may perform a calibration alignment operation based on the first image to be aligned and the second image to be aligned, and use the data after calibration alignment for disparity calculation of the first image and the second image.

[0164] In some embodiments, by way of example, Figure 9 is a schematic diagram of driving calibration alignment in the second frame rate mode provided by an embodiment of the present application. As Figure 9 shown, in the normal frame rate mode (i.e., the second frame rate mode), both the first camera and the second camera output image data at full frame rate, that is, there is image data for each frame. The data alignment period A3 = 8.

[0165] As can be seen from the figure, at the Nth frame, the image processing device can perform a complete match on the main and secondary data of each frame through the IsPrepareRectify identifier. Thereafter, the image processing device records the cumulative number of frames since the last calibration alignment through a calibration alignment counter (i.e., the mRectify.mCurrentGap counter). When the cumulative value of the calibration alignment counter reaches the data alignment period A3, the calibration alignment counter is cleared, that is, mRectify.mCurrentGap = 0; further, the image processing device sets the isDoRectify flag (i.e., isDoRectify = true) to activate the calibration alignment process. And so on, the image processing device performs calibration alignment at the (N + 8)th frame.

[0166] It should be noted that in the second frame rate mode, IsPrepareRectify = true is synchronized with isDoRectify. If it indicates that the main and secondary data are complete, data processing can be performed on the current frame; since isDoRectify meets the conditions, it indicates that calibration alignment processing can be performed on the current frame.

[0167] In an embodiment of the present application, before the image processing device performs disparity calculation on a third image and a fourth image to obtain a second depth image, it may determine a first acquisition time of the data to be aligned based on the data alignment period. Thereafter, when there is no image data corresponding to the first acquisition time in the second preview image and a target frame is determined in the second preview image, if the time interval between the first acquisition time and a second acquisition time corresponding to the fourth image is less than a preset threshold, the first acquisition time is adjusted based on a preset adjustment period to obtain an adjusted first acquisition time; wherein, the position of the target frame is before the first acquisition time.

[0168] That is to say, the target frame is the image data frame corresponding to the nearest moment before the acquisition moment of the data to be aligned.

[0169] In some embodiments, the preset threshold may be a time window parameter (in milliseconds or frame intervals) for determining whether it is necessary to dynamically adjust the acquisition moment of data alignment.

[0170] It should be understood that the preset threshold can be set according to requirements, and the embodiments of the present application do not limit this.

[0171] It should be understood that the time interval between the first acquisition moment and the second acquisition moment corresponding to the fourth image is less than the preset threshold. That is to say, the data image after calibration alignment at the first acquisition moment needs to be used when calculating the parallax at the second acquisition moment. If the time interval between the first acquisition moment and the second acquisition moment is less than the preset threshold, the first acquisition moment may not be able to provide calibrated alignment data to the second acquisition moment in time due to timeout or frame loss, which may affect the accuracy of calculating the second depth image from the third image and the fourth image.

[0172] Exemplarily, assume that the preset threshold is 2. If the first acquisition moment corresponds to the 8th frame and the second acquisition moment corresponds to the 9th frame, it can be seen that the time interval between the first acquisition moment and the second acquisition moment is less than the preset threshold. Then, the image processing device can adjust the first acquisition moment forward. For example, perform calibration alignment at the 7th frame.

[0173] Another exemplarily, assume that the preset threshold is 2. If the first acquisition moment corresponds to the 5th frame and the second acquisition moment corresponds to the 9th frame, it can be seen that the time interval between the first acquisition moment and the second acquisition moment is greater than the preset threshold. Then, the image processing device can not adjust the first acquisition moment, that is, perform calibration alignment at the 5th frame.

[0174] In some embodiments, after the image processing device determines the first acquisition moment of the data to be aligned based on the data alignment period, it will further determine whether there is image data corresponding to this moment in the second preview image. If the image processing device determines that the frame data corresponding to the first acquisition moment is missing (that is, there is no data at this moment in the second preview image), and the position of the target frame is before the first acquisition moment. At this time, the image processing device will determine the time interval between the first acquisition moment and the second acquisition moment.

[0175] If the time interval between the first acquisition moment and the second acquisition moment is greater than or equal to the preset threshold, it means that the distance between the two acquisition moments is large enough, and even if the first acquisition moment is not adjusted, it will not affect the accuracy of parallax calculation. Then, the image processing device does not need to adjust the first acquisition moment.

[0176] On the contrary, if the time interval between the first acquisition time and the second acquisition time is less than a preset threshold, it indicates that the two acquisition times are too close. It may be due to timeout or frame loss at the first acquisition time, and the correction alignment data cannot be provided to the second acquisition time in time, which may affect the accuracy of the second depth image. At this time, the image processing device can adjust the first acquisition time forward based on a preset adjustment period (for example, one period in advance) to obtain the adjusted first acquisition time.

[0177] In some embodiments, during the process of adjusting the first acquisition time by the image processing device based on a preset adjustment period, it can be determined whether there is image data of the adjusted first acquisition time in the second preview image. If there is image data corresponding to the first acquisition time, the image processing device can directly perform correction alignment based on the adjusted first acquisition time.

[0178] In some embodiments, during the process of adjusting the first acquisition time by the image processing device based on a preset adjustment period, if it is determined that there is no image data of the adjusted first acquisition time in the second preview image, the image processing device can prepare data with the target frame as the data to be aligned, and perform a correction alignment operation using the alignment data on the target frame at the adjusted first acquisition time.

[0179] Exemplarily, Figure 10 is a schematic diagram of driving correction alignment in the first frame rate mode provided by the embodiments of the present application. As Figure 10 shown, in the first frame rate mode, the first camera outputs image data at full frame rate, that is, each frame has image data; the second camera starts to adopt the first frame rate mode from the Nth frame, that is, the second camera outputs one frame of image data every 3 frames according to the first frame sampling period. The first frame sampling period A1 = 3 in the first frame rate mode, and the data alignment period T = 8.

[0180] It can be seen from the figure that at the Nth frame, the image processing device can set the frame skipping logic identifier to isSkipLogicEnable = true. Then, the image processing device performs parallax calculation at the Nth frame; performs parallax calculation at the N + 3th frame; performs parallax calculation at the N + 6th frame; performs parallax calculation at the N + 9th frame, performs parallax calculation at the N + 12th frame, and performs parallax calculation at the N + 15th frame.

[0181] Assume that the image processing device performs parallax calculation based on the third image and the fourth image at the N + 9th frame. As Figure 10As shown, in the case of frame skipping of the second camera, the image processing device can determine that the acquisition time of the data to be aligned is the (N + 8)-th frame. However, since there is no sub-camera preview image data arriving at the (N + 8)-th frame due to frame skipping in the second camera preview, the (N + 8)-th frame cannot perform the calibration alignment operation. Therefore, it is necessary to prepare the data to be processed in advance for it. Further, in the case where there is no image data corresponding to the acquisition time in the second preview image, the image processing device determines that the (N + 6)-th frame in the second preview image is before the (N + 8)-th frame, and the (N + 6)-th frame is the target frame with image data in the second preview image. Therefore, the image processing device can prepare data for the data to be aligned at the (N + 6)-th frame.

[0182] Further, if the preset threshold is 2, the image processing device can determine that the time interval between the first acquisition time (i.e., the (N + 8)-th frame) and the second acquisition time (i.e., the (N + 9)-th frame) is less than 2, which indicates that the two acquisition times are too close. It may be that the (N + 8)-th frame times out and cannot provide calibration alignment data to the (N + 9)-th frame in time, which may further affect the accuracy of the second depth image. At this time, the image processing device can adjust the first acquisition time forward by one acquisition time based on the preset adjustment period, that is, adjust it to the (N + 7)-th frame for calibration alignment processing. After that, the image processing device clears the calibration alignment counter at the (N + 7)-th frame, that is, mRectify.mCurrentGap = 0; further, the image processing device sets the isDoRectify flag (i.e., isDoRectify = true) to activate the calibration alignment process.

[0183] An embodiment of the present application provides an image processing method. The image processing method is applied to an image processing device configured with a first camera and a second camera. The method includes: in response to a switching instruction from a first frame rate mode to a second frame rate mode, determining the accumulated value of a beat counter; wherein, the first frame rate mode drives parallax calculation based on a first frame sampling period; the second frame rate mode drives parallax calculation based on a second frame sampling period; the first frame sampling period is greater than or equal to the second frame sampling period; switching from the first frame rate mode to the second frame rate mode based on the accumulated value of the beat counter and the second frame sampling period; performing parallax calculation based on the second frame sampling period, a first preview image, and a second preview image to obtain a first depth image; wherein, the first preview image is captured by the first camera, and the second preview image is captured by the second camera. That is to say, in the embodiment of the present application, in the first frame rate mode, the image processing device continuously records the accumulated value of the beat counter. In response to a switching instruction from the first frame rate mode to the second frame rate mode, the image processing device performs collaborative control based on the accumulated value and the second frame sampling period to ensure the synchronization of the beat signal, thereby avoiding data loss or processing interruption caused by timing misalignment, and effectively preventing problems such as preview screen freezing or unnatural virtualization effects. During the parallax calculation process, the second frame sampling period is used to perform precise frame synchronization and selective processing on the preview images captured by the dual cameras. While optimizing the allocation efficiency of computing resources, it also significantly improves the real-time performance and accuracy of depth calculation, enabling the system to complete complex processing within a limited time. It can be seen that the image processing method proposed in the embodiment of the present application realizes seamless switching between different frame rate modes through the dynamic monitoring of the beat counter and the collaborative control of the sampling period, ensuring the consistency and continuity of the data stream, and improving the overall processing efficiency of the system, thereby enhancing the image processing effect.

[0184] Based on the above embodiment, another embodiment of the present application provides an image processing method. In view of the above problems faced by the traditional portrait virtualization solution, an optimized image processing flow scheme for dynamic adaptation of the preview frame rate of the secondary camera for portraits is designed.

[0185] Among them, the image processing method proposed in the embodiment of the present application involves the following solutions:

[0186] (1) In the second frame rate mode, the core logics such as parallax calculation are driven by the second frame sampling period (i.e., the beat signal); when the second camera switches from the second frame rate mode to the first frame rate mode, the second frame sampling period is switched to the first frame sampling period (i.e., the valid preview frame signal of the secondary camera) for driving, and at the same time, the accumulated value of the beat counter is retained.

[0187] (2) When the second camera resumes from the first frame rate mode to the second frame rate mode, it is smoothly switched back to being driven by the second frame sampling period through the accumulated value of the beat counter.

[0188] (3) In the first frame rate mode, by predicting the data alignment period (i.e., the calibration alignment beat), the main and sub-frame data preparation is selectively completed in advance, and the control beat is finely adjusted.

[0189] In the embodiments of the present application, Figure 11 is a schematic diagram of the implementation framework for binocular portrait blurring preview provided by the embodiments of the present application. As Figure 11 shown, it involves the following processes:

[0190] 1. Input of main and sub-image data. Obtain synchronized main and sub-camera image data: In the binocular blurring algorithm, obtaining synchronized main and sub-camera image data is the basis of the entire algorithm. The synchronously obtained image data ensures the accuracy and effect of the following steps (calibration alignment, disparity calculation, disparity map optimization, etc.).

[0191] 2. Determine the frame rate mode. According to the status of the main and sub-image data and system parameters (such as resource load, scene complexity), the current frame control mode is dynamically determined by the frame rate adaptive control unit: (a) Normal frame rate mode (i.e., the second frame rate mode): Use the beat signal (i.e., the second frame sampling period) to drive the key logic (such as calibration alignment, disparity calculation). (b) Frame rate reduction mode (i.e., the first frame rate mode): Switch from driving by the beat signal to driving the key logic by the sub-camera effective preview frame signal (i.e., the first frame sampling period), and synchronously retain the beat count (i.e., the cumulative value of the beat counter) to provide a timing reference for switching from the first frame rate mode to the second frame rate mode.

[0192] Furthermore, the image processing device can dynamically generate the drive control signals of the key logic (such as isDoRectify, isDoStereoDepth, isDoRefine), and realize the seamless switching of the dual drive mode according to the identifier status of each drive control signal.

[0193] 3. Preprocessing and depth information calculation.

[0194] 3.1. Motion detection (control condition is isDoDetect): Analyze the dynamic changes in the main and sub-images (such as the movement of the main body or background disturbance) to provide input for the frame rate adaptive control.

[0195] 3.2. Calibration alignment (control condition is isDoRectify): (a) Eliminate the disparity error: The main function of calibration alignment is to eliminate the disparity error caused by misalignment of the binocular cameras. Through calibration alignment, the corresponding pixel points of the two images will be on the same horizontal line. (b) Unify the coordinate system: Project the two images into a unified coordinate system to make the subsequent disparity calculation more convenient and accurate.

[0196] For example, in normal frame rate mode, the main and sub-images are aligned based on the data alignment period. In reduced frame rate mode, the main and sub-images are spatially aligned in advance through predictive logic behavior, and the data timeliness is ensured in combination with the beat fine-tuning mechanism.

[0197] 3.3. Depth map calculation.

[0198] 3.3.1. Disparity calculation (control condition isDoStereoDetph): (a) Measuring depth information: The main function of disparity calculation is to find the disparity corresponding to each pixel in the aligned image, that is, the horizontal displacement of the corresponding points of the same object in the left and right images. The disparity value is inversely proportional to the distance from the object to the camera, so it can be used to measure depth information. (b) Constructing a disparity map: The disparity map is a grayscale image, and the grayscale value of each pixel represents the disparity (i.e., depth information) of the point, which is used to further generate a depth map and subsequent processing.

[0199] 3.3.2 Disparity map optimization: (a) Eliminate noise: Noise and mismatch may occur during the disparity calculation process. The purpose of disparity map optimization is to eliminate these noises and improve the quality of the disparity map. (b) Smooth disparity map: Through filtering and interpolation, the disparity map is made smoother and more continuous, improving the accuracy of depth information.

[0200] 3.3.3. Depth map generation: Convert the physical distance into a depth map. The depth map represents the distance from each pixel in the image to the camera, which helps to understand the three-dimensional structure of the scene and is the basic data for subsequent calculation of blur mask and application of blur effect.

[0201] 3.3.4. Time domain fusion: Use the depth information of previous and next frames (such as Flow4 technology) to perform time domain fusion, eliminate single frame noise, and enhance the stability and continuity of the depth map.

[0202] 3.3.5. Artificial Intelligence (AI) optimization (controlled by isDoRefine): The neural network model is used to jointly optimize the main image data of the current frame and the historical depth map to enhance the edge details of the subject and the naturalness of the blur transition.

[0203] 3.3.6. Static processing (control condition isStable). In static mode, perform fast operations on related logic.

[0204] 4. Blur rendering processing: Blur the image according to the depth map, making the foreground clear and the background blurred, achieving a depth of field effect similar to that of a large-aperture SLR camera and enhancing the beauty of the image.

[0205] 4.1. Calculate the blurred encoding map. (a) Determine the degree of blurring: The blurred mask map represents the blurring intensity to be applied to each region in the image. Generally, the foreground remains clear while the background gradually blurs. (b) Control the blurring effect: Control the amount of blurring for each pixel point according to the depth value to enhance the three-dimensional sense.

[0206] 4.2. Render the blurring effect. (a) Implement the blurring effect: Blur the original image according to the blurred mask map, making the foreground clear and the background gradually blurred, thereby simulating the depth-of-field effect. (b) Enhance the aesthetics of the image: It can make the foreground subject more prominent and enhance the artistic effect of the image.

[0207] Finally, output the rendering result to the preview interface, supporting users to adjust the composition in real time or capture it as a high-quality static image.

[0208] According to Figure 11 As shown in the framework schematic diagram, if the motion detection, calibration alignment, depth map calculation module, and AI optimization can all achieve full-frame rate trigger execution, theoretically the optimal effect can be achieved. However, limited by the camera hardware computing power, energy efficiency constraints, and multi-task resource competition, it is necessary to trigger the frequency through the dynamic regulation module to achieve a three-dimensional balance among the effect quality, system performance, and power consumption. Among them, the depth map calculation module has the most stringent requirements for computing resources because it needs to execute high-density disparity operations.

[0209] When the sub-camera input skips frames due to the trigger ratio of the energy efficiency optimization strategy, it is necessary to jointly process the dynamic fluctuations and mutation scenarios of the frame rate. The frame rate adaptive control unit, as the top-level regulation center of the algorithm, is responsible for determining the trigger timing and mode switching of the decision logic to ensure the global optimal solution of resource allocation and real-time constraints.

[0210] In the embodiments of the present application, Figure 12 is a schematic flow diagram of the trigger mechanism of the disparity calculation logic when the frame rate mode is switched in the embodiments of the present application. As Figure 12 shown, the execution steps are as follows:

[0211] S1201. Determine the first frame sampling period and the second frame sampling period.

[0212] S1202. Determine the cumulative value of the beat counter.

[0213] S1203. Determine whether the frame rate mode of the second camera is the first frame rate mode; if so, execute S1204; otherwise, execute S1209.

[0214] Exemplarily, the image processing device may acquire the frame skipping logic state and set a frame skipping logic identifier (which may also be expressed as isSkipLogicEnable) according to the frame skipping logic state. If the frame skipping logic state is the active state, the frame skipping logic identifier may be set to isSkipLogicEnable = true. If the frame skipping logic state is the off state, the frame skipping logic identifier may be set to isSkipLogicEnable = false.

[0215] S1204. Determine whether the current frame of the second camera is valid. If so, execute S1205; otherwise, execute S1208.

[0216] Exemplarily, whether the current frame data of the second camera is valid may be marked by the isCurSlaveExist identifier. That is, if isCurSlaveExist = true, it indicates that the current frame data of the second camera is valid; if isCurSlaveExist = false, it indicates that the current frame data of the second camera is invalid.

[0217] S1205. Determine whether the current state of the second camera is non-static. If so, execute S1206; otherwise, execute S1208.

[0218] Exemplarily, the current state of the second camera may be marked by the isStable identifier. That is, if isStable = true, it indicates that the current state of the second camera is static; if isStable = false, it indicates that the current state of the second camera is non-static.

[0219] S1206. Clear the accumulated value of the beat counter.

[0220] S1207. Perform parallax calculation based on the first frame sampling period, the first preview image, and the second preview image to obtain the second depth image.

[0221] S1208. Record the accumulated value of the beat counter.

[0222] S1209. When the accumulated value of the beat counter reaches the second frame sampling period, clear the accumulated value of the beat counter.

[0223] S1210. Perform parallax calculation (i.e., isDoStereoDetph = true) based on the second frame sampling period, the first preview image, and the second preview image to obtain the first depth image.

[0224] It can be seen that in the image processing method proposed in the embodiments of the present application, in the normal frame rate mode, the core logics such as parallax calculation are driven by the beat signal (i.e., the second frame sampling period); in the frame dropping mode, the driving is switched to the effective preview frame signal of the secondary camera (i.e., the first frame sampling period), and the beat counting is retained at the same time. When restoring to the normal frame rate mode, it is smoothly switched back to the beat drive to achieve a seamless trigger, realizing the seamless connection between the normal frame rate and the frame dropping mode, supporting complex scenarios such as mode switching, zooming, and resolution adjustment, improving the algorithm adaptation ability, greatly reducing the system abnormal interruption rate, and accelerating the convergence and stability of the defocusing effect.

[0225] In the embodiments of the present application, Figure 13 is a schematic flowchart of the correction alignment logic trigger mechanism during frame rate mode switching provided by the embodiments of the present application. As Figure 13 shown, the steps are as follows:

[0226] S1301. Determine the cumulative value of the correction alignment counter (i.e., the mRectify.mCurrentGap counter).

[0227] S1302. Determine whether the frame rate mode of the second camera is the first frame rate mode; if so (i.e., isSkipLogicEnable = true), then execute S1303; otherwise, execute S1304.

[0228] S1303. Execute the multi-logic unit cooperation and beat fine-tuning operations in the first frame rate mode.

[0229] S1304. Determine whether the current frame data of the first preview image and the second preview image are completely matched; if so (i.e., IsPrepareRectify = true), then execute S1305.

[0230] Exemplarily, the image processing device can determine whether the first preview image and the second preview image (which can also be referred to as the main and secondary image data) are completely matched through the IsPrepareRectify identifier. The IsPrepareRectify identifier is a boolean status variable and can be dynamically calculated by the needDoRectify() function.

[0231] S1305. When the cumulative value of the correction alignment counter reaches the data alignment period, clear the correction alignment counter (i.e., mRectify.mCurrentGap = 0).

[0232] S1306. Execute the correction alignment process.

[0233] Among them, during the execution of the correction alignment, the isDoRectify identifier can be set (i.e., isDoRectify = true).

[0234] In an embodiment of the present application, Figure 14 is a schematic flow diagram of multi-logic unit collaboration and beat fine-tuning provided by an embodiment of the present application. As Figure 14 shown, the steps are as follows:

[0235] S1401. Determine whether to perform parallax calculation for the current frame. If so (i.e., isDoStereoDetph == true), then execute S1403; otherwise, execute S1402.

[0236] S1402. Reset the status of the secondary camera data preparation.

[0237] Exemplarily, the image processing device may set the IsPrepareRectify status flag to false. Among them, IsPrepareRectify = false indicates that the current secondary camera preview skips frames and there is no need to prepare data.

[0238] S1403. Update the calibration alignment beat count.

[0239] Exemplarily, calculate and update mRectify.futureGap (the remaining beats until the next calibration alignment trigger). The counter mRectify.futureGap records the remaining beats required until the next trigger for calibration alignment (isDoRectify). It should be noted that mRectify.futureGap = mRectify.getFrameGap() - mRectify.getCurrentGap(), where mRectify.getFrameGap() is the number of frames in the fixed period of calibration alignment; mRectify.getCurrentGap() is the cumulative value of the calibration alignment counter.

[0240] S1404. Determine whether the next calibration alignment falls within the non-critical interval between two parallax calculations (i.e., mRectify.futureGap < mDepth.getFrameGap). If so, then execute S1405.

[0241] That is to say, determine whether mRectify.futureGap of the current frame is less than mDepth.getFrameGap, where mDepth.getFrameGap is the number of frames in the period of parallax calculation.

[0242] S1405. Prepare the target frame data in advance.

[0243] Among them, taking the first parallax calculation frame in the two parallax calculations as the target frame, prepare the data to be aligned for it in advance.

[0244] S1406. Determine whether the interval between the next calibration alignment and the next frame parallax calculation is less than a preset threshold. If so, execute 1407.

[0245] Among them, if the interval between the next calibration alignment and the next frame parallax calculation is less than the preset threshold, it indicates that the next parallax calculation may be affected by timeout.

[0246] S1407. Adjust the position where the next calibration alignment is located based on a preset adjustment period to obtain an adjusted calibration alignment position.

[0247] Exemplarily, according to the preset adjustment period (such as a dynamic offset), adjust the calibration alignment trigger frame from the originally planned N + k to N + (k - 1) to reserve sufficient processing time.

[0248] It can be seen that the multi-logical unit collaboration and beat fine-tuning operation method proposed in the embodiments of the present application can achieve the following effects:

[0249] First aspect: In the first frame rate mode, the system anticipates the beats of the core logic (such as the calibration alignment logic), selectively completes the main and sub-frame data preparation in advance, breaks through the dilemma of limited data processing during frame rate reduction, and improves data timeliness.

[0250] Second aspect: Precisely fine-tune the beats in the frame rate reduction mode to ensure that the key logic closely follows the data frame update, solve the problem of lag in core logic processing, and improve data timeliness. Significantly enhance the efficiency, stability, and system compatibility of the data stream.

[0251] In summary, based on the logic anticipation and beat fine-tuning mechanism, it is ensured that core logics such as calibration alignment and parallax calculation closely follow the data frame update, reducing the data stream processing delay, being compatible with historical logics and multiple hardware platforms, and supporting stable output in high frame rate mutation scenarios (with a significant improvement in the frame rate fluctuation tolerance).

[0252] The embodiments of the present application propose an image processing method that realizes seamless connection between the first frame rate mode and the second frame rate mode through dynamic switching of the dual drive mode (beat signal and sub-camera frame signal), supports complex scenarios such as mode switching, zooming, and resolution adjustment, improves the algorithm adaptation ability, greatly reduces the system abnormal interruption rate, and accelerates the convergence and stability of the defocusing effect.

[0253] Furthermore, based on the logic anticipation and beat fine-tuning mechanism, it is ensured that core logics such as calibration alignment and parallax calculation closely follow the data frame update, reducing the data stream processing delay, being compatible with historical logics and multiple hardware platforms, and supporting stable output in high frame rate mutation scenarios (with a significant improvement in the frame rate fluctuation tolerance).

[0254] Based on the above embodiments, in another embodiment of the present application, Figure 15 It is a schematic structural diagram of the composition of the image processing device provided by the embodiment of the present application. As Figure 15 shown, the image processing device 150 proposed by the embodiment of the present application may include:

[0255] A determination unit 1501, configured to determine the cumulative value of the beat counter in response to a switching instruction from the first frame rate mode to the second frame rate mode; wherein, the first frame rate mode drives parallax calculation based on the first frame sampling period; the second frame rate mode drives parallax calculation based on the second frame sampling period; the cumulative value of the beat counter is the cumulative number of frames after the previous parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period;

[0256] A switching unit 1502, configured to switch from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period;

[0257] A calculation unit 1503, configured to perform parallax calculation based on the second frame sampling period, the first preview image, and the second preview image to obtain a first depth image; wherein, the first preview image is captured by the first camera, and the second preview image is captured by the second camera.

[0258] In the embodiment of the present application, further, Figure 16 It is a schematic structural diagram of the composition of the electronic device provided by the embodiment of the present application. As Figure 16 shown, the electronic device 160 proposed by the embodiment of the present application may include a processor 1601, a memory 1602, a communication interface 1603, and a bus 1604 for connecting the processor 1601, the memory 1602, and the communication interface 1603.

[0259] In an embodiment of the present application, the above-mentioned processor 1601 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic devices for implementing the functions of the above-mentioned processor may also be others, and the embodiments of the present application do not make specific limitations. The electronic device 160 may further include a memory 1602, and the memory 1602 may be connected to the processor 1601. Among them, the memory 1602 is used to store executable program codes, and the program codes include computer operation instructions. The memory 1602 may include a high-speed RAM memory and may also include non-volatile memory, for example, at least two disk memories.

[0260] In an embodiment of the present application, the bus 1604 is used to connect the communication interface 1603, the processor 1601, and the memory 1602 and for mutual communication between these devices.

[0261] In practical applications, the above-mentioned memory 1602 may be a volatile memory, such as a Random-Access Memory (RAM); or a non-volatile memory, such as a Read-Only Memory (ROM), a flash memory, a Hard Disk Drive (HDD), or a Solid-State Drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 1601.

[0262] Further, in an embodiment of the present application, the processor 1601 is configured to: in response to a switching instruction from a first frame rate mode to a second frame rate mode, determine the cumulative value of the beat counter; wherein, the first frame rate mode drives parallax calculation based on a first frame sampling period; the second frame rate mode drives parallax calculation based on a second frame sampling period; the cumulative value of the beat counter is the cumulative number of frames after the previous parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period; switch from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period; perform parallax calculation based on the second frame sampling period, the first preview image, and the second preview image to obtain a first depth image; wherein, the first preview image is captured by a first camera, and the second preview image is captured by a second camera.

[0263] In addition, in this embodiment, each functional module can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0264] If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0265] The embodiment of the present application provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the above image processing method is implemented.

[0266] Specifically, the program instructions corresponding to an image processing method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to an image processing method in the storage medium are read or executed by an electronic device, the following steps are included:

[0267] In response to a switching instruction from the first frame rate mode to the second frame rate mode, determine the cumulative value of the beat counter; wherein, the first frame rate mode drives parallax calculation based on a first frame sampling period; the second frame rate mode drives parallax calculation based on a second frame sampling period; the cumulative value of the beat counter is the cumulative number of frames after the previous parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period;

[0268] Switch from the first frame rate mode to the second frame rate mode based on the cumulative value of the beat counter and the second frame sampling period;

[0269] Perform parallax calculation based on the second frame sampling period, the first preview image, and the second preview image to obtain a first depth image; wherein, the first preview image is captured by a first camera, and the second preview image is captured by a second camera.

[0270] The embodiments of the present application also provide a computer program product.

[0271] In some embodiments, the computer program product may include a computer program or instructions.

[0272] In some embodiments, the computer program product can be applied to the computer device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the computer device in each method of the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0273] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0274] The present application is described with reference to the schematic flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the schematic flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the schematic flowchart and / or block diagram can also be implemented. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in one or more of the processes or multiple processes and / or blocks Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0275] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions specified in one or more of the flowcharts and / or boxes of the implementation process schematic Figure 1 one or more of the processes and / or boxes Figure 1 in the one or more boxes.

[0276] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the flowcharts and / or boxes Figure 1 one or more of the processes and / or boxes Figure 1 in the one or more boxes.

[0277] As described above, it is only the preferred embodiment of the present application, and is not used to limit the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The image processing method is applied to an image processing device equipped with a first camera and a second camera, and the method comprises: In response to a switching instruction from a first frame rate mode to a second frame rate mode, determining a cumulative value of a beat counter; wherein the first frame rate mode drives disparity calculation based on a first frame sampling period; the second frame rate mode drives disparity calculation based on a second frame sampling period; the cumulative value of the beat counter is the cumulative number of frames after the last disparity calculation; the first frame sampling period is greater than or equal to the second frame sampling period; switching from the first frame rate mode to the second frame rate mode based on the accumulated value of the beat counter and the second frame sampling period; A first depth image is obtained by performing parallax calculation based on the second frame sampling period, the first preview image, and the second preview image; wherein the first preview image is collected by the first camera, and the second preview image is collected by the second camera.

2. The method according to claim 1, characterized in that: The switching from the first frame rate mode to the second frame rate mode based on the accumulated value of the beat counter and the second frame sampling period comprises: When the accumulated value of the beat counter reaches an integral multiple of the second frame sampling period, the first frame rate mode is switched to the second frame rate mode.

3. The method according to claim 2, characterized in that The switching from the first frame rate mode to the second frame rate mode based on the accumulated value of the beat counter and the second frame sampling period comprises: When the accumulated value of the beat counter does not reach an integral multiple of the second frame sampling period, updating the accumulated value of the beat counter; Until the accumulated value of the updated beat counter reaches an integral multiple of the second frame sampling period, the first frame rate mode is switched to the second frame rate mode.

4. The method according to any one of claims 1 to 3, characterized in that The performing disparity calculation based on the second frame sampling period, the first preview image and the second preview image to obtain the first depth image includes: determining a first image in the first preview image based on the second frame sampling period; determining a second image in the second preview image based on the second frame sampling period; Disparity calculation is performed based on the first image and the second image to obtain a first depth image.

5. The method according to claim 1, characterized in that Before determining the accumulated value of the beat counter in response to the switching instruction from the first frame rate mode to the second frame rate mode, the method further includes: When the frame rate mode of the second camera is the first frame rate mode, current frame data of the second camera is valid, and the current state of the second camera is non-static, performing disparity calculation based on the first frame sampling period, the first preview image, and the second preview image to obtain a second depth image; The accumulated value of the beat counter is recorded.

6. The method according to claim 5, characterized in that The performing disparity calculation based on the first frame sampling period, the first preview image, and the second preview image to obtain a second depth image includes: determining a third image in the first preview image based on the first frame sampling period; determining a fourth image in the second preview image based on the first frame sampling period; A second depth image is obtained by performing disparity calculation based on the third image and the fourth image; wherein the third image is any frame image before the first image in the first preview image; and the third image is any frame image before the first image in the first preview image.

7. The method according to claim 6, characterized in that Before performing disparity calculation based on the third image and the fourth image to obtain the second depth image, the method further includes: Determine the collection time of the data to be aligned based on the data alignment period; In the case that the image data corresponding to the acquisition time does not exist in the second preview image, determining a target frame in the second preview image; wherein the position of the target frame is before the acquisition time; Data preparation is performed for the data to be aligned based on the target frame.

8. The method according to claim 6, characterized in that Before performing disparity calculation based on the third image and the fourth image to obtain the second depth image, the method further includes: Determine a first acquisition time of the data to be aligned based on the data alignment period; In a case where there is no image data corresponding to the first acquisition moment in the second preview image and a target frame is determined in the second preview image, if the time interval between the first acquisition moment and the second acquisition moment corresponding to the fourth image is less than a preset threshold, the first acquisition moment is adjusted based on a preset adjustment period to obtain an adjusted first acquisition moment; wherein the position of the target frame is before the first acquisition moment.

9. The method according to claim 1, characterized in that: The method further comprises: When the frame rate mode of the second camera is the first frame rate mode, if the current frame data of the second camera is invalid or the current state of the second camera is static, determining the accumulated value of the beat counter; When the accumulated value of the beat counter reaches the second frame sampling period, a disparity calculation is performed based on the second frame sampling period, the first preview image, and the second preview image to obtain a third depth image.

10. An image processing device, characterized in that: The image processing device comprises: A determination unit, configured to determine a cumulative value of a beat counter in response to a switching instruction from a first frame rate mode to a second frame rate mode; wherein the first frame rate mode drives the parallax calculation based on a first frame sampling period; the second frame rate mode drives the parallax calculation based on a second frame sampling period; the cumulative value of the beat counter is the cumulative number of frames after the last parallax calculation; the first frame sampling period is greater than or equal to the second frame sampling period; a switching unit, configured to switch from the first frame rate mode to the second frame rate mode based on the accumulated value of the beat counter and the second frame sampling period; A calculation unit is used to perform disparity calculation based on the second frame sampling period, the first preview image and the second preview image to obtain a first depth image; wherein the first preview image is collected by the first camera, and the second preview image is collected by the second camera.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images