Image preview method and device, electronic equipment and computer storage medium

Abstract

The embodiment of the invention discloses an image previewing method, which comprises the following steps of: acquiring a current frame, continuously determining a processing parameter diagram of a previous frame in a period of the current frame under the condition that the processing parameter diagram of the previous frame is not determined in the period of the previous frame of the current frame to obtain the processing parameter diagram of the previous frame, and previewing the current frame according to the processing parameter diagram of the previous frame. And processing the current frame to obtain a preview picture of the current frame. The embodiment of the invention also provides an image preview device, electronic equipment and a computer storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image preview method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] At present, with the improvement of camera performance of mobile terminal equipment and the development of computational photography technology, portrait mode has become one of the core functions of smart phones, tablet computers and other equipment. This mode simulates the shallow depth of field effect of a professional camera, clearly highlights the shooting subject (portrait), and artistically blurs the background (bokeh effect), thereby improving the visual aesthetics and professionalism of the photo.

[0003] Among them, this effect is not only used for static photo shooting, and users' requirements for the blur effect in real-time preview are also increasing, and they hope to see the final blur effect directly when framing and composing, so as to accurately adjust the pose and composition.

[0004] In related technologies, although the real-time portrait preview blur (RTB) technology can achieve certain effects, there is a technical problem of preview picture lag in high frame rate, high resolution, complex scene and resource limited mobile devices. SUMMARY

[0005] The embodiments of the present application provide an image preview method, device, electronic equipment and computer storage medium, which can improve the smoothness of the preview picture.

[0006] The technical solution of the present application is implemented as follows:

[0007] In a first aspect, the embodiments of the present application provide an image preview method, comprising:

[0008] obtaining a current frame;

[0009] if a processing parameter map of a previous frame of the current frame is not determined within a period of the previous frame, successively determining the processing parameter map of the previous frame within a period of the current frame to obtain the processing parameter map of the previous frame;

[0010] processing the current frame according to the processing parameter map of the previous frame to obtain a preview picture of the current frame.

[0011] In a second aspect, the embodiments of the present application provide an image preview device, comprising:

[0012] an obtaining module configured to obtain a current frame;

[0013] determining, in a case where the processing parameter map of the previous frame is not determined in a period of the previous frame, the processing parameter map of the previous frame in succession in a period of the current frame, to obtain the processing parameter map of the previous frame;

[0014] previewing, according to the processing parameter map of the previous frame, the current frame to obtain a preview picture of the current frame.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a storage medium having processor-executable instructions stored therein; the storage medium performs operations in dependence on the processor; when the instructions are executed by the processor, the image preview method in the above one or more embodiments is performed.

[0016] In a fourth aspect, an embodiment of the present application provides a computer storage medium having executable instructions stored therein; when the executable instructions are executed by one or more processors, the image preview method in the above one or more embodiments is performed.

[0017] The embodiments of the present application provide an image preview method, device, electronic device and computer storage medium, including: obtaining a current frame, in a case where a processing parameter map of a previous frame of the current frame is not determined in a period of the previous frame, determining the processing parameter map of the previous frame in succession in a period of the current frame to obtain the processing parameter map of the previous frame, and processing the current frame according to the processing parameter map of the previous frame to obtain a preview picture of the current frame; that is, in the embodiments of the present application, the processing parameter of the previous frame determined in the period of the previous frame is determined in succession in the period of the current frame, and after the processing parameter map of the previous frame is determined in the period of the current frame, the current frame is processed based on this to obtain the preview picture of the current frame, so that the cross-frame computing power allocation mechanism is established between the two consecutive frames, the single-frame computing power overload is avoided, the smooth allocation and efficient use of system computing power resources are realized, the risk of frame lag caused by computing power peak is reduced, and the fluency of the preview picture is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A schematic diagram of a dual-camera image frame in a binocular portrait preview blurring in the related art;

[0019] Figure 2 A flowchart of an optional image preview method provided by an embodiment of the present application;

[0020] Figure 3 A flowchart of an example one of an optional image preview method provided by an embodiment of the present application;

[0021] Figure 4A flowchart of an example two of an optional image preview method provided for an embodiment of the present application is shown in FIG. 2;

[0022] Figure 5 A flowchart of an example three of an optional image preview method provided for an embodiment of the present application is shown in FIG. 3;

[0023] Figure 6 A flowchart of an example four of an optional image preview method provided for an embodiment of the present application is shown in FIG. 4;

[0024] Figure 7 A flowchart of an example five of an optional image preview method provided for an embodiment of the present application is shown in FIG. 5;

[0025] Figure 8 A structural diagram of an optional image preview device provided for an embodiment of the present application is shown in FIG. 6;

[0026] Figure 9 A structural diagram of an optional electronic device provided for an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0028] Currently, the mainstream technical solution for realizing real-time blur effect of portrait preview on a mobile terminal mainly relies on real-time calculation and synchronous rendering process of a depth map, and the core steps thereof are usually as follows:

[0029] Step 1: image acquisition;

[0030] Among them, the multi-camera system (usually a main camera + a dedicated depth camera, for example, a long-focus, ultra-wide-angle or Time-of-Flight (ToF) sensor) of the device or a single camera cooperates with a specific algorithm to synchronously capture image data (RGB data) of the current frame. In the single camera solution, it can also be necessary to capture multiple frames or utilize auxiliary information of the sensor.

[0031] Step 2: real-time depth map calculation;

[0032] Among them, the calculation can be performed in any of the following ways:

[0033] (1) Dual / multi-camera parallax method: The mainstream method, which uses two or more cameras with different spatial positions (e.g., main camera and auxiliary camera) to capture images simultaneously. By calculating the pixel position difference (parallax) of the same scene point in two images, combining camera calibration parameters (internal and external parameters), and using stereo matching algorithms, the depth map of the scene is calculated in real time. In recent years, learning-based stereo matching algorithms (e.g., Convolutional Neural Network, CNN) have been widely used due to their stronger robustness.

[0034] (2) ToF sensor-based: Some high-end devices are equipped with dedicated ToF sensors. These sensors actively emit modulated infrared light pulses and measure the time difference (or phase difference) of their reflection, directly calculating the depth information of each point in the scene. ToF sensors can provide relatively accurate depth data, but have high cost, high power consumption, and may be limited in strong light or specific reflective material surfaces.

[0035] (3) Single-camera machine learning model: Using a trained depth estimation neural network model (e.g., Monodepth and its variants), a single RGB image can be used to predict the depth map of the scene. This method has low hardware requirements (single camera), but the accuracy of the depth map and the accuracy of edge segmentation are usually lower than multi-camera solutions, and the model inference itself also consumes a large amount of computing resources.

[0036] (4) Fusion solution: Combining various sensor data (e.g., RGB+ToF, RGB+multi-camera parallax) and algorithms, a more robust depth map is generated through sensor fusion technology.

[0037] Step 3: Depth map post-processing;

[0038] In which, the necessary post-processing of the calculated original depth map is performed, such as filtering and noise reduction, hole filling, edge refinement (especially for hair, transparent object edges, etc.), and semantic segmentation (to ensure that the portrait subject is accurately segmented), to improve the quality and usability of the depth map.

[0039] Step 4: Real-time blur rendering;

[0040] In which, based on the processed depth map and the current frame of RGB image, a blur algorithm (usually based on Gaussian blur or more complex lens blur model) is applied. The blur algorithm performs different degrees of blur processing on background pixels according to the depth value (or corresponding blur radius) of each pixel in the depth map, while keeping the foreground portrait clear or performing slight transition processing. This step usually involves a large amount of pixel-level calculation.

[0041] Step 5: Synthesis and preview.

[0042] In this step, the blurred background is synthesized with the clear foreground (or the foreground after appropriate processing) to form the final image, which is displayed in real time on the device screen for user preview.

[0043] It can be seen that the core feature of the related art process is the strong synchronization of depth map calculation and rendering. In order to display the blur effect corresponding to the current frame during preview, the system must complete steps S102-S105 within the processing period of the current frame (e.g., 33ms@30fps). As one of the most time-consuming links in the entire process, depth map calculation (S102) must be completed in time for each frame, otherwise it will block the subsequent rendering and display.

[0044] Among them, although the RTB technology can achieve certain effects, it has significant shortcomings in high frame rate, high resolution, complex scenes and resource-limited mobile devices:

[0045] 1. High foreground processing pressure and easy to freeze: As a computationally intensive operation, depth map calculation is time-consuming and easy to exceed the frame budget time in complex scenes, high-resolution input or high-precision depth requirements, resulting in delayed subsequent rendering and display, resulting in frame drop, freezing or delay. Depth calculation requires a large amount of Central Processing Unit (CPU) and / or Graphics Processing Unit (GPU) resources, which competes with foreground rendering for resources, increasing processing pressure.

[0046] 2. Low and uneven utilization of system resources: The depth calculation requirement varies greatly between different frames, with idle computing power in simple scenes and insufficient computing power in complex scenes, resulting in uneven resource distribution. The synchronization model in the related art requires independent depth map calculation for each frame, which cannot share tasks across frames and passively withstands single-frame calculation pressure fluctuations.

[0047] 3. Lack of dynamic adaptability: The related art responds slowly to the dynamic running environment of mobile devices, and mostly uses fixed strategies, lacking the ability to adjust according to real-time system state. When resources are insufficient, either force calculation to freeze and crash, or simply turn off the function, lacking a mechanism for intelligent adjustment under the premise of ensuring smoothness.

[0048] 4. Tight coupling of depth calculation and rendering: Depth map calculation is strongly bound to the current frame rendering, and the result only serves the corresponding frame, making reuse and utilization inflexible.

[0049] In the portrait preview blurring scene based on the dual-camera system, ideally, when the device computing power and power consumption are redundant, depth map calculation and synchronous rendering are performed in real time for each set of synchronized master and slave frames to ensure that each frame has optimal blurring effect (for example, 30fps full frame rate processing). Under the reality constraints, limited by the three core bottlenecks of mobile terminal:

[0050] 1) Computing power limitation: high-precision stereo matching / neural network inference is difficult to complete within 33ms (@30fps);

[0051] 2) Power consumption constraint: continuous full frame rate calculation leads to temperature control frequency reduction (CPU reduction of 35% when the measured XX mobile phone is >40℃);

[0052] 3) System contention: camera data stream, rendering pipeline, and other applications share limited System on Chip (SOC) resources.

[0053] Therefore, under the premise of ensuring visual fluency, depth map processing is performed once every a frame (the a value may be dynamically adjusted). Figure 1 For the schematic diagram of the dual-camera image frame in the related art when the dual-camera portrait preview blurring is performed, as shown in Figure 1 , a=3, that is, depth map calculation is performed once every 3, Master represents the main camera, and Slave represents the slave camera.

[0054] In summary, the core pain point of the related art is the series of problems caused by the synchronous execution mode of depth map calculation, which is significantly limited in pursuing extreme experience and dealing with complex challenges, and new technologies are needed to decouple depth calculation and rendering, optimize resource scheduling, and improve system capability to achieve smooth, stable, and high-quality portrait preview real-time blurring on mobile platforms.

[0055] In view of the technical problem that the preview picture in the related art has stuttering, the embodiment of the present application provides an image preview method, Figure 2 The flowchart of an optional image preview method provided by the embodiment of the present application is shown in Figure 2 , which can include the following steps:

[0056] S201: acquiring a current frame;

[0057] The image preview method provided by the embodiment of the present application is applied to an electronic device with a display screen, for example, the electronic device can be a smart phone, a tablet computer, etc.

[0058] The electronic device can acquire the current frame through a camera, and the camera can be a monocular camera or a binocular camera, which is not limited in the embodiment of the present application.

[0059] If a monocular camera is used to capture the current frame, one frame is obtained. If a binocular camera is used to capture the current frame, two frames are obtained. It should be noted that, in addition to obtaining the current frame through the camera, the current frame can also be an image frame in a video downloaded from another electronic device. Herein, embodiments of the present application do not make a specific limitation thereto.

[0060] In addition, the image preview method provided by the embodiments of the present application can be applied to portrait preview blurring, and can also be applied to other image processing algorithms. Herein, embodiments of the present application do not make a specific limitation thereto.

[0061] In addition, it should be noted that, after the current frame is obtained, the period of the current frame is entered.

[0062] S202: In a case where the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, the processing parameter map of the previous frame is continuously determined in the period of the current frame, to obtain the processing parameter map of the previous frame.

[0063] After the current frame is obtained by the electronic device in S201, in S202, the electronic device needs to first determine whether the processing parameter map of the previous frame has been determined in the period of the previous frame of the current frame. If yes, it indicates that the calculation of the processing parameter map has been completed in the period of the previous frame, and then the current frame is processed according to the processing parameter map of the previous frame in the period of the current frame, to obtain the preview picture of the current frame.

[0064] If no, it indicates that the processing parameter map of the previous frame has not been determined in the period of the previous frame, and generally, the intermediate step of determining the processing parameter map of the previous frame is still being performed. At this time, since the current frame needs to use the processing parameter map of the previous frame, the processing parameter map of the previous frame is continuously determined based on the intermediate step in the period of the previous frame when the period of the current frame is entered, until the processing parameter map of the previous frame is obtained.

[0065] The processing parameter map can be a depth map, or an image parameter map. The image parameter can be a brightness parameter, a color temperature parameter, and the like. Herein, embodiments of the present application do not make a specific limitation thereto.

[0066] In this way, the processing parameter map of the previous frame is determined across the period of the previous frame and the period of the current frame, so that the calculation power of the processing parameter map can be shared between the two frames, and the preview picture can be prevented from being stuck due to the overload of the calculation power of a single frame.

[0067] S203: The current frame is processed according to the processing parameter map of the previous frame, to obtain the preview picture of the current frame.

[0068] After the processing parameter map of the previous frame is determined in the period of the previous frame through S202, the electronic device can process the current frame according to the processing parameter map of the previous frame to obtain the preview picture of the current frame in S203.

[0069] It should be noted that, in addition to processing the current frame by using the processing parameter map of the previous frame, other processing can also be performed on the current frame. For example, after processing the current frame by using the depth map of the previous frame in the RTB technology, post-processing of the depth map, real-time virtualization rendering processing, and synthesis and preview processing need to be performed on the current frame processed by the depth map, so as to obtain the preview picture of the current frame.

[0070] Further, in order to quickly obtain the processing parameter map of the previous frame and quickly obtain the preview picture of the current frame, in an optional embodiment, S202 can include:

[0071] In the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, the process parameters of the subsequent determination of the processing parameter map of the previous frame are adjusted in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame.

[0072] Here, in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame, in order to obtain the processing parameter map of the previous frame in the period of the current frame while ensuring the smoothness of the preview picture, the process parameters of the subsequent determination of the processing parameter map of the previous frame are adjusted in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame.

[0073] The process parameters of the subsequent determination of the processing parameter map of the previous frame can be the priority of the thread of the subsequent determination of the processing parameter map of the previous frame, or the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame. Here, the embodiments of the present application do not make specific limitations.

[0074] In this way, by adjusting the process parameters of the subsequent determination of the processing parameter map of the previous frame in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame, the processing of the current frame is improved, the preview picture of the current frame can be obtained in time, and the smoothness of the preview picture is ensured.

[0075] Further, in order to adjust the process parameters of the subsequent determination of the processing parameter map of the previous frame, in an optional embodiment, in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, the process parameters of the subsequent determination of the processing parameter map of the previous frame are adjusted in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame, which can include:

[0076] in a case where the processing parameter map of the previous frame is not determined in a period of the previous frame of the current frame, the priority of the thread for the subsequent determination of the processing parameter map of the previous frame is raised in a period of the current frame;

[0077] The thread for the subsequent determination of the processing parameter map of the previous frame is executed based on the raised priority in the period of the current frame, so as to accelerate the determination of the processing parameter map of the previous frame.

[0078] It can be understood that in a case where the process parameter is the priority of the thread, when the processing parameter map of the previous frame is not determined in the period of the previous frame, the priority of the thread for the subsequent determination of the processing parameter map of the previous frame can be raised in the period of the current frame.

[0079] The thread for the subsequent determination of the processing parameter map of the previous frame is executed based on the raised priority in the period of the current frame, so as to accelerate the determination of the processing parameter map of the previous frame.

[0080] In the period of the current frame, the thread for the subsequent determination of the processing parameter map of the previous frame can be raised to the highest priority in the preset priority level, and the thread for the subsequent determination of the processing parameter map of the previous frame is processed based on the highest priority.

[0081] In this way, by raising the thread for the subsequent determination of the processing parameter map of the previous frame in the period of the current frame, the subsequent determination of the processing parameter map of the previous frame can be processed preferentially, so that the processing parameter map of the previous frame can be quickly obtained, the processing of the current frame is accelerated, and the preview picture of the current frame is obtained in time, thereby ensuring the smoothness of the preview picture.

[0082] In addition, in order to adjust the process parameter of the subsequent determination of the processing parameter map of the previous frame, in an optional embodiment, in a case where the processing parameter map of the previous frame is not determined in a period of the previous frame of the current frame, the process parameter of the subsequent determination of the processing parameter map of the previous frame is adjusted in a period of the current frame, so as to accelerate the determination of the processing parameter map of the previous frame, which can include:

[0083] In a case where the processing parameter map of the previous frame is not determined in a period of the previous frame of the current frame, the parameters and operators of the model used for the subsequent determination of the processing parameter map of the previous frame are adjusted in a period of the current frame.

[0084] The processing parameter map of the previous frame is subsequently determined based on the adjusted parameters and operators of the model in the period of the current frame, so as to accelerate the determination of the processing parameter map of the previous frame.

[0085] Understandably, in the case of process parameters being parameters and operators of the used model, when the processing parameter map of the previous frame is not determined in the period of the previous frame, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame can be adjusted in the period of the current frame.

[0086] Among them, the parameters and operators of different levels of the model are pre-set in the electronic device, for example, the higher the level, the more computing power required when the parameters and operators are used, therefore, the level of the parameters and operators of the model can be reduced here, thereby reducing the computational complexity in the subsequent determination and the computing power in the subsequent determination.

[0087] Then, based on the adjusted parameters and operators of the model, the processing parameter map of the previous frame is determined in the period of the current frame, which reduces the computational complexity in the subsequent determination and the computing power in the subsequent determination, thereby being able to quickly determine the processing parameter map of the previous frame.

[0088] Among them, by adjusting the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame in the period of the current frame, the level of the parameters and operators can be reduced. For example, to the next level of parameters and operators, and the level of the priority to which it is reduced can also be determined according to the current system load and / or whether the current frame has a risk of stuttering. Here, the embodiments of the present application do not make specific limitations.

[0089] In this way, by adjusting the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame, the computational complexity in the subsequent determination can be reduced, thereby quickly obtaining the processing parameter map of the previous frame to speed up the processing of the current frame, and the preview picture of the current frame can be obtained in time to ensure the smoothness of the preview picture.

[0090] Further, in order to ensure the smoothness of the preview picture of the current frame, in an optional embodiment, in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame in the period of the current frame can include:

[0091] In the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame and the current frame reaches a preset stuttering condition, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame in the period of the current frame are adjusted.

[0092] It can be understood that, in addition to adjusting the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame when the processing parameter map of the previous frame is not determined in the period of the previous frame, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame can also be adjusted in the period of the previous frame when the processing parameter map of the previous frame is not determined and the current frame reaches the preset stuttering condition, that is, in the case that the previous frame does not determine the processing parameter map of the previous frame, and the current frame still has the risk of stuttering, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame are adjusted in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame for the processing of the current frame.

[0093] It can be seen that, only when the processing parameter map of the previous frame is determined in the period of the current frame, if the current frame has the risk of stuttering, it means that the system load is too high or the resources allocated by the electronic device for the processing of the current frame are not enough, in order to avoid the current frame from stuttering, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame can be simplified, so as to reduce the calculation complexity of the subsequent determination of the processing parameter map of the previous frame, to ensure that the preview picture of the current frame can be determined in time.

[0094] In this way, by reducing the calculation complexity of the subsequent determination of the processing parameter map of the previous frame in the determination of the processing parameter map of the previous frame in the period of the current frame, it is ensured that the preview picture of the current frame can be determined in time in the period of the current frame, and the smoothness of the preview picture is ensured.

[0095] For the previous frame, in order to ensure that the preview picture of the previous frame can be determined in time, in an optional embodiment, the above method can further include:

[0096] In the case that the processing parameter map of the previous frame is determined in the period of the previous frame of the current frame and the previous frame reaches the preset stuttering condition, the priority of the thread for determining the processing parameter map of the previous frame is reduced in the period of the previous frame;

[0097] The thread for determining the processing parameter map of the previous frame is executed based on the reduced priority in the period of the previous frame.

[0098] For the previous frame, not only the processing parameter map calculated in the previous foreground is used to determine the preview picture of the previous frame, but also the previous frame is used in the background to determine the processing parameter map of the previous frame, then, if the previous frame reaches the preset stuttering condition in the period of the previous frame, it means that the preview picture of the previous frame has the risk of stuttering, here, in order to avoid the stuttering of the preview picture of the previous frame, the priority of the thread for determining the processing parameter map of the previous frame is reduced in the period of the previous frame.

[0099] In the period of the previous frame, the thread of determining the processing parameter map of the previous frame is executed again based on the lowered priority, so that the idle system resources are released to be allocated to the thread of generating the preview picture of the previous frame.

[0100] In the period of the previous frame, the thread of determining the processing parameter map of the previous frame is executed again based on the lowered priority, so that the idle system resources are released to be allocated to the thread of generating the preview picture of the previous frame.

[0101] In this way, by lowering the priority of the thread of determining the processing parameter map of the previous frame in the period of the previous frame, the risk of stuttering of the preview picture of the previous frame can be prevented, and the preview picture of the previous frame can be determined in time, thereby ensuring the smoothness of the preview picture.

[0102] Further, in order to restore the priority of the thread of determining the processing parameter map of the previous frame in time in the period of the previous frame, in an optional embodiment, the processing parameter map is a depth map, and the method can further include:

[0103] After the rendering of the previous frame is completed in the period of the previous frame, the priority of the thread of determining the processing parameter map of the previous frame is restored.

[0104] For RTB, the processing parameter map can be a depth map. Since more system resources are required for rendering processing in RTB, the priority of the thread of determining the processing parameter map of the previous frame can be restored after the rendering of the previous frame is completed in the period of the previous frame.

[0105] In this way, by restoring the priority after rendering, the system resources released by lowering the priority can be used for the rendering processing of the previous frame, thereby speeding up the determination of the preview picture of the previous frame and ensuring the smoothness of the preview picture.

[0106] In the case where it is predicted that the previous frame has a risk of stuttering, in an optional embodiment, the method can further include:

[0107] In the case where it is predicted that the previous frame has a risk of stuttering, in an optional embodiment, the method can further include:

[0108] That is, before the previous frame is acquired, it can be predicted whether the previous frame reaches the preset stalling condition. If it is determined that the previous frame reaches the preset stalling condition through the prediction, the processing of the previous frame can be allocated with resources in advance, and / or the frequency of the NPU in the electronic device can be improved. In this way, more system resources can be provided for determining the preview picture of the previous frame by using the allocated resources and / or the improved frequency.

[0109] Therefore, more system resources are reserved in advance for the processing of the previous frame before the period of the previous frame arrives, so that the preview picture of the previous frame can be quickly determined in the processing of the previous frame, and the fluency of the preview picture is ensured.

[0110] For the case that the previous frame reaches the preset stalling condition, in an optional embodiment, the processing parameter map is a depth map, and the method can further include:

[0111] In a case that the average of the rendering processing time of the previous N frames of the previous frame exceeds a preset time threshold, it is determined that the previous frame reaches the preset stalling condition.

[0112] and / or,

[0113] In a case that the load rate of system resources in the period of the previous frame exceeds a preset load rate, it is determined that the previous frame reaches the preset stalling condition.

[0114] It can be understood that if it is determined through the prediction that the previous frame reaches the preset stalling condition, it can be determined whether the average of the rendering processing time of the previous N frames of the previous frame exceeds a preset time threshold. If it exceeds, it indicates that the rendering time of the previous N frames of the previous frame is too long, and the previous frame may also have the risk of too long rendering time, so it is determined that the previous frame reaches the preset stalling condition.

[0115] In the above, N is a positive integer greater than or equal to 2.

[0116] In addition, the load rate of the current system resources in the period of the previous frame is detected. If the load rate exceeds a preset load rate, it indicates that the system is overloaded at this time, so there is a risk that the determination of the preview picture of the previous frame cannot be determined in time to cause the preview picture to stall, so it is determined that the previous frame reaches the preset stalling condition.

[0117] It should be noted that for the current frame to reach the preset stall condition, the above similar manner can be used as the last frame. For example, in the case where the average of the time length of the rendering processing of the first N frames of the current frame exceeds the preset time threshold, it is determined that the current frame reaches the preset stall condition; and / or, in the case where the load rate of the system resources within the period of the current frame exceeds the preset load rate, it is determined that the current frame reaches the preset stall condition.

[0118] In this way, by the above manner, it can be predicted whether the last frame reaches the preset stall condition, or the last frame can be determined in real time whether it reaches the preset stall condition, so as to adjust the determination of the preview picture of the last frame, and determine the preview picture of the last frame in time, and ensure the smoothness of the preview picture.

[0119] For the processing of the last frame described above, in an optional embodiment, S203 can include:

[0120] In the case where the processing parameter graph of the last frame is determined in succession within the preset time period, the current frame is processed according to the processing parameter graph of the last frame to obtain the preview picture of the current frame.

[0121] In the period of the current frame, the processing parameter graph of the last frame is obtained within the preset time period, and then the current frame is processed according to the processing parameter graph of the last frame to obtain the preview picture of the current frame.

[0122] The preset time period is a time period within a preset time length with the acquisition time of the current frame as the starting time; and the preset time length is the product of the preset frame interval time length and the preset proportion coefficient. That is, after the current frame is acquired, it can be judged whether the processing parameter graph of the last frame is determined in succession within the product of the preset frame interval time length and the preset proportion coefficient, wherein the preset proportion coefficient can be one half or one third.

[0123] Through the above judgment, the processing parameter graph of the last frame is determined in succession within the preset time period, and then the processing parameter graph of the last frame is used to process the current frame, and the preview picture of the current frame is determined in time.

[0124] Through the above judgment, the processing parameter graph of the last frame is not determined in succession within the preset time period, and then the processing parameter graph of the last frame is used to process the current frame, and the preview picture of the current frame is determined in time.

[0125] Therefore, the processing parameter map of the previous frame is determined in time through the above-mentioned processing parameter map of the previous frame determined successively in the preset time period, and the preview picture of the current frame is obtained in time, thereby ensuring the fluency of the preview picture.

[0126] To further improve the fluency of the preview picture, in an optional embodiment, the above-mentioned method can further include:

[0127] In a case where the current frame is a frame obtained by the camera application of the electronic device in a preset static scene, the processing parameter map used by the previous frame is obtained in a period of the current frame, and the current frame is processed according to the processing parameter map used by the previous frame to obtain the preview picture of the current frame.

[0128] Here, the current frame can be judged to determine whether the current frame is a frame obtained by the camera application of the electronic device in a preset static scene, wherein it can be first judged whether the camera application of the electronic device is in a preset static scene.

[0129] In the judgment of the camera application, the pixels of the current frame and the previous frame can be compared, for example, if the average value of the pixel change value is lower than the change threshold, it is determined that the camera application is in a preset static scene, otherwise it is in a preset dynamic scene; the data of the gyroscope or the acceleration sensor can also be collected to determine whether the moving distance of the electronic device exceeds the distance threshold, if yes, it is determined that the camera application is in a preset static scene, otherwise it is in a preset dynamic scene.

[0130] When it is determined that the camera application is in a preset static scene, it means that the current frame and the previous frame change little, so when the electronic device obtains the current frame into the current frame period, the processing parameter map used by the previous frame is obtained, thereby processing the current frame according to the processing parameter map used by the previous frame to obtain the preview picture of the current frame.

[0131] It should be noted that the number of times of reusing the processing parameter map used by the previous frame is limited, for example, when the reuse number of the processing parameter map reaches a preset number, even if the camera application is in a preset static scene, the processing parameter map cannot be reused, and a new processing parameter map needs to be determined for the current frame.

[0132] Therefore, the reuse of the processing parameter map used by the previous frame in the preset static scene of the camera application can reduce the resources used for processing the current frame, speed up the determination of the preview picture of the current frame, and ensure the fluency of the preview picture.

[0133] For the case where the camera application is in a preset dynamic scene, in an optional embodiment, the above-mentioned method can further include:

[0134] In a case where the camera application of the electronic device is in a preset dynamic scene, a current frame is predicted to obtain a predicted frame;

[0135] A processing parameter diagram of the predicted frame is determined based on the allocated resources and / or the frequency of the NPU after being improved.

[0136] A processing parameter diagram of the predicted frame is determined based on the allocated resources and / or the frequency of the NPU after being improved.

[0137] After the current frame is obtained, the current frame is processed according to the processing parameter diagram of the predicted frame to determine a preview picture of the current frame.

[0138] It can be understood that, in a case where the camera application is in a preset dynamic scene, the current frame can be predicted to obtain a predicted frame before the current frame is obtained. Since the change of the processing parameter diagram in the dynamic scene is large, resources can be allocated in advance and / or the frequency of the NPU can be improved in advance for processing of the processing parameter diagram of the predicted frame, so that the processing parameter diagram of the predicted frame is determined based on the allocated resources and / or the frequency of the NPU after being improved before entering the period of the current frame. Then, after the current frame is obtained, the current frame is processed according to the processing parameter diagram of the predicted frame to obtain a preview picture of the current frame.

[0139] In this way, the processing parameter diagram of the predicted frame is determined in advance for the processing of the current frame by means of early prediction and early scheduling of resources, so that the preview picture of the current frame can be determined faster, and the smoothness of the preview picture is improved.

[0140] The image preview method described in one or more embodiments is described below by way of example.

[0141] This example focuses on the technical defects such as performance lag faced by the portrait blur preview scheme based on double-camera parallax in related technologies, and designs a portrait preview real-time blur scheme based on depth map background frame calculation, the main idea is:

[0142] (1) Asynchronous depth map calculation and historical data reuse rendering: the current frame depth map calculation background is executed asynchronously, and the front-end rendering reuses the depth map of the previous frame, which significantly reduces the front-end processing time; the background calculation task can be distributed across frames to achieve more stable performance overhead and resource utilization.

[0143] (2) Dynamic depth map scheduling and degradation strategy based on system state awareness: real-time monitoring of system load and frame processing state, dynamic adjustment of background depth map calculation priority and algorithm complexity, and adjustment of front-end rendering process priority to ensure smoothness, eliminate performance lag, and ensure high frame rate stability.

[0144] In this example, the portrait preview blur processing flow is as follows:

[0145] Figure 3 An example of an optional image preview method provided by an embodiment of the present application is shown in the flowchart of Figure 3 In the case of light load, the Nth frame shows a complete portrait preview blur processing flow diagram, and the N+1th frame shows the flow of history depth map data for rendering. For the processing flow completed in the Nth frame, a brief explanation is as follows:

[0146] Stage 1: Prepare primary and secondary images;

[0147] Among them, the current system receives the dual-camera synchronous capture of the primary and secondary camera image data as the input source for subsequent processing.

[0148] Stage 2: Preprocessing and depth information calculation;

[0149] Stage 2.1: Correction alignment;

[0150] In portrait blur, correction alignment is used to eliminate dual-camera disparity error, ensure that the depth map and color map match pixel by pixel, avoid blur edge misplacement, foreground residue or background penetration artifacts, and ensure the authenticity and naturalness of the blur effect.

[0151] Stage 2.2: Stereo depth map calculation;

[0152] The main functions are as follows: disparity calculation (control condition isDoStereoDepth), based on aligned images to perform binocular matching, and generate an initial disparity map.

[0153] Disparity map optimization, through multi-level optimization (such as denoising and edge enhancement) to improve the accuracy of the depth map and ensure accurate distinction between foreground and background.

[0154] Depth map generation, convert disparity information into a gray depth map (Depth Map), and quantify the scene space distance.

[0155] Stage 2.3: Temporal fusion;

[0156] Temporal fusion using front and back frame depth information to eliminate single frame noise and enhance the stability and continuity of the depth map.

[0157] Stage 2.4: Artificial intelligence (AI) optimization (control condition isDoRefine);

[0158] Joint optimization of the current frame primary image data and historical depth map through a neural network model to enhance the main edge details and naturalness of the blur transition.

[0159] Stage 3: Vignette rendering.

[0160] In portrait vignette, the core role of vignette rendering is to simulate the depth of field effect of an optical lens, and to perform progressive blur processing on the background area according to the depth map:

[0161] Spatial level enhancement: dynamically adjust the blur intensity according to the pixel depth value (weak blur for close-up → strong blur for distant view), enhance the stereoscopic sense of the picture.

[0162] Subject visual focus: keep the foreground portrait clear, weaken the background interference, and guide the visual focus.

[0163] Artistic effect generation: simulate aperture shape (such as circular / heart-shaped spot), and render bokeh levels to enhance aesthetic expression.

[0164] Edge transition optimization: feathering processing is performed at the intersection of the portrait and the background to eliminate the harsh feeling of cutting.

[0165] Finally, the shallow depth of field effect of professional photography is realized, and the image quality is improved. The rendering result is output to the preview interface, supporting users to adjust the composition in real time or capture high-quality static images.

[0166] As shown in Figure 3 , in the case of light load, the N+1 frame shows that the history depth map data is reused for rendering. By using the AI optimization module, the current frame image data and the history depth map are jointly optimized by means of a neural network model, the subject edge details and the naturalness of the vignette transition are enhanced, and the result is directly used in the subsequent rendering process.

[0167] In this example, the analysis of the portrait preview processing flow is as follows:

[0168] Figure 4 The flowchart of Example 2 of an optional image preview method provided by the embodiment of the present application is shown in Figure 4 , because the "stereo depth map calculation" is easily affected by the following factors, resulting in running out of time, the processing of the Nth frame may not be completed until the N+1th frame, causing the system to appear stuck, the specific reasons are as follows:

[0169] Heterogeneous computing load and real-time challenge: real-time depth map calculation needs to cooperatively call NPU (execute neural network inference), GPU (accelerate stereo matching) and CPU (post-processing logic), forming a high-density heterogeneous computing load. When multiple module pipelines are working, the computing delay of any hardware unit will directly block the processing link, especially in more complex texture scenes, the power gap is further enlarged, resulting in a sharp increase in the risk of frame processing timeout.

[0170] External environmental disturbances exacerbate system vulnerability:

[0171] The dynamic changes in the mobile terminal operating environment significantly weaken the computing stability, as follows:

[0172] Temperature control and frequency reduction: When the SOC temperature is greater than 42°C, the dynamic voltage and frequency scaling (DVFS) mechanism forcibly reduces the NPU / GPU frequency, and the computing power drops by 40% or more. Multi-task preemption: background applications (such as communication services and navigation) preempt CPU threads, causing deep computing tasks to be suspended for ≥10 ms. Memory bandwidth contention: when high-throughput image transmission occurs, deep data access delay is multiplied. These uncontrollable disturbances cause the deep computing time to fluctuate by ±15 ms, seriously threatening real-time performance.

[0173] Experience degradation chain caused by blocking propagation:

[0174] Deep computing timeout triggers a cascade of failures: timeout frame blocks rendering threads, forcing the display end to repeatedly output historical frames (visual stuttering), while the accumulated frame buffer causes subsequent processing chains to stall. As measured, when the single-frame deep computing delay is >20 ms, the preview frame rate fluctuation rate worsens from ±3 ms to ±12 ms, and the user's perception of stuttering intensity increases by 3 times. More seriously, the traditional timeout skipping strategy can cause deep data discontinuity, resulting in tearing and flickering of the blurred edges in motion scenes, causing secondary experience damage.

[0175] In this example, the deep map calculation is asynchronous and the computing power is allocated:

[0176] Figure 5 An example of an optional image preview method provided by an embodiment of the present application is shown in the flowchart of Example Three of the image preview method as shown in Figure 5 The image preview method can include:

[0177] (1) decoupling the stereo depth calculation and rendering process,

[0178] Step 1: Reuse the historical depth map in the foreground.

[0179] As shown in Figure 5 A dual-track architecture of "foreground reuse of historical depth map + background asynchronous calculation" is designed. The "stereo depth calculation" foreground processing step is cancelled and replaced by directly reusing the most recent valid depth map (usually the N-1 frame depth map) in the cache. The foreground operation only includes processes such as "AI smoothing" and "blurring rendering", reducing the operation of waiting for the generation of a depth map.

[0180] Results: When the "blurring rendering" is completed, the current portrait blurring process can be ended, and the results are passed to the subsequent display process. In addition, actual verification shows that "AI blurring" can ensure the blurring visual effect seen by the human eye.

[0181] Step 2: Background asynchronous computing mechanism (light load);

[0182] In the background, an independent thread pool performs the Nth frame "stereo depth map calculation", which runs in parallel with the foreground rendering, avoids data read-write conflicts through double-buffering mutexes, and stores the calculation results in the depth map cache pool for subsequent frame calls.

[0183] Such a background task is low-priority and preemptible, ensuring that the foreground rendering task can always obtain the required computing power at any time.

[0184] Figure 6 An example of an optional image preview method provided by an embodiment of the present application is shown in the flowchart of Example Four of the Image Preview Method, as shown in Figure 6 The image preview method can include:

[0185] (2) Cross-frame dynamic load sharing:

[0186] Step 3: Background asynchronous computing mechanism (high temperature or heavy load situation);

[0187] The cross-frame dynamic load sharing algorithm realizes space-time smoothing of computing power consumption.

[0188] In the Nth frame, after the foreground handles "blurring rendering", it is finished; "stereo depth map calculation" is executed in the background and may cross to the N+1th frame.

[0189] In the N+1th frame, in order to ensure the blurring effect of the N+1th frame, the N+1th frame foreground processing process will wait for the "stereo depth map calculation" process corresponding to the Nth frame, which is equivalent to cross-frame dynamic load sharing of the computing power of the Nth frame "stereo depth map calculation", realizing space-time smoothing of computing power consumption.

[0190] In this way, the background depth map calculation task is executed in the idle time slice or low load period of the current frame period, and its calculation amount is effectively distributed to the processing period of the subsequent frame, avoiding the peak load caused by single-frame depth map calculation.

[0191] In this example, the dynamic depth map scheduling and degradation strategy based on system state perception:

[0192] The above-mentioned depth map calculation asynchronization and computing power allocation method lays the foundation for performance optimization, reduces the average load and improves the baseline frame rate through asynchronization and computing power allocation. The dynamic depth map scheduling and degradation strategy based on system state awareness in this section focuses on solving the tail latency and instantaneous peak load problem, ensuring that the system remains smooth even in the worst case. It makes up for the shortcomings of the depth map calculation asynchronization and computing power allocation method when facing computing fluctuations and resource contention.

[0193] (1) Dynamic depth map scheduling method based on system state awareness:

[0194] (a) The system selectively turns off or triggers some functions by learning the current running phase.

[0195] Figure 7 An example of an optional image preview method provided by the embodiment of the application is shown in the flowchart of Example Five Figure 7 As shown in the figure, the asynchronous cross-frame scheme relies on an initialized depth data. In this example, in the first frame, the foreground process skips the blur function and directly copies the current preview picture for subsequent display, and schedules "correction alignment" and "stereo depth map calculation" to the background. At the same time, the related "stereo depth map calculation" is set to a "general" priority.

[0196] In the second frame, the depth map result of the first frame needs to be waited for. The "depth map calculation" thread in the background is raised to a higher priority and can preempt CPU resources to speed up its processing. Thus, in the blur process of the second frame, the depth map data of the previous frame can be used, and the user can quickly see the blur effect.

[0197] (b) Dynamic priority adjustment

[0198] When the system detects that the foreground rendering process is at risk of delay (for example, the average T_render of the rendering time of the historical frames exceeds a certain time threshold), or the overall load of the system is too high (for example, the utilization rate of CPU / GPU exceeds a safety threshold), the priority of the background depth map calculation task is actively reduced or suspended, or even the task is completely suspended, and the computing power resources are immediately released to the foreground rendering to ensure that the current frame is completed on time.

[0199] At the same time, after the current "blur rendering" is completed, if the "depth map calculation" is still being executed in the background before the next frame starts (as shown in Figure 6 ), the thread processing priority is raised in advance to speed up the processing result.

[0200] (2) Adaptive computation complexity degradation

[0201] As shown in Figure 6As shown, in the N+1 frame, due to the background "depth map calculation", even when the priority is increased, the system resource is still tight due to the system frequency limiting due to the temperature being too high, which causes performance problems in the N+1 frame process. At this time, the scheduler can dynamically instruct the background depth map calculation module to use a faster, smaller calculation (but the accuracy may be slightly lower) algorithm or parameter configuration (such as reducing the resolution, simplifying the model, and reducing the number of iterations). This ensures that even if the background task is limited in resources, a "usable" depth map can be completed as soon as possible for subsequent frame reuse, avoiding the "blurring rendering" process waiting for too long or even exceeding the frame interval.

[0202] Table 1 is a four-level degradation model for adaptive calculation degradation-depth calculation, and the four-level model is as follows:

[0203] Table 1

[0204] Level Model adjustment L0 Full precision mode (when resources are sufficient) L1 Resolution reduced by 50% (CPU > 75%) L2 Lightweight model replacement (temperature > 42°C) L3 Skip computation (2 consecutive frames timeout)

[0205] (3) Predicting load to balance computing power

[0206] Based on historical data and current scene complexity, the upcoming depth map calculation load is predicted, mainly using NPU / GPU and other hardware resources. Based on historical data, the high load frame (such as the period when NPU and GPU need to be used in the burst shooting and continuous shooting scenarios) is predicted, and the scheduler can reserve more resources or start part of the calculation in advance during the load gap of the shooting frame, and disperse the computing power demand by means of phased frequency increase.

[0207] In this example, in terms of scene adaptation, a scene classification algorithm is introduced, and the strategy is dynamically adjusted according to different scene characteristics. For example, for static scenes, the historical data reuse ratio is increased; and for dynamic scenes, the key area changes are predicted in advance, and the depth map calculation is started in advance, without increasing the system burden, to ensure the real-time and accuracy of the depth information of dynamic pictures and improve the overall rendering effect.

[0208] In summary, the following solutions are provided in this example:

[0209] 1. "Front-end reuse historical depth map + background asynchronous calculation" dual-track architecture: the current frame depth map calculation is executed asynchronously in the background, and the front-end rendering reuses the last frame depth map, which significantly reduces the front-end processing time; the background calculation task can share the computing power across frames, achieving more stable performance overhead and resource utilization.

[0210] 2. Dynamic depth map scheduling and degradation strategy based on system state sensing: real-time monitoring of system load and frame processing state, dynamic adjustment of background depth map calculation priority, algorithm complexity, and prediction of load to balance computing power, to ensure smoothness, eliminate performance lag, and ensure high frame rate stability.

[0211] In the method, the current frame depth map calculation is changed from real-time synchronous execution to asynchronous thread processing in the background through the separation of the depth map calculation and the foreground rendering process, the foreground rendering directly reuses the depth map of the previous frame which has been calculated, the single processing period of the foreground is shortened, the rendering blocking problem caused by the long time consumption of single frame depth calculation is solved, and the foreground processing time is significantly reduced.

[0212] A cross-frame computing power allocation mechanism is established to allow the background depth calculation task to break through the single frame period limit, distribute the depth calculation pressure in a complex scene to multiple continuous frame periods (for example, frame calculation or stage calculation), avoid single frame computing power overload, realize smooth allocation and efficient use of system computing power resources, and reduce the risk of stuttering caused by computing power peaks.

[0213] When the rendering delay or system overload is detected, the priority of the background depth calculation task is immediately reduced / suspended to ensure the exclusivity of the foreground rendering resources and eliminate the frame timeout risk. The priority of the background task is actively raised in the idle window after rendering to accelerate depth calculation using fragmented resources and ensure the timeliness of subsequent frame data.

[0214] Through the priority dynamic reversal mechanism, the tail delay is reduced in the resource contention scene to break through the worst response time bottleneck of the mobile terminal real-time system.

[0215] An adaptive collaborative control system of depth calculation quality and system load is constructed. Multi-level precision elastic degradation: dynamically switch the calculation precision according to the temperature / load (full precision→lightweight model→skip calculation) to ensure the continuous availability of the function in extreme scenarios. Cross-frame load prediction scheduling: based on historical data, predict high-load frames, allocate resources and increase frequency in advance during the idle period to realize the spatiotemporal smoothing of computing power demand.

[0216] As can be seen, through the dual-track architecture to decouple depth calculation and rendering, cross-frame allocation of computing power, and the dynamic scheduling strategy to adapt to the system state, the foreground processing pressure is significantly reduced, the rendering blocking and computing power peak overload are eliminated, and the overall improvement of preview smoothness and high frame rate stability is realized. Relying on asynchronous calculation, cross-frame resource allocation and intelligent degradation mechanism, the system resource utilization efficiency (for example, CPU / GPU load balancing, power consumption control) is optimized, and through dynamic adjustment of algorithm complexity, load balancing prediction, etc., the continuity of virtualization effect and the smoothness of user experience are ensured in complex scenarios and resource-limited environments.

[0217] The embodiment of the present application provides a kind of image preview method, comprising: obtaining current frame, in the period of the last frame of current frame, without determining the processing parameter diagram of the last frame, in the period of current frame, the processing parameter diagram of the last frame is successively determined, the processing parameter diagram of the last frame is obtained, according to the processing parameter diagram of the last frame, current frame is handled, and the preview picture of current frame is obtained;That is to say, in the embodiment of the present application, by successively determining the processing parameter of the last frame in the period of the last frame in the period of current frame, the processing parameter of the last frame is obtained in the period of current frame, and then current frame is handled based on this, to obtain the preview picture of current frame, so, cross town computing power sharing mechanism is established between two frames, avoid single frame computing power overload, realize the smooth distribution and efficient use of system computing power resources, reduce the risk of the card of two frames before and after due to computing power peak, improve the fluency of preview picture.

[0218] Based on the same inventive concept as the foregoing embodiments, the embodiment of the present application provides an image preview device, Figure 8 For the structure of an optional image preview device provided by the embodiment of the present application, as shown in Figure 8 The image preview device comprises: an acquisition module 81, a determination module 82 and a preview module 83; wherein,

[0219] The acquisition module 81 is used for acquiring current frame;

[0220] The determination module 82 is used for successively determining the processing parameter diagram of the last frame in the period of current frame in the case that the processing parameter diagram of the last frame is not determined in the period of the last frame of current frame, to obtain the processing parameter diagram of the last frame.

[0221] The processing module 83 is used for processing current frame according to the processing parameter diagram of the last frame, to obtain the preview picture of current frame.

[0222] In an optional embodiment, the determination module 82 is used for: in the case that the processing parameter diagram of the last frame is not determined in the period of the last frame of current frame, adjusting the process parameter of the successive determination of the processing parameter diagram of the last frame in the period of current frame, to speed up the determination of the processing parameter diagram of the last frame.

[0223] In an optional embodiment, the determining module 82 adjusts the process parameters of the subsequent determination of the processing parameter map of the previous frame in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, including: in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, the priority of the thread of the subsequent determination of the processing parameter map of the previous frame is raised in the period of the current frame; and the thread of the subsequent determination of the processing parameter map of the previous frame is executed based on the raised priority in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame.

[0224] In an optional embodiment, the determining module 82 adjusts the process parameters of the subsequent determination of the processing parameter map of the previous frame in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, including: in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame are adjusted in the period of the current frame; and the processing parameter map of the previous frame is determined based on the adjusted parameters and operators of the model in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame.

[0225] In an optional embodiment, the determining module 82 adjusts the process parameters of the subsequent determination of the processing parameter map of the previous frame in the period of the current frame to accelerate the determination of the processing parameter map of the previous frame in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame, including: in the case that the processing parameter map of the previous frame is not determined in the period of the previous frame of the current frame and the current frame reaches the preset stalling condition, the parameters and operators of the model used in the subsequent determination of the processing parameter map of the previous frame are adjusted in the period of the current frame.

[0226] In an optional embodiment, the apparatus is further configured to: in the case that the processing parameter map of the previous frame is determined in the period of the previous frame of the current frame and the previous frame reaches the preset stalling condition, the priority of the thread of the determination of the processing parameter map of the previous frame is lowered in the period of the previous frame; and the thread of the determination of the processing parameter map of the previous frame is executed based on the lowered priority in the period of the previous frame.

[0227] In an optional embodiment, the processing parameter map is a depth map, and the apparatus is further configured to: after the rendering of the previous frame is completed in the period of the previous frame, the priority of the thread of the determination of the processing parameter map of the previous frame is restored.

[0228] In an optional embodiment, the device is further configured to: in a case where the previous frame reaches the preset stalling condition, allocate resources and / or increase the frequency of the NPU of the electronic device for processing of the processing parameter map of the previous frame to accelerate determination of the preview picture of the previous frame within a period of the previous frame.

[0229] In an optional embodiment, the processing parameter map is a depth map, and the device is further configured to: in a case where the average of the time lengths of the rendering processes of the previous N frames exceeds a preset time threshold, determine that the previous frame reaches the preset stalling condition; and / or in a case where the load rate of system resources within the period of the previous frame exceeds a preset load rate, determine that the previous frame reaches the preset stalling condition.

[0230] In an optional embodiment, the preview module 93 is configured to: in a case where the processing parameter map of the previous frame is continuously determined within a preset time period, process the current frame according to the processing parameter map of the previous frame to obtain the preview picture of the current frame; wherein the preset time period is a time period within a preset time length from a time point of obtaining the current frame; and the preset time length is a product of a preset frame interval time length and a preset proportion coefficient.

[0231] In an optional embodiment, the device is further configured to: in a case where the current frame is a frame obtained by the camera application of the electronic device in a preset static scene, obtain the processing parameter map used for the previous frame within a period of the current frame, and process the current frame according to the processing parameter map used for the previous frame to obtain the preview picture of the current frame.

[0232] In an optional embodiment, the device is further configured to: in a case where the camera application of the electronic device is in a preset dynamic scene, predict the current frame to obtain a predicted frame; allocate resources and / or increase the frequency of the NPU of the electronic device for processing of the processing parameter map of the predicted frame; determine the processing parameter map of the predicted frame based on the allocated resources and / or the increased frequency of the NPU; and after the current frame is obtained, process the current frame according to the processing parameter map of the predicted frame to determine the preview picture of the current frame.

[0233] In actual applications, the obtaining module 81, the determining module 82, and the preview module 83 can be implemented by a processor on the image preview device, specifically, a CPU, a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0234] Figure 9An optional structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 9. As shown in FIG. 9, an embodiment of the present application provides an electronic device 900, which includes: Figure 9

[0235] a processor 91 and a storage medium 92 storing instructions executable by the processor; the storage medium 92 is coupled to the processor 91 through a communication bus 93 to perform operations, and when the instructions are executed by the processor, the image preview method executed by the processor in one or more embodiments described above is performed.

[0236] It should be noted that in actual applications, various components in the computer device are coupled together through the communication bus 93. It can be understood that the communication bus 93 is used to realize the connection and communication between the components. The communication bus 93 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, in order to clearly illustrate, all kinds of buses are marked as the communication bus 93 in the description. Figure 9

[0237] An embodiment of the present application provides a computer storage medium storing executable instructions, and when the executable instructions are executed by one or more processors, the processors execute the image preview method described in one or more embodiments.

[0238] In the embodiment of the present application, the computer readable storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.

[0239] ​​Those skilled in the art will appreciate that embodiments of the application can be further implemented in a computer program product tangibly embodied in a machine-readable storage medium (e.g., memory storage) including instructions that, when executed by a machine (e.g., a processor), cause the machine to perform the steps of embodiments of the application. The terms "machine-readable storage medium" or "computer-readable storage medium" include, but are not limited to, portable or fixed storage devices, optical storage devices, magnetic storage devices, wireline, optical, or other communication links, commonly known as computer communication networks, including the Internet, intranets, local area networks (LANs), wide area networks (WANs), etc. The terms "machine-readable storage medium" or "computer-readable storage medium" also include any medium that is capable of storing or encoding computer readable instructions for execution by a machine (e.g., a processor) and that cause the machine to perform any one or more of the steps that define the procedures described in the detailed description section of the instant disclosure. The terms "machine-readable storage medium" or "computer-readable storage medium" therefore include, but are not limited to, memories (e.g., random access memories (RAMs), electrically programmable read only memories (EPROMs), electrically erasable and programmable read only memories (EEPROMs), etc.), magnetic storage devices (e.g., magnetic tapes and magnetic disks), optical storage devices (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), floppy disks, and other storage devices.

[0240] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0241] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 the functions specified in the flowchart block or blocks.

[0242] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 the functions specified in the flowchart block or blocks.

[0243] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and operation shown.

Claims

1. An image preview method, characterized by, The method comprises: acquiring a current frame; in a case where a processing parameter map of a previous frame of the current frame is not determined within a period of the previous frame, continuously determining the processing parameter map of the previous frame within a period of the current frame to obtain the processing parameter map of the previous frame; processing the current frame according to the processing parameter map of the previous frame to obtain a preview picture of the current frame.

2. The method of claim 1, wherein, The continuously determining the processing parameter map of the previous frame within the period of the current frame in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame comprises: in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame, adjusting a process parameter of the continuously determining the processing parameter map of the previous frame within the period of the current frame to accelerate the determination of the processing parameter map of the previous frame.

3. The method of claim 2, wherein, The adjusting the process parameter of the continuously determining the processing parameter map of the previous frame within the period of the current frame to accelerate the determination of the processing parameter map of the previous frame in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame comprises: in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame, improving a priority of a thread of the continuously determining the processing parameter map of the previous frame within the period of the current frame; executing the thread of the continuously determining the processing parameter map of the previous frame within the period of the current frame based on the improved priority to accelerate the determination of the processing parameter map of the previous frame.

4. The method according to claim 2 or 3, characterized in that, The adjusting the process parameter of the continuously determining the processing parameter map of the previous frame within the period of the current frame to accelerate the determination of the processing parameter map of the previous frame in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame comprises: in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame, adjusting a parameter and an operator of a model used in the continuously determining the processing parameter map of the previous frame within the period of the current frame; continuously determining the processing parameter map of the previous frame within the period of the current frame based on the adjusted parameter and the operator of the model to accelerate the determination of the processing parameter map of the previous frame.

5. The method of claim 4, wherein, The adjusting the parameter and the operator of the model used in the continuously determining the processing parameter map of the previous frame within the period of the current frame in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame comprises: in the case where the processing parameter map of the previous frame is not determined within the period of the previous frame and the current frame reaches a preset stalling condition, adjusting the parameter and the operator of the model used in the continuously determining the processing parameter map of the previous frame within the period of the current frame.

6. The method of claim 1, wherein, The method further comprises: In a case that the processing parameter map of the previous frame is determined in a period of the previous frame and the previous frame reaches a preset stall condition, a priority of a thread for determining the processing parameter map of the previous frame is lowered in the period of the previous frame. The thread for determining the processing parameter map of the previous frame is executed based on the lowered priority in the period of the previous frame.

7. The method of claim 6, wherein, The processing parameter map is a depth map, and the method further comprises: After rendering of the previous frame is completed in the period of the previous frame, the priority of the thread for determining the processing parameter map of the previous frame is restored.

8. The method of claim 1, wherein, The method further comprises: In a case that it is predicted that the previous frame reaches the preset stall condition before the previous frame is acquired, a resource for processing the processing parameter map of the previous frame is allocated and / or a frequency of an NPU of the electronic device is raised to accelerate determination of the preview picture of the previous frame in the period of the previous frame.

9. The method according to any one of claims 6 to 8, characterized in that, The processing parameter map is a depth map, and the method further comprises: In a case that an average of time lengths of rendering processes of the first N frames of the previous frame exceeds a preset time threshold, it is determined that the previous frame reaches the preset stall condition. And / or, In a case that a load rate of system resources in the period of the previous frame exceeds a preset load rate, it is determined that the previous frame reaches the preset stall condition.

10. The method according to any one of claims 1 to 3, characterized in that, The processing of the current frame based on the processing parameter map of the previous frame to obtain the preview picture of the current frame comprises: In a case that the processing parameter map of the previous frame is continuously determined in a preset time period, the processing of the current frame based on the processing parameter map of the previous frame to obtain the preview picture of the current frame; The preset time period is a time period in a preset time length from a time point of acquisition of the current frame; and the preset time length is a product of a preset frame interval time length and a preset proportion coefficient.

11. An image preview device, characterized by, The method comprises: The acquisition module is configured to acquire a current frame. The determination module is configured to, in a case that the processing parameter map of the previous frame is not determined in a period of the previous frame, continuously determine the processing parameter map of the previous frame in a period of the current frame to obtain the processing parameter map of the previous frame. The preview module is configured to process the current frame based on the processing parameter map of the previous frame to obtain a preview picture of the current frame.

12. An electronic device, comprising: The method comprises: The processor and a storage medium having instructions executable by the processor; The storage medium performs operations in dependence on the processor via a communication bus, and when the instructions are executed by the processor, the image preview method in any one of claims 1 to 10 is performed.

13. A computer storage medium, characterized in that The storage medium has executable instructions, and when the executable instructions are executed by one or more processors, the image preview method in any one of claims 1 to 10 is performed.

Images