Image preview method and device, electronic equipment and computer storage medium

Abstract

The embodiment of the invention discloses an image preview method, which comprises the following steps: acquiring a current frame, acquiring a processing parameter diagram of a previous frame of the current frame under the condition that a processing parameter diagram of the current frame is not acquired within a preset time period, and processing the current frame according to the processing parameter diagram of the previous frame to obtain a preview picture of the current frame. The embodiment of the invention also provides an image preview device, electronic equipment and a computer storage medium.

Description

Technical Field

[0001] The present application relates to image processing technologies, and in particular, to an image preview method, apparatus, electronic device, and computer storage medium. Background Art

[0002] Currently, real-time portrait preview algorithms usually need to perform a series of complex computing tasks, including depth estimation, image segmentation, and application of defocus effects, etc. These operations pose relatively high requirements on the processor performance and memory resources of mobile phones. Especially in key links such as "stereo vision" and "depth map generation", the system needs to consume a large amount of computing resources to complete binocular image matching, disparity calculation, and generation and optimization of depth maps.

[0003] However, when the mobile phone is in a high-load state, for example, when a camera event is triggered, the device temperature rises too high due to long-term shooting, or when multiple applications are running in the background at the same time, system resources may be severely preempted, and even the Central Processing Unit (CPU) down-frequency protection mechanism may be triggered. These situations will cause a significant extension in the execution time of the real-time portrait preview bokeh (RTB) algorithm, which may exceed the processing cycle of one frame or even several frames, thus causing problems such as stuttering, delay, or frame rate drop in the preview screen, affecting the picture quality of the preview screen.

[0004] It can be seen that in the related art, there is a technical problem of stuttering in the preview screen when the mobile phone is in a high-load state. Summary of the Invention

[0005] Embodiments of the present application provide an image preview method, apparatus, electronic device, and computer storage medium, which can meet the personal needs of users.

[0006] The technical solution of the present application is implemented as follows:

[0007] In a first aspect, embodiments of the present application provide an image preview method, including:

[0008] Obtain a current frame;

[0009] In the case that a processing parameter map of the current frame is not obtained within a preset time period, obtain a processing parameter map of the previous frame of the current frame;

[0010] Process the current frame according to the processing parameter map of the previous frame to obtain a preview screen of the current frame.

[0011] In a second aspect, embodiments of the present application provide an image preview apparatus, including:

[0012] A first obtaining module, configured to obtain a current frame;

[0013] A second obtaining module, configured to obtain a processing parameter map of the previous frame of the current frame when the processing parameter map of the current frame is not obtained within a preset time period;

[0014] A preview module, configured to process the current frame according to the processing parameter map of the previous frame to obtain a preview image of the current frame.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a storage medium storing processor-executable instructions; the storage medium depends on the processor to execute operations through a communication bus, and when the instructions are executed by the processor, the image preview method described in the above one or more embodiments is executed.

[0016] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing executable instructions, and when the executable instructions are executed by one or more processors, the processor executes the image preview method described in the above one or more embodiments.

[0017] An embodiment of the present application provides an image preview method, apparatus, electronic device and computer storage medium, including: obtaining a current frame, obtaining a processing parameter map of the previous frame of the current frame when the processing parameter map of the current frame is not obtained within a preset time period, and processing the current frame according to the processing parameter map of the previous frame to obtain a preview image of the current frame; that is to say, in the embodiment of the present application, after obtaining the current frame, if the processing parameter map of the current frame is not calculated within the preset time period, the processing parameter map of the previous frame is obtained, and then the current frame is processed by using the processing parameter map of the previous frame, so as to obtain a preview image of the current frame. In this way, the problem of preview image jitter caused by waiting for the processing parameter map of the current frame is avoided, thereby ensuring the smoothness of the preview image. Description of the Drawings

[0018] Figure 1 It is a schematic diagram of the basic process of binocular portrait blurring preview in the related art;

[0019] Figure 2 It is a schematic diagram of a flow of an optional image preview method provided by an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of a flow of an example 1 of an optional image preview method provided by an embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of a flow of an example 2 of an optional image preview method provided by an embodiment of the present application;

[0022] Figure 5Schematic flowchart of the third example of an optional image preview method provided by an embodiment of the present application;

[0023] Figure 6 Schematic flowchart of the fourth example of an optional image preview method provided by an embodiment of the present application;

[0024] Figure 7 Schematic flowchart of the fifth example of an optional image preview method provided by an embodiment of the present application;

[0025] Figure 8 Schematic flowchart of the sixth example of an optional image preview method provided by an embodiment of the present application;

[0026] Figure 9 Schematic structural diagram of an optional image preview device provided by an embodiment of the present application;

[0027] Figure 10 Schematic structural diagram of an optional electronic device provided by an embodiment of the present application. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0029] Portrait blurring is a technology widely used in the field of photography. It mainly highlights the photographed subject by blurring the background or foreground, making the people or objects in the photo more eye-catching. In traditional cameras, portrait blurring usually relies on a large aperture (i.e., a low f value, for example, f / 1.8, f / 2.8) to achieve a shallow depth-of-field effect. A large aperture not only allows more light to enter the lens but also significantly reduces the depth of field, making the areas before and after the focus quickly become blurred, thus creating a visual effect with a clear subject and a blurred background.

[0030] In recent years, mobile cameras have made significant progress in the field of computational photography, especially in portrait photography and blurring effects. Modern smartphones usually come with a dedicated portrait mode, which achieves a high-quality background blurring effect through the collaborative work of software algorithms and hardware (such as a multi-camera system). This technology can effectively highlight the photographed object and reduce background interference, and its effect is similar to that of a photo taken with a large-aperture single-lens reflex camera. Among them, the portrait binocular blurring technology is an advanced technology based on a binocular camera system. It can more accurately obtain depth-of-field information by simulating the stereoscopic vision of the human eye, thus generating a more natural blurring effect and further highlighting the photographed subject (such as a portrait).

[0031] Among them, the two core functions of the camera are preview and taking pictures. The preview function allows users to view the upcoming shot in real time before pressing the shutter and simulates the bokeh effect through the RTB algorithm to help users better compose the picture and adjust shooting parameters. The taking picture function is the process of finally capturing the preview image as a static image, and the user determines the final image by pressing the shutter button. Based on the dual-camera system, the implementation of the preview real-time bokeh algorithm mainly includes the following key technical links:

[0032] 1. Stereo vision: The binocular camera consists of two lenses, simulating the perspective of human eyes, and obtaining images from different angles by simultaneously shooting the same scene. By comparing these two images, the system can accurately calculate the distance and depth information of the object, providing basic data for subsequent bokeh processing.

[0033] 2. Depth map generation: Based on the binocular disparity principle, the device can generate a depth map, which can accurately identify the distance between each object in the scene and the camera, thus effectively distinguishing the foreground from the background and providing accurate spatial information for bokeh processing.

[0034] 3. Precise subject recognition: Using the depth information provided by the depth map, the system can clearly identify the subject (such as a face or a body) in the photo, retain it clearly, and perform intelligent blurring on the background part at the same time, thus ensuring that the subject is more prominent in the picture.

[0035] 4. Natural bokeh transition: Compared with traditional single-lens bokeh methods, binocular bokeh technology can achieve a more natural bokeh transition effect. Especially when there are multiple levels between the foreground and the background, the binocular system can generate a smoother gradient bokeh effect, making the overall picture more realistic and full of layers.

[0036] Through the combination of the above technologies, binocular bokeh technology not only improves the expressiveness of portrait photography but also provides users with a more professional and high-quality shooting experience.

[0037] Figure 1 For the schematic diagram of the basic process of binocular portrait bokeh preview in related technologies, as Figure 1 shown, it mainly includes two modules: the depth map (Depth) calculation module 11 and the bokeh rendering module 12. In the binocular bokeh algorithm, depth map calculation and bokeh rendering are the key steps to achieve background bokeh.

[0038] For the depth map calculation module 11, it can perform: determining the distance from the pixel to the camera, understanding the three-dimensional structure of the scene, and providing basic data for background bokeh. The main processes include the following:

[0039] S101: Obtain the synchronized main and sub-camera image data;

[0040] This step is used to ensure the accuracy and effectiveness of subsequent calibration alignment, parallax calculation and other steps.

[0041] S102: Calibration alignment;

[0042] This step eliminates the parallax error caused by misalignment of the binocular cameras, makes the corresponding pixel points lie on the same horizontal line, and projects the images onto a unified coordinate system.

[0043] S103: Parallax calculation;

[0044] Find the pixel parallax (the horizontal displacement of the corresponding points of the same object in the left and right images) in the aligned images, and construct a parallax map (the gray value represents the parallax, i.e., the depth information).

[0045] S104: Parallax map optimization;

[0046] Among them, noise and mismatches are eliminated, and the parallax map is smoothed by filtering and interpolation to improve the accuracy of the depth information.

[0047] For the virtualization rendering module 12, it can be executed: Blur the image according to the depth map to achieve the depth of field effect of clear foreground and blurred background, and enhance the beauty.

[0048] S105: Convert to depth map;

[0049] Here, the physical distance is converted into a depth map, which provides a basis for subsequent calculation of the blur mask and application of the blur effect.

[0050] S106: Calculate the blur mask map;

[0051] Among them, the blur intensity of each region is determined, and the pixel blur amount is controlled according to the depth value to enhance the three-dimensional sense.

[0052] S107: Blur effect rendering process;

[0053] Here, the original image is blurred according to the blur mask map to highlight the foreground subject, simulate the depth of field, and enhance the artistic effect.

[0054] In order to ensure the basic fluency of the preview screen in high-load scenarios and at the same time take into account the quality of the virtualization effect, it is necessary to design a set of efficient and flexible processing logic strategies. This strategy needs to be able to dynamically adjust the execution priority of the algorithm when the system resources are tense, reasonably allocate computing resources, and perform lightweight processing or degraded operation on the algorithm when necessary to ensure the real-time performance and stability of the preview function.

[0055] In view of the technical problem of stuttering in the preview screen in the related art, the embodiment of the present application provides an image preview method. Figure 2The flowchart of an optional image preview method provided by an embodiment of the present application is shown as follows. Figure 2 As shown, the image preview method may include:

[0056] S201: Obtain the current frame;

[0057] The image preview method provided by an embodiment of the present application is applied to an electronic device with a display screen. For example, the electronic device may be a smart phone, a tablet computer, and so on.

[0058] Among them, the electronic device may obtain the current frame through a camera. Here, the camera may be a monocular camera or a binocular camera. The embodiment of the present application does not make specific limitations on this.

[0059] If the current frame is captured by a monocular camera, then the obtained current frame is one frame. If the current frame is captured by a binocular camera, the obtained current frame is two frames. It should be noted that in addition to obtaining the current frame through the camera, the above current frame may also be an image frame in a video downloaded from other electronic devices. Here, the embodiment of the present application does not make specific limitations on this.

[0060] In addition, the image preview method provided by the embodiment of the present application may be applied to a portrait blurring algorithm or other image processing algorithms. Here, the embodiment of the present application does not make specific limitations on this.

[0061] S202: If the processing parameter map of the current frame is not obtained within a preset time period, obtain the processing parameter map of the previous frame of the current frame;

[0062] After the electronic device obtains the current frame through S201 above, it is necessary to calculate the processing parameter map of the current frame. Among them, for each pixel point in the above processing parameter map, there is a corresponding processing parameter, and this processing parameter is used to process the pixel point to implement the processing of the current frame.

[0063] Among them, the above processing parameter map may be a depth map, a gain map, a brightness mapping map, and so on. Here, the embodiment of the present application does not make specific limitations on this.

[0064] After obtaining the current frame, when calculating the processing parameter map of the current frame, it is necessary to determine whether the processing parameter map of the current frame is obtained within a preset time period. Here, the start time of the preset time period may be a certain moment after obtaining the current frame. It should be noted that with different selections of this moment, the duration of the corresponding preset time period is different.

[0065] Here, after judgment, if the processing parameter map of the current frame is not calculated within the preset time period, it indicates that the load of the CPU of the electronic device is in a heavy load state at this time. Therefore, the electronic device obtains the processing parameter map of the previous frame of the current frame. That is to say, when the calculation of the processing parameter map of the current frame times out, the processing parameter map of the previous frame of the current frame is obtained to replace the processing parameter map of the current frame, so as to avoid the problem of stuttering of the preview screen.

[0066] S203: Process the current frame according to the processing parameter map of the previous frame to obtain the preview screen of the current frame.

[0067] After obtaining the processing parameter map of the previous frame through the above S202, in S203, the electronic device can process the current frame according to the processing parameter map of the previous frame, so as to obtain the preview screen of the current frame for preview display.

[0068] Among them, the preview screen can be directly obtained by processing the current frame with the processing parameter map of the previous frame, or the processing parameter map of the previous frame can be optimized to update the processing parameter map of the previous frame, and then the current frame is processed with the processing parameter map of the previous frame to obtain the preview screen of the current frame. Here, the embodiments of the present application do not make specific limitations on this.

[0069] Regarding the start time of the above preset time period, in an optional embodiment, S202 may include:

[0070] Taking the moment when the current frame is obtained as the start time, in the case that the processing parameter map of the current frame is not obtained within the preset time period, obtain the processing parameter map of the previous frame of the current frame.

[0071] It can be understood that after the current frame is obtained, for different processing parameter maps, different steps are adopted to obtain the processing parameter map. Here, taking the moment when the current frame is obtained as the start time, it is judged whether the processing parameter map of the current frame is obtained within the preset time period. If not, it indicates that the CPU is in a heavy load state and the calculation of the processing parameter map of the current frame has timed out. Therefore, the electronic device obtains the processing parameter map of the previous frame.

[0072] Here, it should be noted that in taking the moment when the current frame is obtained as the start time, the preset time period can be determined based on the execution duration required to calculate the processing parameter map of the current frame after the current frame is obtained under normal CPU load. Among them, the preset time period is greater than the execution duration, and the difference between the preset time period and the execution duration is less than the preset error.

[0073] In this way, by taking the moment when the current frame is obtained as the starting moment of timing and setting a preset time period correspondingly to determine whether it times out, it is possible to timely determine which situations belong to the CPU overload situation, so that the processing parameter map of the previous frame can be obtained in time as the basis for processing the current frame to obtain a preview image, ensuring the smoothness of the preview image.

[0074] In addition, for the starting moment of the above-mentioned preset time period, in an optional embodiment, S202 may include:

[0075] Preprocess the obtained current frame to obtain the preprocessed current frame;

[0076] Taking the start moment of processing the processing parameter map of the preprocessed current frame as the starting moment, in the case that the processing parameter map of the current frame is not obtained within the preset time period, obtain the processing parameter map of the previous frame of the current frame.

[0077] It can be understood that after the current frame is obtained, for different processing parameter maps, different steps are adopted to obtain the processing parameter map. Among them, for the case where the processing parameter map of the current frame is determined in two steps, in addition to taking the moment when the current frame is obtained as the starting moment, the starting moment can also be the moment when the second step starts.

[0078] Here, after the electronic device obtains the current frame, it first preprocesses the current frame to obtain the preprocessed current frame, and then processes the processing parameter map of the preprocessed current frame. In the embodiment of the present application, taking the start moment of processing the processing parameter map of the preprocessed current frame as the starting moment, it is judged whether the processing parameter map of the current frame is obtained within the preset time period. If not, it means that the CPU is in an overloaded state and it is calculated that the processing of the processing parameter map of the current frame has timed out. Therefore, the electronic device obtains the processing parameter map of the previous frame.

[0079] Here, it should be noted that in taking the start moment of processing the processing parameter map of the preprocessed current frame as the starting moment, the preset time period can be determined based on the execution duration required for processing the processing parameter map of the preprocessed current frame under normal CPU load. Among them, the preset time period is greater than the execution duration, and the difference between the preset time period and the execution duration is less than the preset error.

[0080] It should be noted that the starting moment of the preset time period in the embodiment of the present application can also be the start moment of any step before obtaining the processing parameter map of the current frame from the current frame. Here, the embodiment of the present application does not make specific limitations in this regard.

[0081] In this way, by using the start time of processing the processing parameter map of the preprocessed current frame as the starting time of timing, and setting a preset time period correspondingly to determine whether it times out, it is possible to timely determine which situations belong to the CPU overload situation, so that the processing parameter map of the previous frame can be obtained in time as the basis for processing the current frame to obtain a preview screen, ensuring the smoothness of the preview screen.

[0082] For the situation of obtaining the processing parameter map of the current frame within the preset time period, in an alternative embodiment, the above method may further include:

[0083] When the processing parameter map of the current frame is obtained within the preset time period, the current frame is processed according to the processing parameter map of the current frame to obtain a preview screen of the current frame.

[0084] It can be understood that after judgment, if the processing parameter map of the current frame is obtained within the preset time period, it means that the CPU of the electronic device is not in an overloaded state. Therefore, after obtaining the processing parameter map of the current frame, the current frame can be processed according to the processing parameter map of the current frame to obtain a preview screen of the current frame.

[0085] That is to say, in the embodiment of the present application, after obtaining the current frame, the timer is started according to the set starting time. When the timing duration of the timer reaches the preset time period, it is judged whether the processing parameter map of the current frame is obtained. If so, the current frame is processed according to the processing parameter map of the current frame to obtain a preview screen of the current frame. If not, the current frame is processed according to the processing parameter map of the previous frame to obtain a preview screen of the current frame.

[0086] In this way, by whether the processing parameter map of the current frame is obtained within the preset time period, the electronic device can timely obtain the processing parameter map of the current frame under different loads, ensuring the smoothness of the preview screen.

[0087] For the situation where the processing parameter map of the current frame is not obtained within the preset time period, in an alternative embodiment, the above method may further include:

[0088] When the processing parameter map of the current frame is not obtained within the preset time period, continue to determine and store the processing parameter map of the current frame.

[0089] It can be understood that after obtaining the current frame, the current frame needs to be processed to determine the processing parameter map of the current frame. Among them, it is necessary to judge whether the processing parameter map of the current frame is obtained within the preset time period. If not, at this time, the processing parameter map of the previous frame needs to be obtained to process the current frame according to the processing parameter map of the previous frame to obtain a preview screen of the current frame.

[0090] In addition, the background of the electronic device needs to continue to execute the step of determining the processing parameter map of the current frame until the processing parameter map of the current frame is obtained, and the processing parameter map of the current frame is stored so that it can be obtained when needed in the processing of the next frame.

[0091] It should be noted that the starting moment of the above preset time period can be the moment when the current frame is obtained, or the starting moment of processing the processing parameter map for the preprocessed current frame. Here, the embodiments of the present application do not make specific limitations on this.

[0092] Here, after determining the processing parameter map of the current frame, the processing parameter map of the current frame can also be optimized to update the processing parameter map of the current frame. Here, the embodiments of the present application do not make specific limitations on this.

[0093] In this way, when the processing parameter map of the current frame is not obtained within the preset time period, the processing parameter map of the current frame is obtained through the background's continuous execution, so that the processing parameter map of the current frame can be stored locally, facilitating the acquisition of the processing parameter map of the current frame in the processing of the next frame, which helps to ensure the smoothness of the preview screen.

[0094] Regarding obtaining the processing parameter map of the current frame, in an optional embodiment, the above method may further include:

[0095] After obtaining the processing parameter map of the current frame, the processing parameter map of the current frame and the processing parameter map of the previous frame are fused to update the processing parameter map of the current frame.

[0096] It can be understood that after judgment, whether the processing parameter map of the current frame is obtained within the preset time period or outside the preset time period, the obtained processing parameter map of the current frame can be further optimized to update the processing parameter map of the current frame.

[0097] Regarding the optimization of the processing parameter map of the current frame, here, the processing parameter map of the previous frame can be obtained first, and then the processing parameter map of the current frame and the processing parameter map of the previous frame are fused, and the fused result is used as the processing parameter map of the current frame and stored.

[0098] Among them, the above fusion can adopt a preset fusion algorithm or can be fused by using an Artificial Intelligence (AI) model. Here, the embodiments of the present application do not make specific limitations on this.

[0099] Thus, after obtaining the processing parameter map of the current frame, the processing parameter map of the current frame and the processing parameter map of the previous frame are fused to update the processing parameter map of the current frame, so that the processing parameter map of the current frame is further optimized, which helps to improve the picture quality of the preview picture of the current frame.

[0100] For the processing parameter map of the previous frame, in an optional embodiment, the above method may further include:

[0101] Using the current frame to perform AI processing on the processing parameter map of the previous frame to update the processing parameter map of the previous frame.

[0102] It can be understood that for the case of processing the current frame according to the processing parameter map of the previous frame to obtain the preview picture of the current frame, after obtaining the processing parameter map of the previous frame, the electronic device can also perform AI optimization on the processing parameter map of the previous frame using the current frame, so as to obtain the optimized processing parameter map of the previous frame, and use it as the processing parameter map of the previous frame.

[0103] In addition, for the case of processing the current frame according to the processing parameter map of the current frame to obtain the preview picture of the current frame, after obtaining the processing parameter map of the current frame, the electronic device can also perform AI optimization on the obtained processing parameter map of the previous frame using the current frame, so as to obtain the optimized processing parameter map of the previous frame, use it as the processing parameter map of the previous frame, and then fuse the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame.

[0104] Among them, the RGB information of the current frame can be used to enhance the details and suppress the noise of the processing parameter map of the previous frame, so as to obtain the optimized processing parameter map of the previous frame.

[0105] Thus, by performing AI processing on the processing parameter map of the previous frame using the current frame to update the processing parameter map of the previous frame, the accuracy of the processing parameter map of the previous frame is improved, which helps to improve the picture quality of the preview picture of the current frame.

[0106] In order to implement the fusion of the processing parameter map of the current frame and the processing parameter map of the previous frame, in an optional embodiment, fusing the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame may include:

[0107] According to the preset weight value of the current frame and the weight value of the previous frame, perform weighted summation on the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame.

[0108] Understandably, the weight value of the current frame and the weight value of the previous frame are preset in the electronic device. For example, the weight value of the current frame can be 30%, and the weight value of the previous frame can be 70%.

[0109] Of course, the above-mentioned weight value of the current frame and the weight value of the previous frame can be preset in advance, or can be adjusted according to the image parameters of the current frame and the previous frame. Here, the embodiments of the present application do not make specific limitations on this.

[0110] After obtaining the weight value of the current frame and the weight value of the previous frame, according to the weight value of the current frame and the weight value of the previous frame, perform weighted summation on the processing parameter map of the current frame and the processing parameter map of the previous frame, so as to obtain the processed parameter map after weighted summation, and use it as the processing parameter map of the current frame.

[0111] In this way, the processing parameter map of the current frame is updated by means of weighted summation, so that the obtained processing parameter map of the current frame not only considers the processing parameter map of the current frame obtained according to the current frame, but also considers the processing parameter map of the previous frame, that is, the processing parameter map of the previous frame is fused, further improving the accuracy of the processing parameter map of the current frame.

[0112] In order to obtain the preview picture of the current frame, in an optional embodiment, S203 may include:

[0113] Use the current frame to perform AI processing on the processing parameter map of the previous frame to update the processing parameter map of the previous frame;

[0114] Process the current frame according to the processing parameter map of the previous frame to obtain the preview picture of the current frame.

[0115] Understandably, for the case where the processing parameter map of the current frame has not been obtained within the preset time period, after obtaining the processing parameter map of the previous frame, the current frame can be used to perform AI processing on the processing parameter map of the previous frame, so as to update and obtain the parameter map of the previous frame, and then use the updated processing parameter map of the previous frame to process the current frame, so as to obtain the preview picture of the current frame.

[0116] Here, use the RGB information of the current frame to enhance the details and suppress the noise of the processing parameters of the previous frame, so as to obtain the updated processing parameter map of the previous frame.

[0117] In this way, the processing parameter map of the previous frame is updated by performing AI processing on the processing parameter map of the previous frame, so that the obtained processing parameter map of the previous frame not only considers the previous frame, but also considers the image information of the current frame, that is, the image information of the current frame is fused, further improving the accuracy of the processing parameter map of the previous frame.

[0118] For the above-mentioned processing parameter map, in an alternative embodiment, the processing parameter map may be a depth map.

[0119] It can be understood that the above-mentioned processing parameter map may be a depth map. That is to say, in the embodiments of the present application, when the current frame is obtained and the depth map of the current frame is not obtained within a preset time period, the depth map of the previous frame is obtained, and the current frame is processed according to the depth map of the previous frame to obtain the preview picture of the current frame.

[0120] In this way, the current frame can be processed by the determined depth map of the previous frame, so that the preview picture of the current frame is obtained by combining the depth map of the current frame. For example, the preview picture of the current frame with the portrait blurred can be obtained by combining the depth map of the current frame, thereby improving the picture quality of the preview picture.

[0121] When the processing parameter map is a depth map, in an alternative embodiment, S203 may include:

[0122] Generate a blurred mask map of the current frame according to the depth map of the previous frame;

[0123] Blur the current frame according to the blurred mask map of the current frame and the rendering environment of the current frame to obtain the preview picture of the current frame.

[0124] It can be understood that after obtaining the depth map of the previous frame, a blurred mask map of the current frame can be generated first according to the depth map of the previous frame, and then the current frame is blurred according to the blurred mask map of the current frame and the rendering environment of the current frame to obtain the preview picture of the current frame.

[0125] Among them, the above-mentioned rendering environment of the current frame is obtained according to the preprocessed current frame. That is to say, after the current frame is obtained, the current frame is preprocessed to obtain the preprocessed current frame, and then the preprocessed current frame is respectively subjected to depth processing and rendering environment processing, so as to obtain the depth map of the current frame and the rendering environment of the current frame. Since the depth map of the current frame is not obtained within a preset time period, here, a blurred mask map of the current frame can be generated according to the depth map of the previous frame.

[0126] After obtaining the blurred mask map of the current frame and the rendering environment of the current frame, the current frame can be blurred according to these two to obtain the preview picture of the current frame. This preview picture is the blurred preview picture and can be used for portrait blurring processing or for blurring processing of the target object. Here, the embodiments of the present application do not make specific limitations in this regard.

[0127] In this way, the blurred mask graph of the current frame is first generated from the depth graph of the previous frame, and then the current frame is blurred using the blurred mask graph of the current frame and the rendering environment of the current frame to obtain the preview image of the current frame, so that the obtained preview image achieves the blurring effect while ensuring the smoothness of the preview image.

[0128] The following is an example to describe the image preview method described in one or more of the above embodiments.

[0129] Figure 3 FIG. is a schematic flowchart of the first example of an optional image preview method provided by an embodiment of the present application. As Figure 3 shown, General OpenVX is a hardware heterogeneous acceleration framework for computer vision, and its operating mechanism is based on the graph (Virtual Graph) representation method. Users describe and execute vision tasks by constructing and executing graphs. In this example, based on the computational graph, the core functions of the binocular portrait real-time blurring algorithm are split into multiple computational graphs (sub-graphs), and each computational graph (sub-graph) implements a core function (implemented serially or in parallel by 1 to N internal operators). A virtual large graph (as Figure 3 shown) is designed to connect multiple modular functional computational graphs (sub-graphs) of RTB, and control logic is embedded by piling at key nodes where each computational graph (sub-graph) runs with the help of callbacks to flexibly update the operation trajectory of the operator, so as to splice dozens of operator nodes into a functional graph component.

[0130] By disassembling the functions and processes, the operations or processes of related functions are integrated into one computational graph. The execution of an operation is determined by its input and output, and each operation can run on an independent thread or the main thread of the computational graph. In this way, the RTB algorithm function is structured into multiple computational graphs, specifically as follows:

[0131] [g0]CtrlRootGraph: The starting computational graph, including operations such as parameter preprocessing and policy control, responsible for initial parameter processing and control decision-making, starting the subsequent process and being processed by the system framework.

[0132] [g1]RefineWarpGraph: The feature point matrix computational graph, including operations such as optimizing camera parameters and updating the correction matrix, used for parameter preparation before the main and secondary image correction, such as feature point extraction and correction matrix generation.

[0133] [g2]DetectGraph: The motion detection computational graph, including motion detection operations, providing a basis for frame skipping energy saving.

[0134] [g3]CreateTextureGraph: Prepare the GL environment calculation graph, which includes operations for creating GL textures, parallelizing GPU parameters and resources in advance to obtain a rendering environment for image rendering preparation.

[0135] [g4]StereoDepthGraph: Stereo depth calculation graph, which includes operations such as anti-shake compensation, correction, stereo vision, and inverse correction, and is used for disparity map calculation, depth map generation, and blurred mask map calculation.

[0136] [g5]AiRefineGraph: AI-optimized depth calculation graph, which performs AI optimization on the depth map based on the RGB map.

[0137] [g6]AiRefineOnlyCopyGraph: Depth map copy calculation graph, which copies the depth map in the static state to reduce the calculation amount.

[0138] [g7]FusionGraph: Temporal fusion calculation graph, which includes temporal fusion operations. By fusing the current frame depth map with the AI-optimized depth map of the historical frame, it realizes a consistent transition between frames.

[0139] [g8]RenderGraph: Rendering calculation graph, which generates a blurred mask map and includes rendering operations. According to the blurred mask map, it blurs the original image to simulate the depth-of-field effect, highlight the foreground subject, and enhance the artistic expressiveness of the image.

[0140] It should be noted that when the condition mIsDoAction1 is set, the calculation graph g1 will be executed; when the condition mIsDoAction2 is set, the calculation graph g2 will be executed; when the condition mIsDoAction3 is set, the calculation graph g3 will be executed; when the condition mIsDoAction4 is set, the calculation graph g4 will be executed; when the condition mIsDoAction5 is set, the calculation graph g5 will be executed; when the condition mIsDoAction6 is set, the calculation graph g6 will be executed; when the condition mIsDoAction7 is set, the calculation graph CopyRenderYuv will be executed.

[0141] In addition, in Figure 3 , hook1 (graph Start Evt Hook Call back) means listening for specific events through the hook mechanism and automatically executing the preset callback function when the event is triggered. Hook2 (graph Finish Evt HookCall back) means listening for the completion event of a specific task through the hook mechanism and automatically executing the preset callback function when the event is triggered. Node represents an operator.

[0142] Through the above splitting of the computational graph, the RTB algorithm process is structured into multiple computational graphs, and the computational graphs interact through data input and output. This example focuses on systematic design and optimization, and the implementation details of specific operators are not elaborated in this example.

[0143] The following is an analysis of the overloading problem based on the RTB calculation link:

[0144] The basic process of binocular virtualization preview involves the integration and collaborative work of multiple operators, and multiple operators need to be organically combined through a software process. In actual operation, the algorithm depends on various hardware resources, including: CPU, Neural network Processing Unit (NPU), Graphics Processing Unit (GPU), and Digital Signal Processor (DSP), etc. In the design and development process of the algorithm software, the system mainly faces the following challenges during overloading: the operator scheduling takes too long when the CPU is highly loaded in complex camera scenarios, and other algorithms preempt hardware resources when executed concurrently.

[0145] When the system is overloaded, it will affect Figure 3 the longest link in the system operation, that is, when conditions such as mIsDoAction4 and mIsDoAction5 are set, the computational graphs g0, g4, g7, and g8 (with high computing power, long time consumption, and dependence on hardware resources) will be serially executed in sequence. At the same time, the system will also concurrently execute computational graphs such as g1, g2, g2, g5, and g6 (possibly). This will cause the computational graphs of the longest link g0, g4, g7, and g8 to exceed the frame interval, resulting in problems such as system performance lag.

[0146] Figure 4 This is a schematic flowchart of the second example of an optional image preview method provided by the embodiment of the present application. As Figure 4 shown, the image preview method may include:

[0147] S401: In the Node0_2 operator of the g0 computational graph, the system outputs control decision logic. When the mIsDoAction4 flag is set, it indicates that depth calculation is required, and the system will schedule and execute the g4 computational graph (stereo depth computational graph). This step is the core entry of the RTB algorithm, ensuring the start of the depth calculation process.

[0148] S402: At the initial stage of the execution of the g4 computational graph, the system calls the hook1 hook function to initialize relevant resources or perform pre-computation tasks. Assume that in the normal and stable preview state, the execution time of the g4 computational graph is T0. To ensure the real-time performance and stability of the system, a time monitoring threshold T1 (T1 > T0) is set, and a time monitoring mechanism is started to track the execution time of the g4 computational graph in real time. The introduction of the time monitoring mechanism is to prevent frame rate drops or system freezes caused by excessive computational load.

[0149] S403: The time monitoring program detects the execution time and status of the g4 computational graph in real time, and adopts different processing strategies according to the detection results:

[0150] (1) If the running duration of the g4 computational graph does not exceed T1, the system will execute g7 (time fusion computational graph) and g8 (rendering computational graph) in the original process sequence to complete the data calculation and process handling of the current frame. This path ensures that under normal load, the system can efficiently complete depth calculation, time fusion, and rendering tasks.

[0151] (2) If the running duration of the g4 computational graph exceeds T1, the system will trigger a timeout event and enter the callback handler hook3. At this time, the system will directly skip frames and call the g8 computational graph to execute as a foreground task to ensure the real-time performance of rendering. The Depth data required by the g8 computational graph no longer depends on the results generated by the current g4 computational graph, but uses the Depth data of the previous frame or the optimized results of the previous frame's Depth. This design avoids rendering interruptions caused by computational delays by reusing historical data.

[0152] (3) At the same time, the g4 computational graph and the g7 computational graph will be switched to the background to continue execution; the execution of the current g8 computational graph does not depend on their calculation results. This asynchronous processing mechanism ensures that the system can still maintain a smooth rendering effect under high load, while the background tasks can continue to complete depth calculation and time fusion to provide data support for subsequent frames.

[0153] S404: After the g8 computational graph finishes execution, the system will return to the upper-level call process.

[0154] Through the above mechanism, the system freeze problem caused by the overload of the camera system and the excessive computational load of the RTB algorithm is effectively avoided, while ensuring the real-time performance and smoothness of the rendering result. In addition, this design also improves the robustness of the system, which can dynamically adjust the allocation of computational resources under different load conditions to ensure the consistency of the user experience.

[0155] Figure 5 It is a schematic flowchart of the third example of an optional image preview method provided by the embodiment of the present application, as Figure 5As shown, the image preview method may include:

[0156] In Figure 4 During this period, the trigger moment of the timeout monitoring is synchronized with the start moment of the g4 computational graph, mainly used to monitor the execution time of the g4 depth map calculation. However, before this, the g0 computational graph also occupies a certain amount of computing time. Especially in the case of heavy loads, its running duration may fluctuate. To further ensure the stability of the frame rate, the trigger of the time monitoring can start from the start of the g0 computational graph, so as to achieve the timeout monitoring of the entire RTB algorithm link and ensure that the system can still run smoothly under high loads.

[0157] The process is as follows:

[0158] S501: When the RTB algorithm starts to execute, first run the g0 computational graph;

[0159] (1) In the initialization stage of the g0 computational graph, the system calls the hook1 hook function to start the time monitoring mechanism. Assume that in the normal and stable preview state, the total time from the start of the g0 computational graph to the end of the execution of the g4 computational graph is T0.

[0160] (2) To ensure the real-time performance of the system, set the time monitoring threshold T1 (T1 > T0) and start the time monitoring to track the execution time of the entire RTB algorithm link in real time. This improvement extends the monitoring scope from a single g4 computational graph to the entire RTB algorithm process, thus more comprehensively ensuring the system performance.

[0161] S502: In the Node0_2 operator of the g0 computational graph, the system outputs the control decision logic. When the mIsDoAction4 flag is set, it means that depth calculation is required, and the system will schedule and execute the g4 computational graph (stereo depth computational graph). This step is the core entry of the RTB algorithm to ensure the start of the depth calculation process.

[0162] S503: The time monitoring program detects the execution time and status of the entire RTB algorithm link of the current frame in real time, and takes different processing strategies according to the detection results:

[0163] (1) If the running duration of the current frame RTB algorithm does not exceed T1, the system will execute g7 (time fusion computational graph) and g8 (rendering computational graph) in the original process order to complete the data calculation and process handling of the current frame. This path ensures that under normal loads, the system can efficiently complete depth calculation, time fusion, and rendering tasks.

[0164] (2) If the running duration of the RTB algorithm in the current frame exceeds T1, the system will trigger a timeout event and enter the callback handler hook3. At this time, the system will directly skip frames and call the g8 computational graph as the foreground task for execution to ensure the real-time rendering. The Depth data required by the g8 computational graph no longer depends on the results generated by the current g4 computational graph, but uses the Depth data of the previous frame or the optimized results of the previous frame's Depth. This design avoids rendering interruptions caused by calculation delays by reusing historical data.

[0165] (3) At the same time, the g4 computational graph and the g7 computational graph will be switched to the background for continued execution; the execution of the current g8 computational graph does not depend on their calculation results. This asynchronous processing mechanism ensures that the system can still maintain a smooth rendering effect under high load, while the background tasks can continue to complete depth calculation and temporal fusion to provide data support for subsequent frames.

[0166] S504: After the g8 computational graph finishes execution, the system will return to the upper-level call process.

[0167] Through the above mechanism, it effectively avoids system lag problems caused by system overload and excessive computational load in the entire RTB algorithm link, while ensuring the real-time and smoothness of the rendering results.

[0168] Figure 4 and Figure 5 The methods in

[0169] Figure 6 As an optional image preview method provided by the embodiments of this application, for example four of the process schematic diagram, as Figure 6 shown, for the fusion of the current main and secondary parallax calculations with the depth map of the previous frame, it can be called the depth map fusion method 1. For the above method of monitoring the running time of the RTB algorithm, when no timeout event occurs, the system will adopt the depth map fusion method with the best effect (refer to Figure 6 ). This method generates a high-quality depth map by combining the real-time depth data of the current frame and the optimized results of the historical frames, thus providing the best effect for real-time defocus rendering.

[0170] This image preview method may include:

[0171] S601: Through the calculation process of the g4 computational graph (stereo depth computational graph), the system generates a stereo depth map, denoted as Depth-A. Depth-A is the initial depth map calculated based on the main and secondary image parallax of the current frame, with high real-time and accuracy. This step ensures that the depth map can reflect the scene structure of the current frame.

[0172] S602: Using the RGB main image of the current frame, the system performs AI refinement processing on the comprehensive depth map (denoted as Depth-B1) generated in the previous frame. Through an AI optimization algorithm (e.g., AiRefine), combining the RGB information (B2) of the current frame, details are enhanced and noise is suppressed for Depth-B1 to generate a depth map refined by the main image of the current frame, denoted as Depth-B. This step significantly improves the accuracy and visual effect of the depth map, especially in scenes with complex textures or changing lighting.

[0173] S603: Based on Depth-A and Depth-B, the system assigns different weights respectively and performs temporal fusion processing. The temporal fusion algorithm comprehensively considers the real-time nature of Depth-A and the stability of Depth-B to generate the comprehensive depth map of the current frame, denoted as Depth-C. While retaining the depth information of the current frame, Depth-C incorporates the optimization results of historical frames, thus achieving smooth transitions and consistency in dynamic scenes.

[0174] The comprehensive depth map (Depth-C) of the current frame will be provided to the g8 rendering computation graph for real-time defocusing processing of the preview image of the current frame. By combining Depth-C and the blur mask map, the system can accurately simulate the depth-of-field effect, making the foreground clear and the background gradually blurred, thereby enhancing the artistic expressiveness and visual appeal of the image.

[0175] The depth map fusion method 1 generates a high-quality depth map through temporal fusion and AI optimization of multi-frame data, which is applicable to scenarios where no timeout occurs. While ensuring real-time performance, this method maximally improves the quality of the defocusing effect and is suitable for application scenarios with high requirements for visual effects.

[0176] Figure 7 It is a schematic flowchart of Example 5 of an optional image preview method provided by an embodiment of this application. As Figure 7 shown, it is based on AI optimization of the depth maps of the current frame and the previous frame, which can be called the depth map fusion method 2.

[0177] Regarding the above method for monitoring the running time of the RTB algorithm, when a timeout event occurs, the g4 computation graph (stereo depth computation graph) may not have completed the calculation, but the g5 computation graph (AI-optimized depth computation graph) has completed the parallelized operation. At this time, the system will adopt a depth map fusion method with sub-optimal effects (refer to Figure 7 ) to achieve a balance between performance and effect. This method ensures an acceptable defocusing effect can still be provided in case of timeout by reusing historical frame data and AI optimization results.

[0178] This image preview method may include:

[0179] S701: Using the RGB main image of the current frame, the system performs AI refinement processing on the comprehensive depth map (denoted as Depth-B1) generated in the previous frame. Through the AI optimization algorithm (such as AiRefine), combined with the RGB information (B2) of the current frame, details of Depth-B1 are enhanced and noise is suppressed to generate a depth map refined by the main image of the current frame, denoted as Depth-B. This step avoids relying on the real-time depth data of the g4 computational graph in case of timeout, but maintains the quality of the depth map through the optimization of historical frame data.

[0180] S702: Copy or map the data of Depth-B and directly use it as the comprehensive depth map of the current frame, denoted as Depth-C. This operation avoids complex temporal fusion calculations, significantly reduces the computational load, and at the same time ensures the usability of the depth map.

[0181] S703: The comprehensive depth map (Depth-C) of the current frame will be provided to the g8 rendering computational graph for real-time defocusing processing of the preview image of the current frame. By combining Depth-C and the blur mask map, the system can generate a preview image with a certain depth-of-field effect, although its accuracy may be slightly lower than that of depth map fusion method 1.

[0182] When the preview mobile phone is in a stable state, the current depth map and defocusing effect are basically no different from those of depth map fusion method 1, and can provide a high-quality depth-of-field effect.

[0183] When there is a certain movement of the preview mobile phone, since Depth-C does not contain the real-time depth information of the current frame, its defocusing effect will decrease compared with that of depth map fusion method 1. However, considering the performance pressure of the system under heavy load, this decrease in effect is within an acceptable range.

[0184] Depth map fusion method 2 achieves a balance between performance and effect by reusing historical frame data and AI optimization results in case of timeout. Although its defocusing effect may slightly decrease in dynamic scenes, its computational efficiency is significantly improved, making it suitable for use in high-load scenarios to ensure the smoothness and real-time performance of the system.

[0185] Figure 8 It is a schematic flowchart of Example 6 of an optional image preview method provided by an embodiment of this application. As Figure 8 shown, it directly reuses the depth map of the previous frame and can be called depth map fusion method 3.

[0186] Regarding the method for monitoring the running time of the above RTB algorithm, when a timeout event occurs, the g4 computational graph (stereo depth computational graph) and the g5 computational graph (AI-optimized depth computational graph) may both not have completed the calculation (for example, due to waiting for hardware resources or excessive computational load). At this time, the system will adopt the depth map fusion method 3 and directly reuse the depth map data of the previous frame (refer to Figure 8 ), to ensure the real-time performance and smoothness of the system. This method makes a further trade-off between performance and effect and is applicable to extremely high-load scenarios.

[0187] S801: Copy or map the depth map of the previous frame (denoted as Depth-B1) and directly use it as the comprehensive depth map of the current frame, denoted as Depth-C. This operation completely avoids the depth calculation and AI optimization processing of the current frame, significantly reduces the computational load, and at the same time ensures the availability of the depth map.

[0188] S802: The comprehensive depth map (Depth-C) of the current frame will be provided to the g8 rendering computational graph for real-time defocusing processing of the preview image of the current frame. By combining Depth-C and the blur mask map, the system can generate a preview image with a certain depth-of-field effect, although its accuracy may be lower than that of depth map fusion methods 1 and 2.

[0189] When the preview mobile phone is in a stable state, the difference between the current depth map and the defocusing effect and those of depth map fusion method 1 is within a controllable range, and a basically acceptable depth-of-field effect can be provided.

[0190] When the preview mobile phone has a certain movement, since Depth-C completely reuses the depth data of the previous frame and does not contain any real-time information of the current frame, its defocusing effect will decrease compared with depth map fusion methods 1 and 2, and there may be cases of missed defocusing (some areas are not defocused) or incorrect defocusing (some areas are incorrectly defocused).

[0191] In extremely high-load scenarios, depth map fusion method 3 maximally reduces the computational complexity by completely reusing historical frame data and ensures the real-time performance and smoothness of the system. Although its defocusing effect may decrease in dynamic scenarios, due to the low probability of triggering this method under high-load conditions and the fact that performance is the main concern currently, this method is still an acceptable compromise. This method is applicable to scenarios with extremely high requirements for real-time performance but relatively loose requirements for defocusing effects.

[0192] Based on the above dynamic task scheduling and adaptive depth map fusion scheme, the following are the extended technical solutions:

[0193] 1. Hardware resource monitoring and allocation: Real-time monitor the load status of each hardware unit (CPU, GPU, NPU, DSP), including indicators such as utilization rate, temperature, and power consumption. According to the computing requirements of the current frame, dynamically allocate tasks to the most suitable hardware unit. For example, allocate AI optimization tasks (such as AiRefine) to the NPU, rendering tasks to the GPU, and lightweight computing tasks to the DSP.

[0194] 2. Multi-level balancing strategy: In light load scenarios, preferentially use high-performance hardware units (such as GPUs and NPUs) to provide the best picture quality. In heavy load scenarios, migrate some tasks to low-power hardware units (such as DSPs), or enable the energy-saving mode of the hardware unit to reduce the overall power consumption. When detecting hardware resource competition, adopt a task priority scheduling mechanism to ensure that critical tasks (such as virtualized rendering) are executed first.

[0195] 3. Source collaborative optimization: Design a data sharing mechanism between hardware to reduce data copy and transmission overhead. For example, through shared memory or hardware acceleration interfaces, achieve efficient data interaction between the CPU and the GPU. In extremely high load scenarios, enable dynamic frequency scaling or task degradation strategies for hardware resources to ensure system stability.

[0196] This example focuses on solving the problem of preview frame lag in traditional portrait modes under high load scenarios, and proposes a performance optimization strategy based on real-time monitoring of portrait depth map calculations. This solution ensures a smooth preview experience even under heavy loads by dynamically adjusting the allocation of computing resources and the execution priority of algorithms. Its core innovation lies in:

[0197] 1. Dynamic task scheduling: Real-time monitor the execution duration of the current frame's computation graph. If the system is overloaded and times out, jump out of the current processing flow and directly call the virtualized rendering output to ensure a stable frame cycle; at the same time, transfer unfinished tasks such as stereo depth calculation to background asynchronous execution, and apply their calculation results to subsequent frames to ensure the continuity of the image effect.

[0198] 2. Adaptive depth map fusion: Based on system load and effect requirements, design a multi-level depth map fusion strategy. In heavy load scenarios, intelligently select lightweight depth maps for foreground virtualized rendering to achieve the best balance between performance and picture quality.

[0199] It can be seen that in this example, during the operation of the system, the execution duration of the current frame computation graph is monitored in real time. Once system overload causes a timeout, the current processing flow is immediately exited, and the virtualization rendering output is directly called to ensure the stability of the frame cycle, avoid frame freezing and latency, and achieve stable guarantee of the frame cycle through timeout handling. At the same time, tasks such as unfinished stereo depth calculation are transferred to the background for asynchronous execution. After these tasks are calculated in the background, the results will be applied to subsequent frames, thus ensuring the continuity of the image effect and preventing visual discontinuity, and achieving coherent image effects through asynchronous execution of tasks.

[0200] Based on the real-time monitoring of system load and comprehensive consideration of image effect requirements, a multi-level depth map fusion strategy is carefully designed to provide an adaptation solution for the system to operate under different loads. In the overload scenario, due to limited system resources, a lightweight depth map is intelligently selected for foreground virtualization rendering. This depth map has a low data volume and computational complexity, can complete rendering quickly, and combined with an optimized fusion algorithm, takes into account image quality, achieving the best balance between performance and image quality, and realizing depth map selection and balance in the overload scenario.

[0201] In summary, the system dynamically adjusts the task scheduling strategy to cope with different load scenarios by monitoring the execution duration of the current frame computation graph in real time. When the system detects that overload causes the computation graph to execute overtime, it will immediately exit the current processing flow and directly call the virtualization rendering module (for example, the g8 computation graph) for output to ensure the stability of the frame cycle and avoid freezing or frame rate drop caused by computational latency.

[0202] At the same time, the system transfers unfinished stereo depth calculation tasks (such as the g4 computation graph) and other related tasks (such as the g7 computation graph) to the background for asynchronous execution. The calculation results of these background tasks do not affect the output of the current frame, but will be applied to the depth map fusion and rendering of subsequent frames, thus ensuring the continuity and smooth transition of the image effect. This mechanism significantly improves the real-time performance and robustness of the system in high-load scenarios.

[0203] Based on system load and effect requirements, a set of multi-level depth map fusion strategies are designed to intelligently adapt to the performance and image quality requirements in different scenarios. When the system load is light, the depth map fusion method with the best effect (for example, depth map fusion method 1) is adopted, combined with the real-time depth data of the current frame and the optimized results of historical frames, to generate high-quality depth maps and provide the best virtualization effect. In the system overload scenario, it intelligently switches to a lightweight depth map fusion method (for example, depth map fusion method 2 or method 3), reduces the computational load by reusing historical frame data or simplifying the calculation process, and ensures the smooth operation of the system. This adaptive strategy can achieve the best balance between performance and image quality, meeting the real-time requirements in high-load scenarios and providing excellent visual effects in light-load scenarios.

[0204] An embodiment of the present application provides an image preview method, including: obtaining a current frame, and in the case where the processing parameter map of the current frame is not obtained within a preset time period, obtaining the processing parameter map of the previous frame of the current frame, and processing the current frame according to the processing parameter map of the previous frame to obtain a preview picture of the current frame; that is to say, in the embodiment of the present application, after obtaining the current frame, if the processing parameter map of the current frame is not calculated within the preset time period, the processing parameter map of the previous frame is obtained, and then the processing parameter map of the previous frame is used to process the current frame, so as to obtain a preview picture of the current frame. In this way, the problem of preview picture jamming caused by waiting for the processing parameter map of the current frame is avoided, thus ensuring the smoothness of the preview picture.

[0205] Based on the same inventive concept as the foregoing embodiment, an embodiment of the present application provides an image preview device. Figure 9 As a schematic structural diagram of an optional image preview device provided by an embodiment of the present application, as Figure 9 shown, the image preview device includes: a first acquisition module 91, a second acquisition module 92, and a preview module 93; wherein,

[0206] The first acquisition module 91 is configured to acquire a current frame;

[0207] The second acquisition module 92 is configured to acquire the processing parameter map of the previous frame of the current frame in the case where the processing parameter map of the current frame is not obtained within a preset time period;

[0208] The preview module 93 is configured to process the current frame according to the processing parameter map of the previous frame to obtain a preview picture of the current frame.

[0209] In an optional embodiment, the second acquisition module 92 is specifically configured to: take the moment when the current frame is acquired as the starting moment, and acquire the processing parameter map of the previous frame of the current frame in the case where the processing parameter map of the current frame is not obtained within a preset time period.

[0210] In an optional embodiment, the second acquisition module 92 is specifically configured to: preprocess the acquired current frame to obtain a preprocessed current frame; take the starting moment when the processing parameter map of the preprocessed current frame is processed as the starting moment, and acquire the processing parameter map of the previous frame of the current frame in the case where the processing parameter map of the current frame is not obtained within a preset time period.

[0211] In an optional embodiment, the device is further configured to: in the case where the processing parameter map of the current frame is obtained within a preset time period, process the current frame according to the processing parameter map of the current frame to obtain a preview picture of the current frame.

[0212] In an alternative embodiment, the apparatus is further configured to: continue to determine the processing parameter map of the current frame and store it when the processing parameter map of the current frame is not obtained within a preset time period.

[0213] In an alternative embodiment, the apparatus is further configured to: after obtaining the processing parameter map of the current frame, fuse the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame.

[0214] In an alternative embodiment, the apparatus is further configured to: perform AI processing on the processing parameter map of the previous frame using the current frame to update the processing parameter map of the previous frame.

[0215] In the alternative embodiment where the apparatus fuses the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame, it includes: performing weighted summation on the processing parameter map of the current frame and the processing parameter map of the previous frame according to the preset weight value of the current frame and the weight value of the previous frame to update the processing parameter map of the current frame.

[0216] In an alternative embodiment, the preview module 93 is specifically configured to: perform AI processing on the processing parameter map of the previous frame using the current frame to update the processing parameter map of the previous frame; process the current frame according to the processing parameter map of the previous frame to obtain the preview picture of the current frame.

[0217] In an alternative embodiment, the processing parameter map is a depth map.

[0218] In an alternative embodiment, the preview module 93 is specifically configured to: generate a blurred mask map of the current frame according to the depth map of the previous frame; perform blur processing on the current frame according to the blurred mask map of the current frame and the rendering environment of the current frame to obtain the preview picture of the current frame.

[0219] In practical applications, the above-mentioned first acquisition module 91, second acquisition module 92, and preview module 93 can be implemented by a processor located on the image preview device, specifically implemented by a CPU, microprocessor (Microprocessor Unit, MPU), digital signal processor (Digital Signal Processing, DSP), or field programmable gate array (Field Programmable Gate Array, FPGA), etc.

[0220] Figure 10 The structure diagram of an alternative electronic device provided by the embodiments of the present application is as Figure 10 shown. The embodiments of the present application provide an electronic device 1000, including:

[0221] A processor 101 and a storage medium 102 storing executable instructions of the processor; the storage medium 102 depends on the processor 101 to perform operations through a communication bus 103. When the instructions are executed by the processor, the image preview method executed on the processor side in one or more of the above embodiments is executed.

[0222] It should be noted that in actual application, each component in the computer device is coupled together through the communication bus 103. It can be understood that the communication bus 103 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 103 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 10 all kinds of buses are labeled as the communication bus 103.

[0223] An embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the image preview method described in one or more of the above embodiments.

[0224] Among them, the computer-readable storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.

[0225] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program codes.

[0226] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0227] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0228] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0229] As mentioned above, the above is only a preferred embodiment of the present application and is not used to limit the protection scope of the present application.

Claims

1. An image preview method, characterized in that, Including: Obtain the current frame; In the case where the processing parameter map of the current frame is not obtained within a preset time period, obtain the processing parameter map of the previous frame of the current frame; Process the current frame according to the processing parameter map of the previous frame to obtain a preview image of the current frame.

2. The method according to claim 1, wherein The step of obtaining the processing parameter map of the previous frame of the current frame in the case where the processing parameter map of the current frame is not obtained within a preset time period includes: Taking the moment when the current frame is obtained as the starting moment, in the case where the processing parameter map of the current frame is not obtained within a preset time period, obtain the processing parameter map of the previous frame of the current frame.

3. The method according to claim 1, wherein The step of obtaining the processing parameter map of the previous frame of the current frame in the case where the processing parameter map of the current frame is not obtained within a preset time period includes: Preprocess the obtained current frame to obtain a preprocessed current frame; Taking the starting moment when processing the processing parameter map of the preprocessed current frame as the starting moment, in the case where the processing parameter map of the current frame is not obtained within a preset time period, obtain the processing parameter map of the previous frame of the current frame.

4. The method according to claim 1, characterized in that, The method further includes: In the case where the processing parameter map of the current frame is obtained within a preset time period, process the current frame according to the processing parameter map of the current frame to obtain a preview image of the current frame.

5. The method according to claim 1, characterized in that, The method further includes: In the case where the processing parameter map of the current frame is not obtained within a preset time period, continue to determine the processing parameter map of the current frame and store it.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: After obtaining the processing parameter map of the current frame, fuse the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame.

7. The method according to claim 6, wherein The method further includes: Use the current frame to perform AI processing on the processing parameter map of the previous frame to update the processing parameter map of the previous frame.

8. The method according to claim 6, wherein The step of fusing the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame includes: According to the preset weight value of the current frame and the weight value of the previous frame, perform weighted summation on the processing parameter map of the current frame and the processing parameter map of the previous frame to update the processing parameter map of the current frame.

9. The method according to any one of claims 1 to 5, characterized in that The step of processing the current frame according to the processing parameter map of the previous frame to obtain a preview image of the current frame includes: Use the current frame to perform AI processing on the processing parameter map of the previous frame to update the processing parameter map of the previous frame; Process the current frame according to the processing parameter map of the previous frame to obtain a preview image of the current frame.

10. The method according to any one of claims 1 to 5, characterized in that, The processing parameter map is a depth map.

11. The method according to claim 10, wherein The step of processing the current frame according to the processing parameter map of the previous frame to obtain a preview image of the current frame includes: Generate a blurred mask map of the current frame according to the depth map of the previous frame; Perform blur processing on the current frame according to the blurred mask map of the current frame and the rendering environment of the current frame to obtain a preview image of the current frame.

12. An image preview device, characterized in that, Including: A first acquisition module for acquiring the current frame; A second acquisition module, configured to acquire a processing parameter map of the previous frame of the current frame when the processing parameter map of the current frame has not been obtained within a preset time period; A preview module, configured to process the current frame according to the processing parameter map of the previous frame to obtain a preview image of the current frame.

13. An electronic device, characterized in that, Comprising: A processor and a storage medium storing executable instructions of the processor; The storage medium depends on the processor to execute operations through a communication bus. When the instructions are executed by the processor, the image preview method according to any one of claims 1 to 11 above is executed.

14. A computer storage medium, characterized in that, Storing executable instructions, when the executable instructions are executed by one or more processors, the processor executes the image preview method according to any one of claims 1 to 11.

Images