Photographing method, photographing apparatus, and storage medium

Abstract

The present disclosure relates to a photographing method, a photographing device and a storage medium. The photographing method comprises: in response to receiving a photographing instruction, determining an image processing algorithm used for generating a target image, the image processing algorithm comprising a plurality of processing flows executed in sequence. In the process of executing the plurality of processing flows in sequence, determining a processing time consumed by a first processing flow, the first processing flow being a processing flow that is most recently completed in execution among the plurality of processing flows; based on the processing time, determining a second processing flow for generating the target image, and generating the target image based on the second processing flow. Through the present disclosure, it is avoided that the processing cannot return or the processing is timed out due to algorithm complexity, and the terminal camera stability is improved.

Description

Technical Field

[0001] This disclosure relates to the field of photography, and in particular to photographic methods, photographic devices, and storage media. Background Technology

[0002] With the continuous development of terminal imaging technology, taking pictures based on image processing algorithms, optimizing image quality, and acquiring captured images has become an important part of the field of terminal photography.

[0003] In related technologies, image processing algorithms are complex processes involving multiple stages. For heavyweight image processing algorithms that require a large number of input frames or relatively complex processing of single-frame images, the algorithms are time-consuming, have long processing flows, and are complex in their overall processing, which affects the stability of the terminal camera system during imaging. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a photographing method, a photographing device, and a storage medium.

[0005] According to a first aspect of the present disclosure, a method for taking a picture is provided, comprising: in response to receiving a picture-taking instruction, determining an image processing algorithm used to generate a target image, the image processing algorithm including a plurality of processing flows executed sequentially; determining the processing time consumed by a first processing flow during the sequential execution of the plurality of processing flows, the first processing flow being the processing flow most recently completed among the plurality of processing flows; determining a second processing flow for generating the target image based on the processing time, and generating the target image based on the second processing flow.

[0006] In one embodiment, determining a second processing flow for generating the target image based on the processing time, and generating the target image based on the second processing flow, includes: if the processing time exceeds a time threshold, determining the second processing flow as the processing flow that has been completed among the plurality of processing flows; stopping the execution of the processing flows that have not been completed among the plurality of processing flows, taking the image obtained from the most recently completed processing flow as the target image, and outputting the target image.

[0007] In one embodiment, determining a second processing flow for generating the target image based on the processing time, and generating the target image based on the second processing flow, includes: if the processing time does not exceed a time threshold, determining the second processing flow as an incomplete processing flow among the plurality of processing flows; executing the next processing flow, repeatedly executing the processing time for determining the next processing flow, and determining a subsequent process for generating the target image based on the processing time, until the plurality of processing flows are completed and the target image is output, or until the processing time of the subsequent process exceeds the time threshold, ending the image processing algorithm, and using the image obtained from the processing flow whose processing time exceeds the time threshold as the target image, and outputting the target image.

[0008] In one embodiment, determining the processing time consumed by the first processing flow includes: calling back the most recently completed processing flow and determining the processing time consumed by the most recently completed processing flow; determining a second processing flow for generating the target image based on the processing time, and generating the target image based on the second processing flow includes: determining a time threshold corresponding to the most recently completed processing flow, wherein there is a correspondence between the processing flow and the time threshold; determining a second processing flow for generating the target image based on the processing time and the time threshold, and generating the target image based on the second processing flow.

[0009] In one embodiment, the correspondence between the processing flow and the time threshold has a matching relationship with the temperature; determining the time threshold corresponding to the most recently completed processing flow includes: determining the current temperature, and determining the time threshold corresponding to the most recently completed processing flow based on the correspondence between the time threshold and the temperature.

[0010] In one embodiment, the method further includes: if the number of times the same processing flow ends exceeds a threshold, a prompt message is issued, the prompt message being used to indicate the processing flow that exceeds the threshold.

[0011] In one embodiment, the plurality of processing flows include: a data initialization flow, an image resource processing flow, a core algorithm processing flow, a resource algorithm recycling flow, and a data result return flow.

[0012] In one embodiment, the image processing algorithm includes a multi-frame fusion algorithm for fusing multiple frames of images and determining the fused image as the target image.

[0013] According to a second aspect of the present disclosure, a photographing apparatus is provided, comprising: a determining unit, configured to, in response to receiving a photographing command, determine an image processing algorithm used to generate a target image, the image processing algorithm including a plurality of processing flows executed sequentially; and a processing unit, configured to, during the sequential execution of the plurality of processing flows, determine a processing time consumed by a first processing flow, the first processing flow being the most recently completed processing flow among the plurality of processing flows, determine a second processing flow for generating the target image based on the processing time, and generate the target image based on the second processing flow.

[0014] In one embodiment, the processing unit determines a second processing flow for generating the target image based on the processing time in the following manner: if the processing time exceeds a time threshold, the second processing flow is determined to be a processing flow that has been completed among the plurality of processing flows; the processing flows that have not been completed among the plurality of processing flows are stopped, and the image obtained by the most recently completed processing flow is taken as the target image and the target image is output.

[0015] In one embodiment, the processing unit determines a second processing flow for generating the target image based on the processing time, and generates the target image based on the second processing flow, in the following manner: if the processing time does not exceed a time threshold, the second processing flow is determined to be an incomplete processing flow among the plurality of processing flows; the next processing flow is executed, the processing time for determining the next processing flow is repeatedly executed, and the subsequent flow for generating the target image is determined based on the processing time, until the plurality of processing flows are completed and the target image is output, or until the processing time of the subsequent flow exceeds the time threshold, the image processing algorithm is terminated, and the image obtained by the processing flow whose processing time exceeds the time threshold is taken as the target image and the target image is output.

[0016] In one embodiment, the processing unit determines the processing time consumed by the first processing flow and generates a target image based on the second processing flow in the following manner: it calls back the most recently completed processing flow and determines the processing time consumed by the most recently completed processing flow; the step of determining the second processing flow for generating the target image based on the processing time and generating the target image based on the second processing flow includes: determining a time threshold corresponding to the most recently completed processing flow, wherein there is a correspondence between the processing flow and the time threshold; determining the second processing flow for generating the target image based on the processing time and the time threshold, and generating the target image based on the second processing flow.

[0017] In one embodiment, the processing flow has a correspondence with the time threshold and a matching relationship with the temperature; the processing unit determines the time threshold corresponding to the most recently completed processing flow in the following way: determining the current temperature, and determining the time threshold corresponding to the most recently completed processing flow based on the correspondence between the time threshold and the temperature.

[0018] In one embodiment, the processing unit is further configured to: if the number of times the same processing flow ends exceeds a threshold, issue a prompt message, the prompt message being used to indicate the processing flow that exceeds the threshold.

[0019] In one embodiment, the plurality of processing flows include: a data initialization flow, an image resource processing flow, a core algorithm processing flow, a resource algorithm recycling flow, and a data result return flow.

[0020] In one embodiment, the image processing algorithm includes a multi-frame fusion algorithm for fusing multiple frames of images and determining the fused image as the target image.

[0021] According to a third aspect of the present disclosure, a photographing apparatus is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the photographing method described in the first aspect or any embodiment of the first aspect.

[0022] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the photographing method of the first aspect or any embodiment of the first aspect.

[0023] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: In response to processing an image according to an image processing algorithm, multiple sub-processes in the image processing algorithm are sequentially processed on the image to be processed. The processing time used by the most recently completed process is monitored, and based on the processing time, the subsequent algorithm processing flow is determined, and the target image is output. This disclosure avoids processing failures or timeouts due to algorithm complexity, thus improving the stability of the terminal camera.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] Figure 1 This is a block diagram illustrating a method for taking a photograph based on an image processing algorithm to generate a target image, according to an exemplary embodiment of the present disclosure.

[0027] Figure 2 This is a block diagram illustrating a method for taking a picture based on an image processing algorithm to generate a target image, according to yet another exemplary embodiment of this disclosure.

[0028] Figure 3 This is a block diagram illustrating an image signal processing method according to an exemplary embodiment of the present disclosure.

[0029] Figure 4 This is a block diagram illustrating the execution process of an image processing algorithm according to yet another exemplary embodiment of the present disclosure.

[0030] Figure 5 This is a flowchart illustrating a photographing method according to an exemplary embodiment.

[0031] Figure 6 This is a block diagram illustrating a method for taking a picture based on an image processing algorithm to generate a target image, according to yet another exemplary embodiment of this disclosure.

[0032] Figure 7 This is a flowchart illustrating a method for image processing based on processing time and time threshold according to an exemplary embodiment.

[0033] Figure 8 This is a block diagram illustrating a method for taking a picture based on an image processing algorithm to generate a target image, according to yet another exemplary embodiment of this disclosure.

[0034] Figure 9 This is a flowchart illustrating a method for image processing based on processing time and time threshold according to yet another exemplary embodiment.

[0035] Figure 10 This is a flowchart illustrating a method for determining processing time and performing image processing based on processing time, according to an exemplary embodiment.

[0036] Figure 11 This is a flowchart illustrating a method for determining a time threshold according to an exemplary embodiment.

[0037] Figure 12 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0038] Figure 13 This is a block diagram illustrating a multi-frame image processing method according to yet another exemplary embodiment of the present disclosure.

[0039] Figure 14This is a block diagram illustrating a photographing device according to an exemplary embodiment.

[0040] Figure 15 This is a block diagram illustrating an apparatus for taking a photograph according to an exemplary embodiment. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0042] The photographing method provided in this disclosure is applicable to scenarios where image processing algorithms are used for taking photographs. For example, it is applicable to scenarios requiring multi-frame fusion algorithms for photographing, such as night scene imaging or high dynamic range imaging.

[0043] Among them, night scene imaging can capture images with rich detail in low-light nighttime conditions. High dynamic range imaging captures images with high resolution and abundant shadow detail under normal exposure conditions. Both night scene imaging and high dynamic range imaging are essential photography functions for current mobile devices.

[0044] In related technologies, imaging techniques using image processing algorithms fall into two categories: one is based on processing RAW domain images (the raw images output by the sensor); the other is based on processing YUV domain images (the images obtained after conversion from the raw images). Both methods aim to highly reproduce the real-world scene information in nighttime conditions or acquire images with abundant detail. RAW domain image processing can be considered an improvement over YUV domain image processing, offering better reproduction of nighttime scene information and cleaner noise reduction. The main difference between RAW and YUV domain image processing lies in the sequence of image signal processing (ISP) and the application of image processing algorithms; however, the processing flow for multiple frames is essentially the same.

[0045] The following explains RAW domain image processing and YUV domain image processing in related technologies:

[0046] (1) Image processing scheme based on RAW domain image processing algorithm: The main feature of this scheme is that it performs multi-frame fusion processing based on RAW domain images. The purpose is to reduce the noise impact of the image signal processing module (ISP module) by performing multi-frame processing on RAW domain images directly output by the sensor. Figure 1The block diagram of the image processing algorithm-based target image generation method is shown. The RAW domain-based image processing algorithm scheme mainly includes the following steps: a. User presses the shutter: After selecting an image processing algorithm (such as a night scene algorithm) in the terminal camera application, the user clicks the virtual button to issue the shooting command and takes a picture. b. Night scene exposure parameters: Based on the shooting mode selected by the user, the terminal adaptively adjusts the exposure parameters during shooting (such as adjusting the aperture size and controlling the flash brightness) and acquires the image. c. Sensor outputs multiple frames: The sensor acquires light signals, converts them into electrical signals, and continuously outputs multiple frames of RAW domain images. d. Image signal processing - bad pixel correction: The image signal processing module is called to perform bad pixel correction on the generated multiple frames of RAW domain images. e. RAW domain image processing algorithm processing: The algorithm processing is performed on the multiple frames of images with bad pixel correction to output the target image. f. Image signal processing: Noise reduction, white balance, and other image signal processing are performed on the fused image, and the RAW domain image is converted into a YUV domain image. g. Other single-frame algorithm processing: The converted YUV domain image is optimized. h. Output: Output the target image.

[0047] (2) Image processing scheme based on YUV domain image processing algorithm: The difference between this scheme and the scheme based on RAW domain image processing algorithm lies in the different order of image signal processing (ISP processing) and algorithm processing. In this scheme, after acquiring multiple frames of RAW domain images through a sensor, image signal processing is first performed on the RAW domain images to obtain multiple YUV domain images, which are then processed by the algorithm. For example... Figure 2 The block diagram of the image processing algorithm-based target image generation method is shown in the figure. The image processing scheme based on the YUV domain image processing algorithm mainly includes the following steps: a. The user presses the shutter. b. Night scene exposure parameters. c. The sensor outputs multiple frames. d. Image signal processing. e. YUV domain image processing algorithm processing. f. Other single-frame algorithm processing. g. Image output. The specific implementation details of each step are the same as or similar to the scheme of image processing based on the RAW domain image processing algorithm, so they are not discussed in detail.

[0048] Both RAW domain image processing and YUV domain image processing mentioned above require image signal processing, such as... Figure 3The block diagram of the image signal processing method is shown. The general process of image signal processing in related technologies is as follows: Synchronization clock (Sensor-DDR) - Bad pixel correction / color focusing (BPC / PDPC / BCC) - Channel Gains - High dynamic range (HDR rec / HDR MAC) - Green Imbalance Correction - Raw domain noise reduction (ABF) - Black level Substration - Lens Roll Off - White Balance - Demosaic - Color Correction - Local brightening (GTM) - Overall brightening (Gamma) - Color Space Transform - High-frequency noise reduction (Hybrid Noise Reduction) - Downsampling - Color Correction - Comprehensive processing (IPE NPS / PPS) - Image output. The image signal processing procedures in RAW domain image processing and YUV domain image processing differ in detail. In RAW domain image processing, RAW domain images need to be converted into YUV domain images through image signal processing. Therefore, the image signal processing procedure in RAW domain image processing also has the function of image format conversion.

[0049] The RAW domain image processing and YUV domain image processing described above have the same or similar image processing algorithm-based image processing workflows, such as... Figure 4 The flowchart of the image processing algorithm execution process is shown in the figure. In related technologies, the general process of processing multiple frames of images based on multi-frame processing algorithms includes the following steps:

[0050] a) Enable Image Processing Algorithm: The response enables the image processing algorithm by issuing a command from the user to process the image based on the image processing algorithm.

[0051] b) Data initialization: Based on user-set parameters (such as image size), initialize various data for the image processing algorithm.

[0052] c) Image resource preprocessing: Preprocessing images used for algorithm processing, such as noise reduction, cropping, and alignment.

[0053] d) Core algorithm processing: The preprocessed image is processed based on image processing algorithms.

[0054] e) Algorithm resource reclamation: In response to the completion of various algorithmic processing of the image, algorithmic resources used to execute different algorithmic processing flows are gradually reclaimed.

[0055] f) Return data results and end the algorithm: Output the target image obtained by the image processing algorithm and end the image processing algorithm.

[0056] Among them, the YUV domain image processing algorithm and the RAW domain image processing algorithm both have good effects in outputting high-quality images, but their drawbacks are also quite obvious. If the image processing algorithm is heavyweight, requiring a large number of input frames, processing each frame individually, or if the processing of a single frame is relatively complex, it will lead to long image processing time and complex processing procedures. Therefore, the YUV domain image processing algorithm and the RAW domain image processing algorithm are characterized by high algorithm consumption, long processing flow, and complex processing procedures. Stuttering may occur at any stage of the image processing algorithm, causing the algorithm to be unable to return or to time out, affecting the stability of the terminal when imaging based on the image processing algorithm.

[0057] In one implementation, the target image is generated based on a multi-frame processing algorithm. The optical sensor used in the terminal camera acquires multiple original images through continuous exposure. In imaging processes requiring multi-frame fusion algorithms, such as night scene imaging or high dynamic range imaging, multiple images with different exposures (including a normally exposed image EV0, an overexposed image EV+, and a weakly exposed image EV-) are typically synthesized into a single fused image. This type of night scene image, with its ability to surpass human visual perception, has become a key research focus in the current image and video fields. However, for such heavyweight algorithms that require a large number of input image frames, prolonged algorithm processing can affect the system stability of the terminal camera during imaging. Therefore, how to quickly output the image while ensuring the image quality meets expectations, and improve camera stability, has become a significant problem, preventing the algorithm from failing to return or timing out.

[0058] In view of this, this disclosure proposes a photo-taking method that breaks down the entire image processing algorithm into multiple sub-processes. After image processing begins, the processing time of the current process is monitored, and a threshold judgment is made based on the processing time to determine the subsequent process of image processing. This avoids terminal camera failures caused by the algorithm's inability to return or processing timeouts, thereby improving the stability of the terminal camera.

[0059] Figure 5 This is a flowchart illustrating a photographing method according to an exemplary embodiment. Figure 5 As shown, the method includes steps S101 to S103.

[0060] In step S101, in response to receiving the photo-taking command, the image processing algorithm used to generate the target image is determined.

[0061] The image processing algorithm includes multiple processing steps executed sequentially.

[0062] Image processing algorithms can be understood as algorithms that improve image quality. For example, they involve processing multiple frames of images to obtain a target image, including image feature fusion to obtain a fused multi-frame image, or image stitching to combine images from multiple different regions into a single large image.

[0063] The image processing algorithm process includes different processing steps, such as: a sub-process for configuring data processing based on shooting parameters, a preprocessing process for image alignment and cropping, and a formal process for processing the image based on the algorithm. The algorithms executed in each sub-process are different.

[0064] In step S102, during the sequential execution of multiple processing flows, the processing time consumed by the first processing flow is determined.

[0065] The first processing flow is the processing flow that was most recently completed among multiple processing flows.

[0066] In step S103, a second processing flow for generating the target image is determined based on the processing time, and the target image is generated based on the second processing flow.

[0067] In this embodiment of the disclosure, the image processing algorithm process is divided into multiple sub-processes based on different stages of the algorithm process. These sub-processes are arranged according to the execution order of the image processing algorithm process. Based on the execution order of the sub-processes, this disclosure sequentially determines the processing time consumed by the outgoing processing flow and further performs image processing based on the processing time.

[0068] In an exemplary embodiment of this disclosure, the image processing algorithm used to process the image during the photographing process can be either a RAW-domain based image processing algorithm or a YUV-domain based image processing algorithm. In other words, the photographing method of this disclosure can be applied to both RAW-domain based and YUV-domain based image processing algorithms.

[0069] In an exemplary embodiment of this disclosure, an image processing algorithm based on YUV domain images is used as an example for explanation. Figure 6 This paper demonstrates a method for processing YUV domain images using image processing algorithms to obtain a fused image. For example... Figure 6The block diagram of the image processing algorithm-based target image generation method is shown. In image processing algorithm modes such as night scene imaging or high dynamic range imaging, the processing flow of the image processing algorithm based on YUV domain images includes: Responding to the user's image capture command, the terminal receives the command and controls the camera to start acquiring images. The camera acquires images and generates multiple frames of raw images (RAW domain images), then performs image signal processing on the raw images, converting the RAW domain images into YUV domain images. The processed multi-frame images are input into the algorithm processing module, which performs image processing algorithms on the processed multi-frame images to obtain a fused image. Finally, the fused image is output.

[0070] It is understandable that the main difference between RAW domain image processing and YUV domain image processing lies in the sequence of image signal processing (ISP processing) and image processing algorithms. The overall algorithmic processing flow is essentially the same. This disclosure, when processing images based on RAW domain images, does not change the execution method of each step; only the execution order of image signal processing (ISP processing) and image processing algorithms needs to be adaptively adjusted.

[0071] In this embodiment of the disclosure, based on the ideal processing time of different sub-processes during the execution of the image processing algorithm (the time to control the overall imaging time within a certain period without affecting the user experience), the maximum value of the ideal processing time of each sub-process can be set as a corresponding time threshold. By obtaining the actual processing time of the sub-process and comparing it with the corresponding time threshold, a threshold determination is achieved. Different subsequent execution processes exist depending on whether the processing time meets the threshold or not. The following embodiments of this disclosure describe the method for determining and executing the subsequent processing flow of the processing algorithm based on the processing time.

[0072] Figure 7 This is a flowchart illustrating a method for image processing based on processing time, according to an exemplary embodiment. Figure 7 As shown, the method includes steps S201 to S202.

[0073] In step S201, in response to the processing time exceeding the time threshold, the second processing flow is determined to be the processing flow that has been completed among multiple processing flows.

[0074] In step S202, the execution of the processing flow that has not been completed in the multiple processing flows is stopped, and the image obtained by the most recently completed processing flow is taken as the target image and output.

[0075] In this embodiment of the disclosure, the second processing path is the processing flow for finally generating the target image.

[0076] In this embodiment, a time threshold corresponds to the current process flow. The time threshold is a critical time value used to determine whether the processing time of the current process flow is abnormal. If the processing time is greater than the time threshold, it indicates that the current process flow has timed out and there is an anomaly. If the processing time is less than the time threshold, it indicates that the current process flow is executed normally. By collecting the processing time of the current process flow and executing subsequent processes of the image processing algorithm based on the processing time and the corresponding time threshold, the problem of the algorithm failing to return or timeout during image processing is avoided.

[0077] In an exemplary embodiment of this disclosure, such as Figure 8 The block diagram of the image-taking method based on image processing algorithms for generating target images is shown below. Figure 8 The "Algorithm Processing Entry Point - Algorithm Processing 4" in the text corresponds to Figure 6 The "image processing algorithm" disclosed herein is divided into multiple sub-processes through callbacks. After the user presses the shutter button, the underlying layer receives the request, the camera outputs the image, and the image signal is processed, the algorithm enters the algorithm processing entry point and executes Algorithm Processing 1, Callback Interface 1, Algorithm Processing 2, Callback Interface 2, Algorithm Processing 3, Callback Interface 3, Algorithm Processing 4, and Callback Interface 4 in sequence. After execution, the target image is output. Each algorithm processing stage corresponds to a sub-process. The processing time is determined through the callback interface. A threshold judgment is performed based on the processing time and the time threshold built into the callback interface. Based on the judgment result, the subsequent processing flow is executed.

[0078] In this embodiment, if the most recently completed processing flow exceeds a time threshold, it indicates that the processing flow is too long, has processing anomalies, and affects system stability. The image processed by the current flow is then directly output as the target image. This avoids system stability issues caused by long subsequent processing flows and improves the user experience.

[0079] It is understandable that the complexity of different sub-processes in an image processing algorithm varies, and the processing time required for different sub-processes also varies. Therefore, different sub-processes correspond to different time thresholds.

[0080] It's understandable that when taking a picture based on image processing algorithms, the acquired image has already undergone image processing by the image signal processing module before being processed by the algorithm. Furthermore, after algorithmic processing, subsequent single-frame image processing is performed before the output image is finalized. The algorithm also selects relatively high-quality images for further processing and then composites them. Therefore, even when the image processing algorithm is not fully completed, modifying the process to only output the image after intermediate processing and using it as the target image for subsequent processing can still, to a certain extent, meet the user's expectations.

[0081] In this embodiment of the disclosure, if the processing time of the most recently completed processing flow does not meet the threshold, the execution of subsequent image processing algorithms will be stopped. Conversely, the most recently completed processing flow will continue to the next processing flow of the image processing algorithm. The following embodiments of this disclosure describe the subsequent processing flow when the threshold is met.

[0082] Figure 9 This is a flowchart illustrating a method for image processing based on processing time and a time threshold, according to yet another exemplary embodiment. For example... Figure 9 As shown, this disclosure performs threshold judgment based on the processing time and the corresponding time threshold of the processing flow, and then executes subsequent processing flows until the target image is output:

[0083] Determine the processing time consumed by the most recently completed processing flow 1 and determine the corresponding time threshold. Check if the processing time of the most recently completed processing flow 1 exceeds the time threshold. If the processing time exceeds the time threshold, terminate the image processing algorithm, use the processed image obtained from the current processing flow 1 as the target image, and output the target image. If the processing time does not exceed the time threshold, execute processing flow 2, determine the processing time consumed by processing flow 2, and determine the corresponding time threshold. Then check if the processing time of processing flow 2 exceeds the time threshold… Repeat the above threshold determination process until entering processing flow n, which completes the image processing algorithm. If the processing time of processing flow n does not exceed the time threshold, enter the data result return process, fuse the multi-frame images to determine the target image, and output the target image.

[0084] In this embodiment, if the processing time of the current processing flow meets a time threshold, upon entering the next processing flow, the process of obtaining the processing time of the current processing flow and performing threshold judgment on the processing time is repeated. Based on whether the processing time of subsequent processing flows meets the time threshold, there are two final output results: one is that the processing time of each subsequent sub-processing flow meets the threshold, and the finally fused image is output as the target image; the other is that if one of the subsequent sub-processing flows does not meet the threshold, the processing result of the sub-process that does not meet the threshold is used as the target image and output. By splitting the image processing algorithm and performing threshold judgment in stages, the subsequent processing flow and the output target image are determined. This avoids terminal camera malfunctions caused by excessively long algorithm processing flows leading to failure to return or processing timeouts, thus ensuring image quality and improving terminal camera stability.

[0085] In this embodiment of the disclosure, a mechanism for monitoring the image processing flow is required to determine the processing time between the most recently completed processing flows, and then to determine the threshold of the processing time and execute the subsequent processing flow. The following embodiments of this disclosure describe the method of executing the subsequent processing flow by determining the processing time.

[0086] Figure 10 This is a flowchart illustrating a method for determining processing time and performing image processing based on processing time, according to yet another exemplary embodiment. Figure 10 As shown, the method includes steps S301 to S303.

[0087] In step S301, the most recently completed processing flow is called back to determine the processing time consumed by the most recently completed processing flow.

[0088] In step S302, a time threshold corresponding to the most recently completed processing flow is determined.

[0089] There is a corresponding relationship between the processing flow and the time threshold.

[0090] In step S303, a second processing flow for generating the target image is determined based on the processing time and time threshold, and the target image is generated based on the second processing flow.

[0091] In this embodiment of the disclosure, such as Figure 8 As shown, after the image to be processed enters the image processing algorithm's processing flow, each sub-process of the image processing algorithm has a corresponding callback interface, and the callback interface has a preset time threshold for the corresponding sub-process. After the sub-process is completed, the completed process is called back to the corresponding callback interface based on the callback mechanism. The callback interface determines the start time and end time of the sub-process, and then determines the processing time of the sub-process based on the difference between the start and end times. After obtaining the processing time, the preset time threshold in the callback interface is obtained, a threshold judgment is performed, the relationship between the time threshold and the processing time is compared, and the subsequent multi-frame processing flow is executed based on the threshold judgment result.

[0092] In this embodiment of the disclosure, there is a correspondence between the processing flow and the time threshold, and the time threshold is not fixed. It can be adaptively changed based on the current operating conditions of the terminal. The following embodiments of this disclosure illustrate a method for determining the time threshold.

[0093] Figure 11 This is a flowchart illustrating a method for determining a time threshold according to an exemplary embodiment. Figure 11 As shown, the method includes steps S401 to S402.

[0094] In step S401, the current temperature is determined.

[0095] The process flow has a correspondence with the time threshold and a matching relationship with the temperature.

[0096] In step S402, the time threshold corresponding to the current processing flow is determined based on the correspondence between the time threshold and temperature.

[0097] In this embodiment, the terminal employs different scheduling strategies at different temperatures. At low temperatures, the scheduling strategy is aggressive, allowing the terminal to release more power, resulting in faster image processing algorithms and shorter processing times. At high temperatures, to prevent overheating, the terminal's scheduling strategy becomes more conservative, reducing power release and consequently slowing down image processing algorithms, leading to longer processing times. Based on these different scheduling strategies at varying temperatures, two or more time thresholds can be set. The size of the time threshold is positively correlated with the terminal temperature.

[0098] It is understandable that the processing speed of image processing algorithms is not only related to the power consumption of the terminal, but also to the number of images processed. Different image processing algorithms require different numbers of images. Therefore, different time thresholds can be set for different image processing algorithms (such as night scene imaging and high dynamic range imaging).

[0099] In this embodiment, the time thresholds corresponding to different processes in the image processing algorithm are adjusted in a targeted manner. This avoids incorrect threshold judgments on the processing time of sub-processes in the image processing algorithm when the terminal operating conditions change, further improving the system stability of the image processing algorithm process.

[0100] In this embodiment of the disclosure, corresponding data of the image processing algorithm process can be recorded, thereby optimizing the image processing algorithm. The following embodiments of the disclosure further illustrate the method of obtaining a target image based on an image processing algorithm.

[0101] Figure 12 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 12 As shown, the method includes steps S501 to S502.

[0102] In step S501, the processing data of the image processing algorithm is acquired and stored.

[0103] In step S502, if the number of times the same processing flow ends exceeds the number threshold, a prompt message is issued.

[0104] The prompt message is used to indicate the processing procedure when the number of attempts exceeds the threshold.

[0105] In this embodiment of the disclosure, abnormal data can be recorded each time an image processing algorithm malfunctions. The abnormal data includes the corresponding process that fails to meet the threshold. After determining that the number of times the same process malfunctions exceeds the threshold, a prompt message can be sent to the cloud server. After receiving the feedback, the cloud server can perform targeted optimization of the algorithm for the corresponding process, thereby improving the user experience.

[0106] In one embodiment of this disclosure, multiple processing flows include: a data initialization flow, an image resource processing flow, a core algorithm processing flow, a resource algorithm recycling flow, and a data result return flow.

[0107] In an exemplary embodiment of this disclosure, night scene imaging is performed based on a multi-frame processing algorithm. After a user requests to take a night scene photo through a camera application, the underlying layer receives the photo request and sets a multi-frame exposure combination EVList to acquire images with different exposure levels. For example: EV0, EV0, EV0, EV0, EV0, EV0, EV0, EV-, EV+, where EV0 is a normally exposed image, EV+ is an overexposed image, and EV- is a weakly exposed image. After acquiring the multi-frame exposure group, the Automatic Exposure Control (AEC) generates the exposure time and gain. The AEC converts the exposure time and gain of the multi-frame images with different exposure levels into frame length and line count through a sensor module, realizing the conversion of the image's light signal into an electrical signal, and finally outputting the original RAW image. After the original RAW image passes through the image signal processing module, multiple frames of YUV domain images with different exposure levels are acquired. After receiving the multi-frame images, the night scene algorithm begins to perform image processing.

[0108] In one embodiment of this disclosure, the image processing algorithm includes a multi-frame fusion algorithm for fusing multiple frames of images and determining the fused image as the target image.

[0109] This disclosure provides an exemplary embodiment illustrating the image processing flow when the image processing algorithm is a multi-frame fusion algorithm. For example... Figure 13The block diagram of the multi-frame image processing method is shown. This method includes the following steps: After starting the multi-frame processing algorithm, the algorithm includes a data initialization process: initializing various data for multi-frame image processing based on user-set parameters (such as image size). An image resource processing process: preprocessing the multi-frame images used for fusion, such as noise reduction, cropping, and alignment. A core algorithm processing process: extracting image features from multiple frames of images with different exposures, and fusing the extracted target features into a target image. A resource algorithm recycling process: gradually recycling the algorithm resources used for feature extraction and feature fusion in response to the completion of feature extraction and feature fusion for each frame. A data result return process: outputting the fused target image and ending the multi-frame processing algorithm. Except for the final output target image data result return process, each of the other processes has a corresponding threshold judgment process, including: data initialization threshold judgment, image resource processing threshold judgment, core algorithm processing threshold judgment, and resource algorithm recycling threshold judgment. These are used to obtain the processing time of the corresponding process and perform threshold judgment based on the built-in time threshold to determine whether an anomaly has occurred in the corresponding processing process.

[0110] In an exemplary embodiment of this disclosure, the time thresholds for each processing flow in multi-frame image processing are designed as follows: data initialization flow - 1000ms, image resource preprocessing flow - 1000ms, core algorithm processing flow - 2000ms, algorithm resource recycling flow - 1000ms, and data result return flow - 150ms.

[0111] In this embodiment, multi-frame image processing is divided into multiple sub-stages based on different phases of multi-frame image processing. Each sub-stage has a corresponding time threshold, which can be adaptively adjusted based on the operating scenario of the camera terminal. After acquiring multiple frames of images to be processed and starting the multi-frame image processing flow, each sub-flow is executed sequentially according to the execution order. After the flow is completed, the processing time of the sub-flow is obtained through a callback, and the processing time of the sub-flow is judged against the corresponding time threshold. If the processing time does not meet the time threshold, the processing result of the sub-flow is output as the target image; if the processing time meets the time threshold, the next sub-flow is entered, and the above threshold judgment process is repeated until the entire multi-frame image processing flow is completed and the target image is output. In response to a sub-flow processing time not meeting the time threshold, the corresponding processing result is output as the target image. Through this disclosure, the entire multi-frame image processing flow is divided into multiple sub-flows, and a threshold judgment is performed on each sub-flow to determine whether to enter the subsequent flow. To avoid the multi-frame image processing algorithm failing to return or timeout due to factors such as the large number of image frames processed, the complexity of the processing flow, and the long processing time when directly executing the entire multi-frame image processing process, this method ensures the smooth completion of the multi-frame image processing flow while maintaining image quality, preventing the algorithm from failing to return or the multi-frame processing from timeout, and improving the stability of the terminal camera.

[0112] Based on the same concept, this disclosure also provides a photographing device 100.

[0113] It is understood that the photographing device 100 provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0114] Figure 14 This is a block diagram illustrating a photographing device 100 according to an exemplary embodiment. (Refer to...) Figure 14 The device includes a determining unit 101 and a processing unit 102.

[0115] The determining unit 101 is used to determine the image processing algorithm used to generate the target image in response to receiving a photo-taking command.

[0116] The multi-frame image processing process includes multiple processing flows executed sequentially.

[0117] The processing unit 102 is used to determine the processing time consumed by the first processing flow during the sequential execution of multiple processing flows, wherein the first processing flow is the processing flow that was most recently completed among the multiple processing flows, and based on the processing time, determine the second processing flow for generating the target image, and generate the target image based on the second processing flow.

[0118] In one embodiment, the processing unit 102 determines a second processing flow for generating the target image based on the processing time as follows: if the processing time exceeds a time threshold, the second processing flow is determined to be a processing flow that has been completed among multiple processing flows. The processing flows that have not been completed among the multiple processing flows are stopped, and the image obtained from the most recently completed processing flow is taken as the target image and output.

[0119] In one embodiment, the processing unit 102 determines a second processing flow for generating a target image based on processing time, and generates the target image based on the second processing flow as follows: If the processing time does not exceed a time threshold, the second processing flow is determined to be an incomplete processing flow among multiple processing flows. The next processing flow is executed, and the process of determining the processing time for the next processing flow and determining the subsequent flow for generating the target image based on the processing time is repeated until multiple processing flows are completed and the target image is output, or until the processing time of a subsequent flow exceeds the time threshold, at which point the image processing algorithm ends, and the image obtained from the processing flow whose processing time exceeds the time threshold is taken as the target image and output.

[0120] In one embodiment, the processing unit 102 determines the processing time consumed by the first processing flow as follows: It calls back the most recently completed processing flow and determines the processing time consumed by that flow. Based on the processing time, it determines a second processing flow for generating the target image and generates the target image based on the second processing flow, including: determining a time threshold corresponding to the most recently completed processing flow, wherein there is a correspondence between the processing flow and the time threshold. Based on the processing time and the time threshold, it determines the second processing flow for generating the target image and generates the target image based on the second processing flow.

[0121] In one embodiment, the correspondence between the processing flow and the time threshold is matched with the temperature. The processing unit 102 determines the time threshold corresponding to the most recently completed processing flow in the following manner: determining the current temperature, and determining the time threshold corresponding to the most recently completed processing flow based on the correspondence between the time threshold and the temperature.

[0122] In one embodiment, the processing unit 102 is further configured to: if the number of times the same processing flow ends exceeds a threshold, issue a prompt message, the prompt message being used to indicate the processing flow that exceeds the threshold.

[0123] In one implementation, the multiple processing flows include: a data initialization flow, an image resource processing flow, a core algorithm processing flow, a resource algorithm recycling flow, and a data result return flow.

[0124] In one embodiment, the image processing algorithm includes a multi-frame fusion algorithm for fusing multiple frames of images and determining the fused image as the target image.

[0125] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0126] Figure 15 This is a block diagram illustrating a device 200 for taking pictures according to an exemplary embodiment. Device 200 can be provided as a terminal. For example, device 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0127] Reference Figure 15 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0128] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0129] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0130] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0131] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0132] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0133] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0134] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0135] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0136] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0137] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0138] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0139] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0140] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0141] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0142] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0143] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0144] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for taking photos, characterized in that, include: In response to receiving a photo-taking command, the image processing algorithm used to generate the target image is determined, wherein the image processing algorithm includes multiple processing flows executed sequentially; During the sequential execution of the plurality of processing flows, the processing time consumed by the first processing flow is determined, wherein the first processing flow is the processing flow that was most recently completed among the plurality of processing flows. Based on the processing time, a second processing flow for generating the target image is determined, and the target image is generated based on the second processing flow; The second processing flow for generating the target image based on the processing time includes: If the processing time exceeds the time threshold, then the second processing flow is determined to be the processing flow that has been completed among the plurality of processing flows; If the processing time does not exceed the time threshold, then the second processing flow is determined to be the processing flow that has not been completed among the plurality of processing flows.

2. The method according to claim 1, characterized in that, If the processing time exceeds a time threshold, generating the target image based on the second processing flow includes: Stop executing any incomplete processing steps among the multiple processing steps, take the image obtained from the most recently completed processing step as the target image, and output the target image.

3. The method according to claim 1, characterized in that, If the processing time does not exceed the time threshold, generating the target image based on the second processing flow includes: The next processing flow is executed repeatedly, determining the processing time of the next processing flow and, based on the processing time, determining the subsequent flow to generate the target image, until the execution of the multiple processing flows is completed and the target image is output, or until the processing time of the subsequent flow exceeds the time threshold, the image processing algorithm ends, and the image obtained by the processing flow whose processing time exceeds the time threshold is taken as the target image and output.

4. The method according to any one of claims 1 to 3, characterized in that, Determining the processing time consumed by the first processing flow includes: Call back the most recently completed processing flow and determine the processing time consumed by the most recently completed processing flow; The step of determining a second processing flow for generating the target image based on the processing time, and generating the target image based on the second processing flow, includes: Determine the time threshold corresponding to the most recently completed processing flow, wherein there is a correspondence between the processing flow and the time threshold; Based on the processing time and time threshold, a second processing flow for generating the target image is determined, and the target image is generated based on the second processing flow.

5. The method according to claim 4, characterized in that, The processing flow has a correspondence with the time threshold and a matching relationship with the temperature. Determining the time threshold corresponding to the most recently completed processing flow includes: Determine the current temperature, and based on the correspondence between the time threshold and the temperature, determine the time threshold corresponding to the most recently completed processing flow.

6. The method according to claim 2, characterized in that, The method further includes: If the number of times the same processing flow ends exceeds the threshold, a prompt message will be issued to indicate that the processing flow has exceeded the threshold.

7. The method according to claim 1, characterized in that, The multiple processing flows include: The process includes data initialization, image resource processing, core algorithm processing, resource algorithm recycling, and data result return.

8. The photographing method according to claim 1, characterized in that, The image processing algorithm includes a multi-frame fusion algorithm, which is used to fuse multiple frames of images and determine the fused image as the target image.

9. A photographing device, characterized in that, include: A determining unit is configured to, in response to receiving a photographing command, determine the image processing algorithm used to generate the target image, wherein the image processing algorithm includes multiple processing flows executed sequentially; A processing unit is configured to determine the processing time consumed by a first processing flow during the sequential execution of the plurality of processing flows, wherein the first processing flow is the most recently completed processing flow among the plurality of processing flows, determine a second processing flow for generating the target image based on the processing time, and generate the target image based on the second processing flow. The processing unit determines the second processing flow for generating the target image based on the processing time in the following manner: If the processing time exceeds the time threshold, then the second processing flow is determined to be the processing flow that has been completed among the plurality of processing flows; If the processing time does not exceed the time threshold, then the second processing flow is determined to be the processing flow that has not been completed among the plurality of processing flows.

10. A photographing device, characterized in that, include: processor: Memory used to store processor-executable instructions; The processor is configured to perform the photographing method according to any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium stores instructions that, when executed by a processor, enable the processor to perform the photographing method according to any one of claims 1 to 8.

Images