Photographing processing method, electronic device, and computer-readable storage medium

Abstract

This application discloses a photo processing method, an electronic device, and a computer-readable storage medium. The method includes: opening a camera application; acquiring image data to be processed in response to a photo-taking operation; performing a first-stage algorithmic post-processing on the image data to be processed while the camera application is running in the foreground to obtain first image data, which is image data for which the algorithmic post-processing is incomplete; if a target operation is detected, or the camera application is running in the background or closed, performing a remaining stage of algorithmic post-processing on the first image data to obtain second image data, which is image data for which the algorithmic post-processing is complete. The target operation is used to view, edit, or share the image, and the remaining stage of algorithmic post-processing includes at least one stage of algorithmic post-processing. This application improves photo-taking performance and enhances the user's photo-taking experience through segmented photo-taking.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a photo processing method, electronic device, and computer-readable storage medium. Background Technology

[0002] With the increasing popularity of mobile phones and other electronic devices, taking photos using the camera function of mobile phones and other electronic devices has gradually become a trend.

[0003] There is a significant gap between the camera hardware specifications of mobile phones and other electronic devices and those of professional photography equipment. Therefore, after mobile phones and other electronic devices capture images with their cameras, they often rely on powerful computing resources to run a series of complex post-processing algorithms to improve the image quality caused by the less-than-ideal camera hardware specifications, thereby obtaining higher-quality images.

[0004] However, mobile phones and other electronic devices continuously run a series of complex post-processing algorithms during the photo-taking process to obtain the processed image in real time. These complex post-processing algorithms consume significant computing and memory resources, causing the processor to be under excessive load for extended periods. This leads to decreased photo-taking performance, device lag, and overheating, ultimately impacting the user's photography experience. Summary of the Invention

[0005] This application provides a photo processing method, an electronic device, and a computer-readable storage medium, which can solve the problem that the continuous and uninterrupted execution of all post-processing algorithms during the existing photo-taking process leads to low photo-taking performance and a poor photo-taking experience.

[0006] In a first aspect, embodiments of this application provide a photo processing method applied to an electronic device, the method comprising:

[0007] Open a camera app;

[0008] In response to a photo-taking operation, acquire the image data to be processed;

[0009] When a camera application is running in the foreground, the first stage of algorithmic post-processing is performed on the image data to be processed to obtain the first image data, which is the image data in which the algorithmic post-processing is not yet completed.

[0010] If a target operation is detected, or if the camera application is running in the background or closed, the remaining stage of the algorithm post-processing is performed on the first image data to obtain the second image data. The second image data is the image data after the algorithm post-processing. The target operation is used to view, edit, or share the image. The remaining stage of the algorithm post-processing includes at least one stage of algorithm post-processing.

[0011] As can be seen from the above technical solution, in this embodiment, the first stage of post-processing is executed when the camera application is running in the foreground. When the camera application is running in the background, closed, or when a target operation is detected, the remaining stage of post-processing is executed. That is, only a portion of the post-processing algorithm is executed during the photo-taking process; the remaining part is executed when no photo is being taken or when image manipulation is required. Compared to continuously executing all post-processing algorithms during the photo-taking process, this embodiment executes only a portion of the post-processing algorithms, reducing the consumption of computing and memory resources during the photo-taking process. This alleviates problems such as device lag and overheating, improves photo-taking performance, and enhances the user's photo-taking experience.

[0012] In some possible implementations of the first aspect, the target operation is viewing a thumbnail, which is displayed in the viewfinder of a camera application. The thumbnail corresponds to the image data to be processed. In this implementation, when the camera is in the foreground, if the user needs to view the image, the remaining post-processing algorithm can be triggered promptly, allowing the user to quickly obtain the processed image data and view it immediately. When the user views the thumbnail, the electronic device exits the shooting process and enters image preview mode. In image preview mode, there is no need for processing such as previewing video streams, thus reducing the computational and memory resource consumption during shooting, improving shooting performance and the user's shooting experience.

[0013] In some possible implementations of the first aspect, if the camera application is running in the background or closed, the second image data is obtained by performing post-processing of the remaining stage algorithm on the first image data, including:

[0014] If the camera application is running in the background or closed, and a preset trigger condition is detected, the remaining stage algorithm is performed on the first image data to obtain the second image data.

[0015] In some possible implementations of the first aspect, the preset triggering conditions include at least one of the following: opening an image access application, viewing an image, sharing an image, editing an image, the system being in an idle state, the screen being off or turned off, the running memory being higher than a first threshold, the device temperature being lower than a first temperature threshold, and the processor load being lower than a second threshold.

[0016] In this implementation, when the camera application is running in the background or closed, it further determines whether a preset trigger condition is met. Only when the preset trigger condition is met is the remaining stage algorithm post-processing executed. This allows the remaining stage algorithm post-processing to be performed promptly when the user needs to view, edit, or share the image data, enabling the user to view the final image in a timely manner. In addition, the remaining stage algorithm post-processing can be automatically triggered when the system status meets certain conditions, such as the running memory, device temperature, and processor load meeting requirements, which can further alleviate problems such as device lag and overheating.

[0017] In some possible implementations of the first aspect, the second image data is obtained by post-processing the first image data using the remaining stage algorithm, including:

[0018] Monitor system status data;

[0019] Determine the scheduling strategy based on system status data;

[0020] According to the scheduling strategy, the target task in the task queue is executed to perform the remaining stage algorithm post-processing on the first image data to obtain the second image data;

[0021] The target task is the task corresponding to the post-processing of the remaining stage algorithm of the first image data. The task queue includes the tasks corresponding to the post-processing of the remaining stage algorithm of each image data. The target task is generated based on the task identifier of the post-processing of the remaining stage algorithm of the first image data.

[0022] In this implementation, by monitoring system status data and scheduling the remaining post-processing tasks based on the system status data, problems such as device lag and overheating can be further alleviated.

[0023] In some possible implementations of the first aspect, if a viewing operation on the first image data is detected before the second image data is obtained, then in response to the viewing operation, a first target image is displayed, wherein the first target image is the image corresponding to the first image data;

[0024] After obtaining the second image data, the first target image is refreshed to the second target image, which is the image corresponding to the second image data.

[0025] In this implementation, if the first-stage post-processing algorithm is not completed but the user needs to view the image, the image data with incomplete post-processing is displayed first. Then, when the post-processing algorithm is completed, the image data with incomplete post-processing is refreshed to the image data with complete post-processing. In this way, while reducing the consumption of computing and memory resources during the shooting process and alleviating problems such as device lag and overheating, it can also ensure the timeliness of the user's image viewing.

[0026] In some possible implementations of the first aspect, after obtaining the image data to be processed, the method further includes:

[0027] Cut the image data to be processed from the volatile storage medium to the non-volatile storage medium;

[0028] The image data to be processed undergoes a first-stage algorithmic post-processing to obtain the first image data, including:

[0029] The image data to be processed is read from the non-volatile storage medium, and the image data to be processed is subjected to the first stage of algorithm post-processing to obtain the first image data, which is then stored in the non-volatile storage medium.

[0030] In this implementation, the intermediate data and image data to be processed in the post-processing of the algorithm are transferred from volatile storage media to non-volatile storage media, which can further reduce the memory usage of the post-processing process and thus further alleviate problems such as device lag.

[0031] In some possible implementations of the first aspect, before performing the first-stage algorithm post-processing on the image data to be processed to obtain the first image data, the method further includes:

[0032] If the conditions for entering segmented shooting are met, the process proceeds to the first stage of algorithmic post-processing of the image data to be processed, in order to obtain the first image data.

[0033] In some possible implementations of the first aspect, the conditions for entering segmented shooting include at least one of the following: detecting a burst shooting operation, running memory being lower than a third threshold, device temperature being higher than a second temperature threshold, and post-processing algorithm taking more than a preset time threshold.

[0034] In this implementation, when the system is in poor condition, it can actively enter segmented shooting mode, which can not only alleviate problems such as device lag and overheating, but also ensure the real-time shooting.

[0035] Secondly, embodiments of this application provide an electronic device, including an algorithm post-processing module and an image acquisition module;

[0036] The image acquisition module is used to: acquire image data to be processed in response to a photo-taking operation;

[0037] The algorithm post-processing module is used to: when the camera application is running in the foreground, perform a first stage of algorithm post-processing on the image data to be processed to obtain first image data, which is the image data incomplete in the algorithm post-processing; if a target operation is detected, or the camera application is running in the background or closed, perform a remaining stage of algorithm post-processing on the first image data to obtain second image data, which is the image data after algorithm post-processing. The target operation is used to view, edit, or share the image. The remaining stage of algorithm post-processing includes at least one stage of algorithm post-processing.

[0038] In some possible implementations of the second aspect, the target operation is to view a thumbnail, which is displayed in the viewfinder of a camera application. The thumbnail is a thumbnail corresponding to the image data to be processed.

[0039] In some possible implementations of the second aspect, the algorithm post-processing module is specifically used for:

[0040] If the camera application is running in the background or closed, and a preset trigger condition is detected, the remaining stage algorithm is performed on the first image data to obtain the second image data.

[0041] In some possible implementations of the second aspect, the preset triggering conditions include at least one of the following: opening an image access application, viewing an image, sharing an image, editing an image, the system being in an idle state, the screen being off or turned off, the running memory being higher than a first threshold, the device temperature being lower than a first temperature threshold, and the processor load being lower than a second threshold.

[0042] In some possible implementations of the second aspect, the algorithm post-processing module includes a system state awareness submodule and a scheduling submodule;

[0043] The system status awareness submodule is used to monitor system status data;

[0044] The scheduling submodule is used to determine the scheduling strategy based on system status data; according to the scheduling strategy, it executes the target task in the task queue to perform the remaining stage algorithm post-processing on the first image data to obtain the second image data.

[0045] The target task is the task corresponding to the post-processing of the remaining stage algorithm of the first image data. The task queue includes the tasks corresponding to the post-processing of the remaining stage algorithm of each image data. The target task is generated based on the task identifier of the post-processing of the remaining stage algorithm of the first image data.

[0046] In some possible implementations of the second aspect, a refresh module is also included, configured to: detect a viewing operation on the first image data; in response to the viewing operation, display a first target image, the first target image being the image corresponding to the first image data; and after obtaining the second image data, refresh the first target image to the second target image, the second target image being the image corresponding to the second image data.

[0047] Among some possible implementations of the second aspect, non-volatile storage media are also included; the algorithm post-processing module is also used for:

[0048] Cut the image data to be processed from the volatile storage medium to the non-volatile storage medium;

[0049] The image data to be processed is read from the non-volatile storage medium, and the image data to be processed is subjected to the first stage of algorithm post-processing to obtain the first image data, which is then stored in the non-volatile storage medium.

[0050] In some possible implementations of the second aspect, the electronic device further includes a segmented image capture trigger module, which, if it is determined that the conditions for entering segmented image capture are met, proceeds to the step of performing a first-stage algorithmic post-processing on the image data to be processed to obtain the first image data.

[0051] In some possible implementations of the second aspect, the conditions for entering segmented shooting include at least one of the following: detecting continuous shooting operation, running memory being lower than a third threshold, device temperature being higher than a second temperature threshold, and post-processing algorithm taking more than a preset time threshold.

[0052] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects above.

[0053] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0054] Fifthly, embodiments of this application provide a chip system including a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the method described in any of the first aspects above. The chip system may be a single chip or a chip module composed of multiple chips.

[0055] Sixthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method described in the first aspect above.

[0056] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the image processing process provided in the embodiments of this application;

[0058] Figure 2A This is a schematic diagram of the mobile phone photography process provided in an embodiment of this application;

[0059] Figure 2B This is a schematic diagram of the imaging system framework provided in an embodiment of this application;

[0060] Figure 3 This is a schematic diagram illustrating the division of the algorithm post-processing stage in an embodiment of this application;

[0061] Figure 4 This is a schematic diagram of the image processing process provided in the embodiments of this application;

[0062] Figure 5 This is a schematic diagram of the segmented photography system architecture provided in an embodiment of this application;

[0063] Figure 6A This is a flowchart illustrating the first-stage algorithm post-processing provided in an embodiment of this application.

[0064] Figure 6B A schematic block diagram of the system architecture provided in the embodiments of this application;

[0065] Figure 7A A schematic diagram of a viewing image provided in an embodiment of this application;

[0066] Figure 7B Another schematic diagram of the viewing image provided in the embodiments of this application;

[0067] Figure 8 A schematic flowchart illustrating the on-demand triggering of the second-stage algorithm post-processing provided in this application embodiment;

[0068] Figure 9 This is a schematic diagram of the process for automatically triggering the second-stage algorithm post-processing in an embodiment of this application.

[0069] Figure 10 A schematic diagram of the scheduling strategy for the second-stage algorithm post-processing provided in the embodiments of this application;

[0070] Figure 11 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application;

[0071] Figure 12A schematic flowchart of a photo processing method provided in an embodiment of this application;

[0072] Figure 13 A schematic block diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0073] In the following description, specific details such as particular system architectures and technologies are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application.

[0074] Post-processing can refer to the process of enhancing the image effects of image data output by camera hardware using post-processing algorithms.

[0075] Post-processing algorithms are software algorithms used to improve image effects and image quality. For example, post-processing algorithms may include, but are not limited to: beautification algorithms, filter algorithms, rotation algorithms, watermarking algorithms, blurring algorithms, high dynamic range imaging (HDR) algorithms, multi-frame processing algorithms, noise reduction algorithms, and detail enhancement algorithms.

[0076] During algorithm post-processing, one or more post-processing algorithms are executed. For example, see... Figure 1 The diagram shown illustrates the image processing process, which may include an image acquisition stage and an algorithm post-processing stage.

[0077] During the image acquisition phase, after the image sensor acquires image signal data, this data is transmitted to the Image Signal Processor (ISP). The image data output by the ISP is stored in a volatile storage medium, such as Random Access Memory (RAM). Furthermore, thumbnails can be obtained from the image data output by the ISP.

[0078] In the post-processing stage, post-processing algorithm one, post-processing algorithm two, and post-processing algorithm three are executed sequentially. Specifically, image data output from the ISP is read from RAM, and post-processing algorithm one is used to process the image data output from the ISP to obtain a first processing result, which is then stored in RAM. The first processing result is read from RAM, and post-processing algorithm two is used to process the first processing result to obtain a second processing result, which is then stored in RAM. The second processing result is read from RAM, and post-processing algorithm three is used to process the second processing result to obtain a third processing result, which is then stored in RAM. The third processing result is read from RAM, and the final image is obtained based on the third processing result. It is understood that, for ease of description, Figure 1 Taking the post-processing stage of the algorithm, which includes three post-processing algorithms, as an example.

[0079] In related technologies, electronic devices continuously execute all post-processing algorithms during the photo-taking process. The following section combines... Figure 2A and Figure 2B To provide an introduction and explanation. Figure 2A This is a diagram illustrating the process of taking a photo with a mobile phone. Figure 2B This is a schematic diagram of the camera system framework.

[0080] like Figure 2A As shown, the main interface of mobile phone 11 displays applications such as Settings, Calendar, Camera 111, and Gallery. When the user taps the Camera application 111, mobile phone 11 responds to the tap, opening the Camera application 111 and displaying its viewfinder 112. The viewfinder 112 includes a viewfinder frame and a shutter control 113, with a preview video stream displayed within the viewfinder frame. When the user taps the shutter control 113, mobile phone 11 responds to the tap, performing a shutter action, acquiring the image data to be processed, and performing real-time post-processing algorithms on the image data to obtain the final image. The final image is the image after algorithm post-processing. From acquiring the image data to obtaining the final image, the Camera application 111 runs in the foreground, continuously executing all post-processing algorithms during the shutter action to perform real-time post-processing on the image data and obtain the post-processed image in real-time.

[0081] like Figure 2BAs shown, the camera system framework of the iPhone 11 includes an application layer, a middleware layer, and a functional module layer. The application layer includes a camera application and an image access application, such as a gallery application or an instant messaging application. The middleware layer includes camera services and a media library. The camera service includes a real-time shooting sub-service, which includes camera devices and camera streams. The real-time shooting sub-service is used to implement the shooting function. The media library includes a Media Store database interface and a Media Provider-DB (Media Provider database). The functional module layer includes an acquisition module, an algorithm post-processing module, and a file system. The acquisition module is used to acquire image data. The algorithm post-processing module includes sub-modules such as an image buffer, algorithm processing, and algorithm scheduling management, used to perform algorithm post-processing on the image data. The file system is used to store the final image, which is typically stored in a non-volatile storage medium, such as read-only memory (ROM).

[0082] based on Figure 2B The illustrated camera system framework works as follows: when the user presses the camera control 113, the camera application on the phone 11 sends a photo request to the camera service. In response, the camera service calls the acquisition module to acquire the image data, obtains the image data to be processed, and stores it in the image buffer. The image buffer is then processed in real-time using algorithms. The image buffer resides in the RAM of the phone 11.

[0083] In the post-processing stage, the algorithm scheduling and management submodule schedules and manages the various post-processing algorithms within the algorithm processing submodule. This ensures that the algorithm processing submodule reads the image data to be processed from the image buffer and sequentially executes multiple algorithm processes (pro) based on the image data to obtain the final image in real time. The final image is then returned to the camera application via the camera service. After obtaining the final image in real time, it is also updated in the image library. The image access application can access the final image in the file system through the media library.

[0084] During their long-term research, the inventors discovered that continuously executing all post-processing algorithms during the photo-taking process to obtain the post-processed image in real time can meet the real-time requirements of photo taking, but it will reduce photo-taking performance and affect the user's photo-taking experience.

[0085] Specifically, during the photo-taking process, the camera application running in the foreground, processing the preview video stream, and performing complex post-processing algorithms all require significant computing and memory resources. The processing of the preview video stream, the operation of the camera application in the foreground, and image rendering already consume substantial computing and memory resources. If all post-processing algorithms are then executed continuously and uninterruptedly during the photo-taking process, the computational load increases further, exacerbating the consumption of computing and memory resources, leading to excessive processor load and high memory usage. Furthermore, the simultaneous operation of the camera application in the foreground, processing of the preview video stream, and performing complex post-processing algorithms can result in competition for computing and memory resources, further increasing system pressure.

[0086] Excessive processor load and RAM usage will significantly slow down the shooting speed, causing stuttering and even unresponsiveness in some scenarios (such as rapid burst shooting), resulting in poor image quality. Furthermore, prolonged high processor load during shooting will cause significant overheating. Overheating triggers temperature control strategies. These strategies limit processor usage to reduce temperature, such as lowering the clock speed and reducing the usage time of large cores. These processor limitations reduce processing power, further degrading image quality.

[0087] To address the aforementioned issues, this application provides a segmented shooting solution. By executing post-processing algorithms in segments, it reduces processor load and memory usage during the shooting process, thereby alleviating issues such as lag and device overheating, improving shooting performance, and enhancing the user's shooting experience.

[0088] Unlike methods that continuously execute all post-processing algorithms during the photo-taking process to obtain the post-processed image data in real time, the segmented photo-taking scheme of this application divides the post-processing stage into at least two stages. The first stage of post-processing is executed during the photo-taking process to obtain image data that is not yet fully processed. Then, at an appropriate time, the remaining stages of post-processing are performed on the incomplete image data to obtain the fully processed image data. This avoids continuously executing all post-processing algorithms during the photo-taking process. An appropriate time could be, for example, when a user or application needs to access, view, share, or edit image data, the system is idle, or the camera application is closed or running in the background, and system status data (such as processor load, RAM, and device temperature) are below preset values.

[0089] When dividing the post-processing stage of the algorithm into multiple stages, the position and number of segmentation points can be set according to actual needs. The number of segmentation points can determine the number of stages.

[0090] For example, see Figure 3 The diagram illustrates the post-processing stage of the algorithm, which may include the following processes: a denoising module, a brightness and color module, a postprocessing module, and a compression module. The denoising module removes noise and enhances image details, the brightness and color module processes the color and brightness of the image data, and the postprocessing module enhances image details or performs skin smoothing. It is understood that each module may include one or more post-processing algorithms; for example, the denoising module may include a denoising algorithm, and the postprocessing module may include a skin smoothing algorithm. The denoising module and the brightness and color module belong to RAW or RGB image processing, while the postprocessing module and the compression module belong to YUV domain processing.

[0091] like Figure 3 As shown, the input to the post-processing stage of the algorithm is a RAW image, and the output of the post-processing stage of the algorithm is a Joint Photographic Experts Group (JPEG) image, which is the image after the algorithm has completed post-processing.

[0092] In the post-processing stage of the algorithm, the RAW image can be segmented during the original input stage, such as... Figure 3 The segmentation point 1 is shown in the image; alternatively, the RAW image can be partially processed using post-processing algorithms before segmentation, as shown in the image below. Figure 3 The segmentation point 2 is shown in the figure.

[0093] Understandably, the location of the segmentation point is used to divide the post-processing of the algorithm into different stages. For example, when the segmentation point is as follows... Figure 3 At segment point 2, the first-stage post-processing of the algorithm can include a denoising module and a brightness and color module, while the second-stage post-processing of the algorithm can include a Postprocess module and a Compression module.

[0094] Typically, the post-processing stage of a completed algorithm can be divided into a first-stage post-processing process and a second-stage post-processing process. In practical applications, the post-processing stage can also be divided into three, four, or more stages, which is not limited here.

[0095] For example, see Figure 4The illustrated image processing diagram includes an image acquisition stage and an algorithm post-processing stage. The algorithm post-processing stage comprises a first-stage algorithm post-processing and a second-stage algorithm post-processing; that is, the complete algorithm post-processing stage is divided into two stages. The first-stage algorithm post-processing is performed on the image data output by the ISP, resulting in an incomplete image. This incomplete image is used as the input for the second-stage algorithm post-processing.

[0096] It is understandable that the first and second stages of algorithm post-processing can be further divided into multiple sub-stages. For example, the second stage of algorithm post-processing can be divided into a third and a fourth stage. For ease of description, the following explanation will use the division into the first and second stages of algorithm post-processing as an example.

[0097] It should be pointed out that, Figure 4 In addition to dividing the complete algorithm post-processing stage into the first stage and the second stage, the intermediate data of the post-processing algorithm is also stored in the ROM. This can reduce the memory usage during the shooting process and further alleviate the device lag problem.

[0098] Specifically, to ensure image quality, the post-processing stage often employs multi-frame RAW domain algorithms. These algorithms require caching multiple raw images to synthesize a single, high-quality image. For example, a 12.5MB image requires six raw images, consuming 150MB of RAM. Furthermore, the multi-frame RAW domain algorithm itself also consumes memory. If six photos are taken consecutively, this would accumulate to 1GB of RAM, placing significant pressure on the system and causing noticeable lag on the user's device. In addition, excessive RAM usage can lead to process termination and image loss. In other words, storing intermediate data from post-processing algorithms in RAM buffers results in a large amount of intermediate data consuming RAM, causing device lag and other problems.

[0099] In this embodiment, the intermediate data of the post-processing stage of the algorithm is saved to the ROM flash. The capacity of the ROM flash is usually large, reaching more than 128G or even 1T. Therefore, the ROM can store enough image data without causing RAM buff blockage, reducing the memory usage during the shooting process and alleviating problems such as device lag.

[0100] Specifically Figure 4 In this process, the image output by the ISP is cached in RAM by default. If segmented shooting is entered, the image data in RAM is cut and moved to ROM, that is, the image data in RAM is transferred to ROM; if segmented shooting is not entered, it can be processed according to... Figure 1 The process shown is followed.

[0101] After entering the segmented shooting mode, the post-processing algorithms in the first and second stage algorithms all obtain image data from the ROM, process the image data, obtain intermediate data, and store the intermediate data in the ROM.

[0102] It is understandable that post-processing algorithms can perform a series of post-processing algorithms on image data to obtain images with better quality and effects. Therefore, the image after post-processing (i.e.,...) Figure 4 The final image (in the algorithm) is usually of better quality and effect than the image before post-processing. For example, suppose the image before post-processing is a base quality image of 80 points, while the image after post-processing is a full quality image of 100 points.

[0103] To better illustrate the segmented photography scheme provided in the embodiments of this application, the following will use... Figure 5 The following is an example of a segmented photography system architecture diagram.

[0104] like Figure 5 As shown, the segmented photography system architecture includes an application layer, a framework and service layer, and a hardware abstraction layer (HAL).

[0105] The application layer includes camera apps and image access apps. Camera apps enable photo taking and image viewing. Image access apps can access images in the file system through the media library.

[0106] The framework and service layer include a camera service and a media library. The camera service includes a real-time capture sub-service and a time-lapse capture sub-service. The time-lapse capture sub-service is a newly added module used to control the second-stage post-processing algorithm. Specifically, it can schedule tasks for the second-stage post-processing algorithm and, after determining the scheduling strategy based on the system state, execute the second-stage post-processing algorithm using the determined strategy. The media library, in addition to the database interface and database, also includes an interception and trigger module. This module detects trigger operations and, based on these operations, triggers the time-lapse capture sub-service to schedule tasks for the second-stage post-processing algorithm to obtain a 100-point full-quality image. Trigger operations can be, for example, a user clicking a thumbnail or a user viewing an image. When the interception and trigger module detects a user trigger operation, it triggers the time-lapse capture sub-service to prioritize the second-stage post-processing algorithm for the corresponding image to obtain the 100-point full-quality image corresponding to the image the user wants to view.

[0107] HAL comprises the camera HAL, post-processing services, and a file system. The camera HAL includes an acquisition module for acquiring image data. The post-processing services include sub-modules such as algorithm scheduling management, stage one algorithm processing, and stage two algorithm processing. Algorithm scheduling management manages the scheduling of each post-processing algorithm. Both stage one algorithm processing (i.e., first-stage algorithm post-processing) and stage two algorithm processing (second-stage algorithm post-processing) include multiple algorithm procedures (proc). The file system stores 80-point baseline quality images and 100-point full-quality images.

[0108] Figure 5 The segmented image processing system architecture shown divides the post-processing stage into two phases: Phase 1 and Phase 2. Based on these two phases, the image processing process is further divided into Phase 1 and Phase 2. Phase 1 generates a baseline quality image (80 points), and Phase 2 generates a full quality image (100 points).

[0109] based on Figure 5 The segmented photography system architecture, combined with the following... Figure 6A The flowchart of the first-stage algorithm post-processing is shown, and the process of the first-stage algorithm post-processing is described in detail.

[0110] like Figure 6A As shown, the process may include the following steps:

[0111] Step S601: The camera application sends a photo request to the camera HAL through the real-time photo capture sub-service.

[0112] For example, a user can trigger the camera app to generate a photo-taking request by inputting a photo-taking action into the camera app. See, for example, [link to relevant documentation]. Figure 2A In the illustrated photo-taking scenario, the user can input a photo-taking action by clicking the photo-taking control 113. In response to the photo-taking action, the camera application sends a photo-taking request to the real-time photo-taking sub-service, which then transmits the request to the camera HAL.

[0113] Step S602: The camera HAL responds to the photo capture request by acquiring image data and obtaining the image data to be processed.

[0114] Understandably, the camera's HAL can respond to a photo capture request by calling the image sensor and ISP to acquire image data, and obtain the image data to be processed output by the ISP. In segmented shooting, the image data to be processed output by the ISP is stored in the RAM buffer, and then the image data to be processed in the RAM buffer is cut and pasted into the ROM.

[0115] The image data to be processed can be Raw image data formed from RGB or RYB data signals. During image data acquisition, in addition to the image data itself, metadata and auxiliary device information can also be obtained. Metadata includes, but is not limited to: 3A (Auto Exposure Control AE, Auto Focus Control AF, Auto White Balance AWB) information, image information (e.g., image width and height), and exposure parameters (aperture size, shutter speed, and ISO / Aperture value). Auxiliary device information includes, but is not limited to: laser information and spectral information. The metadata and auxiliary device information can be used in subsequent post-processing algorithms.

[0116] In some embodiments, after receiving a photo capture request or obtaining image data to be processed, the camera HAL can further determine whether to enter a segmented photo capture architecture. If it is determined that a segmented photo capture architecture should be entered, the subsequent segmented photo capture steps are then performed. If it is determined that a segmented photo capture architecture should not be entered, the subsequent process can adopt the following... Figure 2B The illustrated image capture scheme eliminates the need to divide the post-processing stage of the algorithm into multiple stages. Of course, in other embodiments, the camera HAL can also default to a segmented image capture architecture.

[0117] For example, see Figure 6B The illustrated system architecture block diagram shows that when the conditions for entering the segmented image capture architecture are not met, real-time post-processing algorithms are performed. This means that all post-processing algorithms are executed continuously and uninterruptedly during the image capture process, and the acquired images are updated to the image library in real time. When the conditions for entering the segmented image capture architecture are met, the data in the RAM buffer is cut and pasted into the ROM flash, and segmented post-processing algorithms are performed, updating the acquired images to the image library.

[0118] In practical applications, the system state can be used to determine whether to enter a segmented image capture architecture. System state can include, but is not limited to, the amount of data in the RAM buffer, device temperature, and post-processing algorithm latency. If the amount of data in the RAM buffer exceeds a preset threshold (i.e., the RAM usage is too high), then the system is considered ready to enter a segmented image capture architecture. If the device temperature exceeds a preset temperature threshold, then the device is considered to be overheating, and the system has already limited processor usage according to temperature control strategies; similarly, if the post-processing algorithm latency exceeds a preset time threshold, then the system is considered ready to enter a segmented image capture architecture.

[0119] Step S603: The camera HAL obtains a fast thumbnail based on the image data to be processed.

[0120] Step S604: The camera HAL returns the quick thumbnail to the camera application, which saves the quick thumbnail and inserts the thumbnail record into the media library.

[0121] Step S605: The post-processing service performs the first-stage algorithm post-processing on the image data to be processed, obtains 80-part images, and saves the 80-part images.

[0122] It is understandable that the 80-point image refers to the result of the first stage of algorithm post-processing. It is image data that has not been fully processed by the algorithm. The use of 80-point image here is just for the convenience of description.

[0123] Step S606: The post-processing service returns the 80-segment image and segment metadata to the camera application.

[0124] Segmented metadata may include the task identity document (ID) from the second-stage algorithm post-processing. In addition, segmented metadata may also include all or part of the data from the aforementioned image metadata and assistive device information.

[0125] Step S607: The camera application saves 80 segments of the image and inserts the records and segment metadata of the 80 segments into the media library.

[0126] Step S608: The media library constructs a new task for the second-stage algorithm post-processing based on the segmented metadata.

[0127] Step S609: The media library will notify the time-lapse photography sub-service of the newly constructed task so that the time-lapse photography sub-service can update the task queue according to the new task.

[0128] Understandably, the task queue includes one or more tasks. Each task is a post-processing task for the second-stage algorithm corresponding to an 80-part image. The second-stage post-processing of each 80-part image is performed by executing the tasks in the task queue. In practical applications, tasks in the task queue can be processed sequentially in a first-in, first-out (FIFO) order to execute the second-stage post-processing of each 80-part image in an orderly manner. Alternatively, a specific task in the task queue can be prioritized for execution, allowing for the timely acquisition of a particular post-processing task. For example, if a user needs to view an image, but the task corresponding to that image has not yet been executed, then that task will be executed first to ensure the image is processed and ready for immediate access.

[0129] Multiple tasks may exist simultaneously in the task queue. For example, if a user takes five photos with their phone within a short period, the phone will execute a command for each photo request. Figure 6AThe steps in the algorithm yielded five 80-point images. At this point, assuming the second-stage post-processing algorithm has not yet been triggered, the task queue simultaneously contains the second-stage post-processing tasks corresponding to these five 80-point images.

[0130] It can be seen that in the segmented shooting scheme, only the first stage of post-processing algorithm is executed during the shooting process, that is, only part of the post-processing algorithm is executed, rather than all of the post-processing algorithm. This reduces the amount of computation in the shooting process, thereby reducing the processor load and memory usage in the shooting process.

[0131] The first stage of the algorithm post-processing is executed during the photo-taking process, during which the camera application typically runs in the foreground. After the first stage of algorithm post-processing is completed, the user or the system automatically triggers the execution of the second stage of algorithm post-processing to obtain the post-processed image.

[0132] Users can trigger the execution of the second-stage algorithm post-processing process by viewing or sharing images.

[0133] For example, see Figure 7A The illustration shown is a schematic diagram of viewing an image, based on Figure 1 In the illustrated photo-taking scenario, the mobile phone 11 responds to the user's click operation on the camera control 113 by executing the above... Figure 6A The process shown yields an 80-point image and a thumbnail, which can be displayed as follows: Figure 7A The thumbnail 114 is shown. When the user clicks on the thumbnail 114, the mobile phone 11 considers that the user has viewed the image, triggers the second-stage algorithm post-processing process, and displays the interface 115, which shows the image 116. The image 116 can be an 80-part image or a 100-part image (i.e., the image obtained from the second-stage algorithm processing).

[0134] It should be noted that the execution of the second-stage algorithm post-processing takes a certain amount of time. Therefore, if the second-stage algorithm post-processing corresponding to thumbnail 114 is not triggered in advance, the mobile phone 11 may not be able to obtain the 100-point image in time after the user clicks on thumbnail 114, that is, it may not be able to obtain the image after the algorithm post-processing is completed in time. Therefore, the 80-point image (the result of the first-stage algorithm post-processing) can be displayed on the interface 115 first, and the 80-point image on the interface 115 can be refreshed to the 100-point image after the second-stage algorithm post-processing is completed.

[0135] Of course, in some other embodiments, after the user clicks on thumbnail 114, the phone 11 does not immediately display the 80-point image on interface 115. Instead, it waits for the second-stage algorithm post-processing to complete before displaying the 100-point image on interface 115. In this case, after the user clicks on thumbnail 114, they may need to wait a certain amount of time before seeing image 116. At this time, image 116 is the 100-point image.

[0136] exist Figure 7A In the scenario shown, when the user clicks on thumbnail 114, the phone 11 exits the camera shooting process and enters image preview mode. In image preview mode, no preview video stream processing is performed. This means that during the second-stage algorithm post-processing, preview video stream processing is not performed simultaneously, resulting in lower processor load and memory usage.

[0137] For example, see also Figure 7B Another schematic diagram of viewing the image is shown, based on Figure 1 In the illustrated photo-taking scenario, after the user takes a photo by clicking the camera control 113, the camera application 111 is exited. During this process, the phone 11 responds to the user's click on the camera control 113 by executing the aforementioned... Figure 6A The process shown obtains an 80-part image and a thumbnail, which means that the first stage of algorithm post-processing has been performed and the result of the first stage algorithm post-processing has been obtained.

[0138] After the user exits the camera app 111, they tap the gallery 117 to view the photos taken. In response to this tap, the phone 11 displays the album interface 118. The user then taps the control 119 in the album interface 118, and the phone displays the photo page 121, which shows images 122, 123, 124, 125, 126, and 127. If the user wants to view image 125, the phone 11 displays interface 128 in response to a tap on image 125. It's understandable that images 122 through 127 displayed on photo page 121 could be the result of the first-stage algorithm post-processing (i.e., 80-point images) or thumbnails.

[0139] At this point, when phone 11 detects a click on image 125, it considers this a user viewing action and triggers the second-stage post-processing algorithm corresponding to image 125. Similarly, after the user clicks on image 125, phone 11 can immediately display an 80-point image of image 125 on interface 128. After the second-stage post-processing algorithm is completed, the 80-point image on interface 128 will be refreshed to a 100-point image. Of course, phone 11 can also choose not to immediately display the 80-point image on interface 128, but instead wait for the second-stage post-processing algorithm to complete before displaying the 100-point image on interface 128.

[0140] exist Figure 7B In the scenario, when the phone 11 detects that the user exits the camera application or clicks on the gallery 117 (i.e., opens the image access application), it can also trigger the second-stage algorithm post-processing.

[0141] exist Figure 7A and Figure 7B In this scenario, when a user clicks on thumbnail 114 or image 125, mobile phone 11 can prioritize triggering the corresponding task to promptly execute the second-stage algorithm post-processing of thumbnail 114 or image 125, and obtain the post-processed image in a timely manner. In other words, the segmented photo-taking scheme of this application embodiment can prioritize triggering the second-stage algorithm post-processing of a specified image according to actual business needs. The following will combine... Figure 8 This paper introduces and explains the relevant process of the second-stage algorithm post-processing that prioritizes the triggering of a specified image.

[0142] See Figure 8 The illustrated flowchart shows the on-demand triggering of the second-stage algorithm post-processing, which may include the following steps:

[0143] Step S801: In response to the viewing operation of the target image, the gallery application reads 80% of the target image from the file system, displays the 80% image immediately, and synchronously requests the 100% image of the target image from the media library.

[0144] For example, in Figure 7B In the scenario shown, when the user clicks on image 125, phone 11 recognizes that a viewing operation has been detected for the target image, which is image 125. At this time, the gallery application, based on the relevant information of image 125, reads 80% of image 125 from the file system and immediately displays the 80% of image 125 on screen 128. In addition, the gallery application also simultaneously requests 100% of image 125 from the media library based on the relevant information of image 125.

[0145] Understandably, the media library contains information about the target image. For example, see... Figure 6AThe first-stage algorithm post-processing process shown in the diagram involves the camera application inserting 80 segmented image records and segmented metadata into the media library.

[0146] Step S802: In response to the 100-part image request from the gallery application, the media library triggers the delayed photo capture sub-service to schedule and execute the second-stage algorithm post-processing task for the target image.

[0147] Understandably, a 100-segment image request from a photo library application carries information about an 80-segment image. The media library can use this information to find the corresponding segment metadata for the 80-segment image, and then trigger the delayed image capture sub-service to execute the second-stage post-processing task corresponding to the 80-segment image based on the segment metadata. For example, the segment metadata includes a task ID; the media library can trigger the delayed image capture sub-service to execute the task corresponding to that task ID based on the task ID.

[0148] Step S803: The time-delayed photography sub-service triggers the post-processing service to prioritize the execution of the second-stage algorithm post-processing task for the target image based on the trigger from the media library.

[0149] Understandably, when a user views a target image, if the second-stage post-processing algorithm for the target image has not yet been executed, the second-stage post-processing algorithm for the target image will be scheduled and triggered first; if the task of the second-stage post-processing algorithm for the target image has been completed, the 100-point image of the target image can be read directly and displayed immediately.

[0150] For example, suppose in Figure 7B In the scenario described, during the photo-taking process, the user consecutively took photos of images 122, 123, 124, 125, and 126. After taking the photos, the user exited the camera application. During the photo-taking process, the phone performed a first-stage post-processing algorithm for each image, obtaining 80 sub-images for each image, and constructing a second-stage post-processing task for each image. At this point, the task queue contains the second-stage post-processing tasks for images 122, 123, 124, 125, and 126. Assuming the image-taking order is: images 122, 123, 124, 125, 126, the execution order of the second-stage post-processing tasks for each image in the task queue is: images 122, 123, 124, 125, 126.

[0151] After the user exits the camera application, if the phone detects that the system is idle at a certain time, or if the user opens the gallery application 117, it will automatically trigger the execution of various tasks in the task queue. Normally, the tasks in the task queue are executed in a first-in, first-out (FIFO) order; that is, the second-stage algorithm post-processing task for image 122 is executed first, and the second-stage algorithm post-processing task for image 126 is executed last.

[0152] When the mobile phone 11 finishes the second-stage algorithm post-processing task for image 123, it detects the user's click operation on image 125, that is, the user needs to browse image 125 in a large size at the current moment. Therefore, the second-stage algorithm post-processing task for image 125 is triggered first, instead of triggering the second-stage algorithm processing task for image 124 in the order of priority.

[0153] Of course, if the user clicks on image 125, and the second-stage algorithm post-processing has not yet been triggered on phone 11, then phone 11 will trigger the second-stage algorithm post-processing and prioritize the execution of the second-stage algorithm post-processing task for image 125.

[0154] Step S804: The post-processing service prioritizes the second-stage algorithm post-processing task of the target image to obtain a 100-point image of the target image.

[0155] Step S805: The post-processing service saves 100 images to the file system.

[0156] Step S806: The time-delayed photo capture sub-service replaces the 80-point image of the target image with a 100-point image.

[0157] After obtaining the 100-point image from the post-processing service, the time-lapse photography sub-service first queries the media library for the file records of the 80-point images. Based on these 80-point image file records, it replaces the 80-point image in the file system with the 100-point image and updates the 100-point image file records in the media library. In this way, the media library contains the 100-point image file records, which the gallery application can use to read the 100-point image of the target image from the file system.

[0158] Step S807: The delayed photo-taking sub-service notifies the gallery application to refresh the image.

[0159] Step S808: The gallery application reads the 100-point image of the target image from the file system and refreshes the 80-point image to the 100-point image.

[0160] It's important to note that when a user needs to access a specific image, the 80-point image is displayed immediately. Then, according to the user's needs, the second-stage algorithm post-processing of the specified image is triggered first. After the second-stage algorithm post-processing is completed, the 80-point image is refreshed to the 100-point image. In this way, the segmented shooting solution improves shooting performance while still meeting the user's need to view images in real time, ensuring a good user experience.

[0161] Understandable, Figure 7A The scenario also fits the on-demand access scenario. That is, when the user clicks on the thumbnail 114, the mobile phone 11 can also prioritize scheduling and triggering the second-stage algorithm post-processing task corresponding to the thumbnail 114, and immediately display the 80-point image. After obtaining the result of the second-stage algorithm post-processing, the 80-point image is refreshed to the 100-point image.

[0162] In addition to being able to trigger the second-stage algorithm post-processing through specific user operations, the system can also automatically trigger the execution of the second-stage algorithm post-processing based on system status data.

[0163] For example, system status data includes system processor load, device temperature, and RAM. When a user exits the camera application and at least one of the following conditions is met: processor load is below a preset load threshold, device temperature is below a preset temperature threshold, and RAM is above a preset value, the execution of the second-stage algorithm post-processing is automatically triggered, and the tasks of each second-stage algorithm post-processing task in the task queue are executed cyclically.

[0164] For example, when it is detected that a user exits the camera app or opens an image access app (such as clicking on a gallery app), the execution of the second-stage algorithm post-processing is automatically triggered.

[0165] After triggering the second-stage algorithm post-processing, a scheduling strategy can be determined based on the system status data, and the tasks of the second-stage algorithm post-processing can be executed according to the scheduling strategy.

[0166] At this point, the segmented photography system can include a system perception module and a state control module. The system perception module can monitor the status of the image library, camera, AMS, battery power consumption, and system resources, and determine the current state of the system based on the monitored data. The state control module can execute corresponding scheduling strategies according to the current state of the system to implement different resource allocation strategies in different states.

[0167] For example, see Figure 9 The diagram shown illustrates the process by which the system automatically triggers the second-stage algorithm post-processing. This process may include the following steps:

[0168] Step S901: The delayed photo capture sub-service registers system status monitoring with iAware.

[0169] It is understood that iAware is a user-aware system status, and in some other embodiments, iAware can be replaced with other system awareness modules, which is not limited here.

[0170] Step S902: iAware sends a system status notification to the time-lapse photography sub-service.

[0171] System status notifications include system status data sensed by iAware. For example, system status data that iAware can sense includes: device temperature and Central Processing Unit (CPU) load.

[0172] Step S903: The delayed photo-taking sub-service determines the scheduling strategy based on the system status data.

[0173] It is understandable that system status data can include not only data perceived by iAware, but also system status data perceived by other system awareness modules. For example, see... Figure 10 The diagram shows the scheduling strategy for the second-stage algorithm post-processing. The system status data perceived by the system may also include whether the gallery, camera, and AMS are running in the foreground, as well as screen on / off data.

[0174] The time-delayed photo capture sub-server can determine the system status based on system status data, and determine the scheduling strategy based on the correspondence between the system status and the scheduling strategy.

[0175] For example, the system states include high-performance mode, balanced mode, low-power mode, and emergency mode. The correspondence between system states and resource provisioning strategies (i.e., scheduling strategies) can be shown in Tables 1 and 2 below.

[0176] Table 1

[0177]

[0178] Table 2

[0179]

[0180] Tables 1 and 2 above provide examples of the scenarios and resource provisioning strategies corresponding to various modes. For instance, when the gallery application is in the process of browsing large images or sharing pictures, the resource provisioning strategy corresponding to the high-performance mode is used to execute the second-stage algorithm post-processing task. Similarly, after the camera application exits, the various second-stage algorithm post-processing tasks can be processed in the background, and the resource provisioning strategy corresponding to the balanced mode is used to execute each task. When a processing exception occurs, the second-stage algorithm post-processing can be skipped, and the result of the first-stage algorithm post-processing (i.e., the 80-point image) can be displayed instead. For example, if RAW data is lost or the gallery access times out, the 80-point image is displayed directly without executing the second-stage algorithm post-processing.

[0181] In low-power mode, if the system predicts that an image will be viewed by the user after a certain period of time, and the timeliness and rate requirements of the second-stage post-processing algorithm for that image are low, a resource allocation strategy aimed at saving power is adopted to execute the second-stage post-processing algorithm for that image.

[0182] Understandably, the system awareness module can monitor whether applications such as the gallery and camera are running in the foreground, closed, or in large image browsing or sharing mode.

[0183] like Figure 10 As shown, high-performance mode, low-power mode, and balanced mode can be switched between each other. For example, based on the system status data perceived by the current system, the resource allocation strategy corresponding to high-performance mode is determined to execute the task; if an anomaly occurs during high-performance execution, it can switch from high-performance mode to emergency mode, or it can switch to balanced mode and low-power mode.

[0184] Step S904: The delayed photo capture sub-service processes tasks in the task queue cyclically according to the scheduling policy. After determining the scheduling policy, each task in the task queue is processed using the scheduling policy. For example, when the scheduling policy is determined to be the resource allocation policy corresponding to the high-performance mode, when executing tasks after the second-stage algorithm, the CPU is in a state of locking large cores + boost + background protection, the NPU has high priority, and the GPU has medium priority.

[0185] like Figure 9As shown, for each task in the task queue, the following process is executed: The delayed photo capture sub-service schedules the task, triggering the post-processing service to perform the second-stage algorithm post-processing to obtain a 100-point image; after obtaining the 100-point image, the post-processing service first queries the file records of the 80-point images in the media library, and based on the file records of the 80-point images, replaces the 80-point images in the file system with the 100-point images, and updates the file records of the 100-point images in the media library. During this process, resources are allocated according to the determined scheduling strategy.

[0186] As mentioned above, the system can automatically trigger the execution of the second-stage algorithm post-processing based on the perceived system state. For example, if the system detects that a user clicks on the gallery or exits the camera application, detects that the system is in an idle state, or detects that at least one of the following conditions—CPU load, RAM, and device temperature—meets a predetermined condition after the user exits the camera application, then the execution of the second-stage algorithm post-processing is triggered.

[0187] Furthermore, after determining that the second-stage algorithm post-processing has been triggered, the system can further determine a scheduling strategy based on system state data, and execute the second-stage algorithm post-processing task according to the determined scheduling strategy. Alternatively, instead of determining a scheduling strategy based on system state data, a pre-defined resource supply strategy can be used, and this pre-defined resource supply strategy will be adopted for all instances of triggering the second-stage algorithm post-processing.

[0188] It is understandable that when a user triggers the execution of the second-stage algorithm post-processing by viewing or sharing images, the scheduling strategy can be further determined based on the system status data, and the second-stage algorithm post-processing task can be executed according to the scheduling strategy.

[0189] It is worth noting that determining the scheduling strategy for the second-stage algorithm post-processing based on the real-time system status can make the system more intelligent, make the resource supply strategy more in line with user needs, and save device power consumption as much as possible while ensuring user experience.

[0190] The segmented shooting scheme provided in this application embodiment can reduce the processor load and memory usage during the shooting process, improve shooting performance, and enhance the user's shooting experience.

[0191] Specifically, if single-segment shooting is used, the RAM usage will continuously increase with the number of photos taken until the system memory is exhausted or reaches the limit, thus slowing down the system. However, if segmented shooting is used, because the image is transferred from RAM to ROM, the system's RAM will not be exhausted even as the number of photos increases, effectively alleviating RAM pressure and improving shooting performance. In other words, the segmented shooting solution can effectively alleviate the problem of excessive RAM usage during shooting, thereby reducing device lag and other issues.

[0192] Segmented shooting effectively alleviates memory pressure, so the system will not lag after using the camera; when the RAM is low, there will be no lag when using the camera app to take pictures, effectively alleviating the device lag problem.

[0193] Normally, when taking continuous photos with a camera, the device generates a lot of heat due to computation, causing the device temperature to rise continuously. Segmented shooting can effectively control the amount of computation, extending the computation time and thus making the device temperature rise more slowly and the maximum temperature lower. In other words, the segmented shooting solution can effectively alleviate the device's heat generation problem, reduce the amount of computation during the shooting process, and lower the processor load.

[0194] In addition, images obtained through a segmented shooting method are of better quality and effect than images obtained through a single-segment shooting method.

[0195] In single-segment image capture schemes, all post-processing algorithms are executed during the image capture process. The concentrated execution time of these post-processing algorithms and the large computational load during the capture process cause the processor to remain under high load for extended periods, potentially leading to device overheating. When overheating occurs, single-segment image capture schemes typically employ thermal escape mechanisms and abandon post-processing algorithms, directly using a single frame image or a simple algorithm. This results in poor image quality and effects.

[0196] The segmented image capture scheme of this application only executes part of the post-processing algorithm during the image capture process, resulting in a smaller computational load. Furthermore, by performing the first-stage and second-stage post-processing algorithms at different time intervals, the processor is prevented from being under high load for extended periods, mitigating the overheating problem. In addition, when the device overheats, the segmented image capture scheme performs segmented post-processing on the image data without abandoning subsequent processing algorithms, resulting in better image quality and effects.

[0197] The segmented photography scheme provided in this application can be applied to electronic devices, such as mobile phones or tablets, and the type of electronic device is not limited here.

[0198] For example, see Figure 11 The schematic diagram of the electronic device 100 shown may include, but is not limited to, a processor 110, a memory 120, a sensor module 130, a camera 140, and a display screen 150. The sensor module 130 may include, but is not limited to, a temperature sensor 131 and a touch sensor 132.

[0199] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, when the electronic device 100 is a mobile phone, the electronic device 100 may also include: a universal serial bus (USB) interface, a charging management module, a power management module, a battery, an antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headphone jack, buttons, a motor, an indicator, a subscriber identification module (SIM) card interface, a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0200] Processor 110 may include one or more processing units, such as application processors (APs), graphics processing units (GPUs), image signal processors, controllers, video codecs, digital signal processors (DSPs), and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0201] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0202] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, etc.

[0203] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 132, camera 140, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 132 through the I2C interface, enabling the processor 110 and the touch sensor 132 to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0204] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 150 and the camera 140. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 140 communicate via the CSI interface to enable the electronic device 100 to perform its shooting function. The processor 110 and the display screen 150 communicate via the DSI interface to enable the electronic device 100 to perform its display function.

[0205] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 140, display screen 150, sensor module 130, etc. The GPIO interface can also be configured as an I2C interface or a MIPI interface, etc.

[0206] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0207] Electronic device 100 implements display functions through a GPU, a display screen 150, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 150 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0208] Display screen 150 is used to display images and videos, etc. Display screen 150 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 150, where N is a positive integer greater than 1.

[0209] Electronic device 100 can perform shooting functions through ISP, camera 140, video codec, GPU, display 150 and application processor.

[0210] The ISP (Image Signal Processor) is used to process data fed back from the camera 140. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimizations on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera 140.

[0211] Camera 140 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 140, where N is a positive integer greater than 1.

[0212] Digital signal processors are used to process digital image signals.

[0213] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0214] The memory 120 can be used to store computer executable program code, which includes instructions. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.). Furthermore, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 120 and / or instructions stored in memory disposed within the processor.

[0215] In the segmented shooting scheme of this application embodiment, the image data output by the ISP can be first stored in RAM. When entering the segmented shooting architecture, the image data is cut from RAM to ROM. In addition, the intermediate results of the post-processing algorithm are stored in ROM instead of RAM. This reduces the memory resource consumption of the algorithm post-processing stage, further alleviating problems such as lag and overheating, and improving shooting performance. Especially for scenarios with rapid memory increases, such as rapid continuous shooting, storing the intermediate results of the algorithm post-processing stage in ROM can effectively alleviate the problem of rapid memory growth.

[0216] Temperature sensor 131 is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 131 to execute a temperature processing strategy. For example, when the temperature reported by temperature sensor 131 exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 131 to reduce power consumption.

[0217] Touch sensor 132, also known as a "touch device," can be located on display screen 150. The touch sensor 132 and display screen 150 together form a touchscreen, also known as a "touchscreen." Touch sensor 132 is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 150.

[0218] After introducing the possible hardware architecture of the electronic device 100 according to the embodiments of this application, the following content will be introduced using the electronic device 100 as an example.

[0219] See Figure 12 The illustrated flowchart represents a method for image processing, which may include the following steps:

[0220] Step S1201: Electronic device 100 opens camera applications.

[0221] Camera applications, in particular, refer to applications with camera functionality. Typically, these are camera apps, but they can also be third-party applications with camera capabilities.

[0222] Step S1202: Electronic device 100 responds to the photo-taking operation and acquires the image data to be processed.

[0223] Specifically, when a camera application is running in the foreground, the electronic device 100 responds to a photo-taking operation by acquiring image data to be processed through an image acquisition device. The image acquisition device may include an image sensor and an image signal generator (ISP).

[0224] For example, see Figure 2A In the illustrated photo-taking scenario, when the phone 11 detects that the user taps the camera application 111, it opens the camera application. While the camera application is running in the foreground, the user taps the photo-taking control 113 to input a photo-taking operation. The phone 11 responds to the photo-taking operation by acquiring image data.

[0225] The aforementioned image data to be processed can refer to image data output by the ISP. For example, the image data to be processed is raw image data. In addition to obtaining the image data to be processed, the image acquisition process can also obtain auxiliary device information and image metadata.

[0226] Step S1203: When the camera application is running in the foreground, the electronic device 100 performs a first-stage algorithm post-processing on the image data to be processed to obtain the first image data, which is the image data in which the algorithm post-processing is not completed.

[0227] In some embodiments, before performing the first-stage algorithm post-processing, the electronic device 100 may first determine whether the conditions for entering segmented shooting are met. If the conditions are met, segmented shooting is entered and the first-stage algorithm post-processing is performed on the image data to be processed. If the conditions are not met, segmented shooting is not entered. In this case, all post-processing algorithms can be executed continuously and uninterruptedly during the shooting process to process the image data to be processed in real time.

[0228] Furthermore, the conditions for entering segmented shooting mode can include at least one of the following: detection of continuous shooting, memory usage below a third threshold, device temperature above a second temperature threshold, and post-processing algorithm execution time exceeding a preset time threshold. The third threshold, second temperature threshold, and preset time threshold can all be set according to actual needs. Actively entering the segmented shooting architecture when the system is in poor condition can not only alleviate device lag and overheating issues but also ensure the real-time performance of shooting as much as possible. When the system is in good condition, post-processing algorithms can be performed in real time to meet the real-time shooting requirements.

[0229] For example, when a user takes multiple images consecutively, causing the data displayed in the RAM buffer to increase, meaning the remaining memory capacity falls below the third threshold, problems such as lag or unresponsiveness during image capture may occur if a segmented image capture architecture is not adopted. To alleviate lag caused by excessive memory usage, segmented image capture can be implemented to transfer image data from the RAM buffer to the ROM flash, thus relieving memory pressure.

[0230] For example, if the phone is overheating, meaning the device temperature exceeds the second temperature threshold, the system has already activated a temperature control strategy to limit CPU usage. In this case, to improve camera performance, segmented shooting mode can be used.

[0231] For example, if the current post-processing algorithm takes longer than expected, i.e., the post-processing algorithm takes longer than the preset time threshold, and the memory and image processing time cannot meet the real-time shooting requirements, then segmented shooting can be used to alleviate system pressure.

[0232] Typically, the image data to be processed output by the ISP is stored in RAM. To address the issue of rapid memory usage during image capture, in some embodiments, after entering segmented image capture mode, the electronic device 100 can transfer the image data to be processed from volatile storage to non-volatile storage. At this point, when performing the second-stage algorithm post-processing, the electronic device can first read the image data to be processed from the non-volatile storage, then perform the first-stage algorithm post-processing on the image data to obtain the first image data, and finally store the first image data in the non-volatile storage. This transfer of intermediate data and image data from volatile storage to non-volatile storage further reduces the memory usage during algorithm post-processing, thereby alleviating device lag and other issues. Similarly, during the remaining stages of algorithm post-processing, intermediate data and the final image can also be stored in ROM to alleviate memory pressure.

[0233] In some embodiments, the electronic device 100 may also directly read the image data to be processed from the RAM buff, perform a first-stage algorithm post-processing on the image data, and then store the result of the first-stage algorithm post-processing and other intermediate data in the ROM. The remaining post-processing stages then read the result of the first-stage algorithm post-processing from the ROM and execute subsequent post-processing algorithms. In this case, it is not necessary to transfer the data in the RAM buff to the ROM; instead, the first-stage algorithm post-processing can be performed based on the data in the RAM buff, and then the result of the first-stage algorithm post-processing and related data can be stored in the ROM.

[0234] It is understandable that the first image data is the output image of the first stage algorithm post-processing, and the quality and effect of this image are relatively poor, such as the 80-point image mentioned above.

[0235] When a camera application is running in the foreground, it can be considered to be in the process of taking a picture. Unlike the continuous execution of all post-processing algorithms during the picture-taking process, this embodiment only executes a portion of the post-processing algorithms during the picture-taking process, and then executes the remaining post-processing algorithms at appropriate times. The computational load of a partial post-processing algorithm is less than that of the entire post-processing algorithm. Therefore, the computational load of the segmented picture-taking scheme during the picture-taking process is less than that of the single-segment picture-taking scheme. In other words, the segmented picture-taking scheme can reduce the computational load during the picture-taking process, thereby reducing the processor load and memory usage, and alleviating problems such as device lag and overheating. Furthermore, it can also reduce the time the processor is under high load during the picture-taking process, thus alleviating the device overheating problem.

[0236] After performing the first-stage algorithm post-processing, the electronic device 100 can save the first image data to the file system, generate a task ID for the remaining stage algorithm post-processing, and construct the task for the remaining stage algorithm post-processing based on the task ID. The task for the remaining stage algorithm post-processing is then triggered at a suitable subsequent time to perform the remaining stage algorithm post-processing.

[0237] Step S1204: If the electronic device 100 detects a target operation, or if a camera application is running in the background or closed, it performs post-processing of the remaining stage algorithm on the first image data to obtain the second image data. The second image data is the image data after the algorithm post-processing. The target operation is used to view, edit, or share the image. The remaining stage algorithm post-processing includes at least one stage algorithm post-processing.

[0238] In some embodiments, the target operation is to view a thumbnail, which is displayed in the viewfinder of a camera application. The thumbnail is a thumbnail corresponding to the image data to be processed.

[0239] For example, see Figure 7A In the scenario shown, the target action is the user clicking on thumbnail 114. For another example, see... Figure 7B In the scenario shown, the target action is the user clicking on image 125.

[0240] It should be noted that when a camera application is running in the foreground, if a user needs to view an image, the second-stage post-processing algorithm for that specific image can be triggered promptly to obtain the processed image data. At this time, after triggering the second-stage post-processing algorithm for the specified image, tasks currently executing the second-stage post-processing algorithm for other images can be prioritized for execution of the task processing the specified image.

[0241] Of course, when a target operation is detected, the second-stage algorithm post-processing can be skipped and instead triggered after the camera application is closed or running in the background. In this case, the electronic device 100 responds to the target operation by displaying the first image data to the user, that is, displaying the image output from the first-stage algorithm post-processing.

[0242] It's worth noting that when the electronic device 100 detects a target operation, it typically exits the photo-taking process, either by closing or running the camera application in the background. For example, if the target operation is viewing a thumbnail, and the camera application is running in the foreground, the electronic device 100 responds to the user's tap on the thumbnail by exiting the photo-taking process and entering preview mode. Once in preview mode, the electronic device 100 doesn't need to process the preview video stream. Compared to performing post-processing algorithms during the photo-taking process, this reduces processor load and memory usage, improving photo-taking performance and user experience.

[0243] Typically, when a target operation is detected, it indicates that a user or application needs to manipulate the image. In this case, it's usually necessary to trigger and execute the remaining stages of the post-processing algorithm as quickly as possible to obtain the post-processed image promptly, ensuring the user or application can manipulate the image in a timely manner. Therefore, when the electronic device 100 detects a target operation, it can prioritize scheduling and executing the remaining stages of the post-processing algorithm for the corresponding image, and use a high-performance mode to execute the post-processing algorithm to obtain the post-processed image as quickly as possible.

[0244] In some embodiments, the electronic device 100 can directly trigger the post-processing of the remaining stage algorithm when the camera application is running in the background or closed. However, in this case, although the camera application is running in the background or closed, the processor load or device temperature may be high. If the post-processing of the remaining stage algorithm is triggered directly, it may increase the pressure on the system's processor and memory.

[0245] In other embodiments, when the camera application is running in the background or closed, it is further determined whether a preset trigger condition is met. If the preset trigger condition is met, the remaining stage algorithm post-processing is triggered to perform the remaining stage algorithm post-processing on the first image data to obtain the second image data. Of course, the electronic device 100 may also trigger the remaining stage algorithm post-processing after detecting that the user has exited the camera application.

[0246] For example, the preset trigger conditions include at least one of the following: opening an image access application, viewing an image, turning off or turning off the screen, sharing an image, editing an image, the system being in an idle state, the RAM being higher than a first threshold, the device temperature being lower than a first temperature threshold, and the processor load being lower than a second threshold. The first threshold, the first temperature threshold, and the second threshold can all be set according to actual needs and are not limited here.

[0247] Normally, after exiting a camera application, if the system status meets certain conditions (e.g., CPU load, device temperature, etc. are lower than preset values), the remaining stage of the algorithm post-processing is triggered.

[0248] Opening an image access application can be, for example, a user clicking on a gallery app. Viewing an image can be, for example, clicking on an image within an image app. It's important to note that viewing, editing, and sharing images can be user-initiated actions, such as a user clicking on or sharing an image; or they can be application-initiated actions, such as a third-party instant messaging application proactively viewing or sharing an image.

[0249] Operations such as viewing, editing, and sharing images can be performed on the first image data, the image data to be processed, or other image data. Other image data refers to image data that is neither the first image data nor the image data to be processed. If the operation is not performed on the first image data or the image data to be processed, the task corresponding to the remaining stage algorithm post-processing of the first image data can be triggered and scheduled for execution first. If the operation is not performed on the first image data or the image data to be processed, it can be processed according to the first-in-first-out order of the task queue. When the task corresponding to the remaining stage algorithm post-processing of the first image data is reached, the remaining stage algorithm post-processing of the first image data can be performed.

[0250] It should be noted that when camera applications are closed or running in the background, and preset triggering conditions are met, the system can either trigger a task in the second-stage algorithm post-processing queue or trigger a specific task. For example, when a user is detected opening an image access application, the system is idle, or the running memory is above a first threshold, the device temperature is below a first temperature threshold, and the processor load is below a second threshold, electronic device 100 will trigger a task in the second-stage algorithm post-processing queue. In this case, tasks in the task queue are executed sequentially according to the first-in-first-out order. During the sequential execution of tasks, if a user is further detected viewing or sharing an image, the task triggering the second-stage algorithm post-processing of the specified image can be prioritized.

[0251] For example, when a user is detected viewing or sharing an image, the electronic device 100 can prioritize scheduling the task that triggers the second-stage algorithm post-processing of the specified image, regardless of whether it is currently in the state of sequentially executing tasks in the task queue or in the state of processing tasks that have not yet triggered the entire task queue.

[0252] The remaining stage post-processing can include at least one stage of post-processing. For example, the remaining stage post-processing can be a second stage, meaning there is only one stage; or, the remaining stage post-processing can include a second stage and a third stage, meaning there are two stages. Each stage of post-processing in the remaining stage can be triggered by certain triggering conditions. The triggering conditions for each stage can be found in the predefined triggering conditions mentioned above, and are not limited here.

[0253] During the post-processing of the remaining stages of the algorithm, the post-processing procedures for each stage can be executed on the edge or on the cloud. For example, the post-processing of the remaining stages includes the second stage and the third stage. The second stage is executed on the edge to obtain the result, and the result and related data are sent to the cloud. The cloud then executes the third stage and returns the result to the edge.

[0254] Furthermore, during the post-processing of the remaining stage algorithm, intelligent execution can be based on the current system state. For example, the post-processing of the remaining stage algorithm is the second-stage algorithm post-processing. At a certain moment, if the electronic device 100 detects that the system is in an idle state, it triggers the execution of the second-stage algorithm post-processing; however, if it detects that the current system's processor load and memory usage are too high halfway through execution, it can stop the execution of the second-stage algorithm post-processing and record the intermediate results in the current state; at the next moment, if the system is detected to be idle, it can continue to execute the remaining process of the second-stage algorithm post-processing based on the recorded intermediate results.

[0255] Typically, when camera applications are closed or running in the background, and the user does not need to view the images immediately, the electronic device 100 can "slowly" trigger and execute the remaining stage of the algorithm post-processing. The purpose of "slowly" execution and triggering is to minimize device functionality. For example, the remaining stage of the algorithm post-processing can be triggered only when the system is completely idle or the screen is off.

[0256] In this embodiment, when a camera application is running in the background or closed, it is further determined whether a preset trigger condition is met. Only when the preset trigger condition is met is the remaining stage algorithm post-processing executed. This allows the remaining stage algorithm post-processing to be performed promptly when the user needs to manipulate the image data, enabling the user to view the final image in a timely manner. In addition, the remaining stage algorithm post-processing can also be executed only when the system status meets certain conditions, such as the running memory, device temperature, and processor load meeting requirements. This can further alleviate problems such as device lag and overheating.

[0257] After triggering the residual stage algorithm post-processing, the residual stage algorithm post-processing task can be executed according to a pre-set scheduling strategy, or the scheduling strategy can be determined based on system status data, and the residual stage algorithm post-processing task can be executed according to the determined scheduling strategy. At this time, the electronic device 100 first monitors the system status data, then determines the scheduling strategy based on the system status data, and finally executes the target task in the task queue according to the scheduling strategy to perform residual stage algorithm post-processing on the first image data to obtain the second image data. The target task is the task corresponding to the residual stage algorithm post-processing process of the first image data, and the task queue includes tasks corresponding to the residual stage algorithm post-processing processes of each image data. The target task is generated based on the task identifier of the residual stage algorithm post-processing process of the first image data. By monitoring system status data and scheduling the execution of residual stage algorithm post-processing tasks according to the system status data, problems such as device lag and overheating can be further alleviated.

[0258] Understandably, when executing tasks in the task queue, they can be executed sequentially in a first-in-first-out order, or a specific task can be triggered for execution as needed.

[0259] As can be seen from the above, compared to continuously executing all post-processing algorithms during the photo-taking process, the embodiments of this application execute only a portion of the post-processing algorithms during the photo-taking process, and execute the remaining post-processing algorithms at a suitable time later. This can reduce the consumption of computing and memory resources during the photo-taking process, thereby alleviating problems such as device lag and device overheating, improving photo-taking performance, and enhancing the user's photo-taking experience.

[0260] In some embodiments, before obtaining the second image data, if the electronic device 100 detects a viewing operation on the first image data, it displays a first target image in response to the viewing operation. The first target image is the image corresponding to the first image data. After obtaining the second image data, the electronic device 100 refreshes the first target image to the second target image, which is the image corresponding to the second image data.

[0261] In other words, when a user needs to view an image, the output image of the first-stage post-processing algorithm is displayed in real time, and the remaining post-processing algorithms are triggered in real time. After the remaining post-processing algorithms are completed, the output image of the first-stage post-processing algorithm is refreshed to the output image of the second-stage post-processing algorithm. In this way, the computational and memory resources used during the shooting process are reduced, thereby alleviating problems such as device lag and overheating, while ensuring the real-time viewing of images for users.

[0262] For example, see Figure 7A or Figure 7B In the scenario shown, when the user clicks on thumbnail 114 or image 125, the mobile phone 11 can first display the 80-point image (i.e., the first target image) in real time, and after obtaining the 100-point image (i.e., the second target image), refresh the 80-point image to the 100-point image.

[0263] See Figure 13 The schematic block diagram of the electronic device shown may include an image acquisition module 1301 and an algorithm post-processing module 1302.

[0264] The image acquisition module 1301 is used to: acquire image data to be processed in response to a photo-taking operation.

[0265] The algorithm post-processing module 1302 is used to: perform a first stage of algorithm post-processing on the image data to be processed when the camera application is running in the foreground to obtain first image data, which is image data incomplete in the algorithm post-processing; if a target operation is detected, or the camera application is running in the background or closed, perform the remaining stage of algorithm post-processing on the first image data to obtain second image data, which is image data completed in the algorithm post-processing, and the target operation is used to view, edit or share the image.

[0266] In some possible implementations, the target operation is to view a thumbnail, which is displayed in the viewfinder of a camera application. The thumbnail is a thumbnail corresponding to the image data to be processed.

[0267] In some possible implementations, the algorithm post-processing module 1302 is specifically used to: if the camera application is running in the background or closed, and a preset trigger condition is detected, perform the remaining stage algorithm post-processing on the first image data to obtain the second image data.

[0268] In some possible implementations, the preset trigger conditions include at least one of the following: opening an image access application, viewing an image, sharing an image, editing an image, the system being idle, the running memory being higher than a first threshold, the device temperature being lower than a first temperature threshold, and the processor load being lower than a second threshold.

[0269] In some possible implementations, the algorithm post-processing module 1302 includes a system state awareness submodule 1303 and a scheduling submodule 1304.

[0270] The system status awareness submodule 1303 is used to monitor system status data.

[0271] The scheduling submodule 1304 is used to determine the scheduling strategy based on the system status data; and according to the scheduling strategy, to execute the target task in the task queue to perform the remaining stage algorithm post-processing on the first image data to obtain the second image data.

[0272] The target task is the task corresponding to the post-processing of the remaining stage algorithm of the first image data. The task queue includes the tasks corresponding to the post-processing of the remaining stage algorithm of each image data. The target task is generated based on the task identifier of the post-processing of the remaining stage algorithm of the first image data.

[0273] In some possible implementations, the electronic device further includes a refresh module for: detecting a viewing operation on the first image data; displaying a first target image in response to the viewing operation, the first target image being the image corresponding to the first image data; and refreshing the first target image to a second target image after obtaining the second image data, the second target image being the image corresponding to the second image data.

[0274] In some possible implementations, the electronic device further includes a non-volatile storage medium; the algorithm post-processing module 1302 is further configured to: cut the image data to be processed from the volatile storage medium to the non-volatile storage medium; read the image data to be processed from the non-volatile storage medium, and perform a first-stage algorithm post-processing on the image data to be processed to obtain first image data, and store the first image data in the non-volatile storage medium.

[0275] In some possible implementations, the electronic device also includes a segmented image capture triggering module, which, if it is determined that the conditions for entering segmented image capture are met, proceeds to the step of performing a first-stage algorithmic post-processing on the image data to be processed to obtain the first image data.

[0276] In some possible implementations, the conditions for entering segmented shooting include at least one of the following: detecting a burst shooting operation, running memory being lower than a third threshold, device temperature being higher than a second temperature threshold, and post-processing algorithm taking longer than a preset time threshold.

[0277] The aforementioned electronic device has the function of implementing the above-mentioned photo processing method. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function. The modules can be software and / or hardware.

[0278] It should be noted that the information interaction and execution process between the modules of the above-mentioned electronic device are based on the same concept as the method embodiment of this application. For details on their specific functions and technical effects, please refer to the method embodiment section, and they will not be repeated here.

[0279] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0280] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0281] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the embodiments provided in this application, it should be understood that the disclosed devices, electronic devices, and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative; for example, the division of modules or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0282] The electronic device provided in this application embodiment may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method as described in any of the above-described image processing method embodiments. This application embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in the above-described image processing method embodiments. This application embodiment provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps in the above-described image processing method embodiments. This application embodiment also provides a chip system including a processor coupled to a memory. The processor executes the computer program stored in the memory to implement the methods described in the above-described image processing method embodiments. The chip system may be a single chip or a chip module composed of multiple chips.

[0283] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, in the description of this application specification and appended claims, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. References to "one embodiment" or "some embodiments" in this application specification mean that one or more embodiments of this application include specific features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in yet other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. Finally, it should be noted that the above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A photographing processing method characterized by comprising: The method is applied to an electronic device, and the method comprises: starting a camera application; obtaining to-be-processed image data in response to a photographing operation; when the camera application is running in the foreground, performing first-stage algorithm post-processing on the to-be-processed image data to obtain first image data, the first image data being image data for which algorithm post-processing is not completed; storing the first image data and segmented metadata to a nonvolatile storage medium, the segmented metadata comprising all or part of data in photographing metadata and auxiliary device information of the first image data and a task identifier of a target task, the target task being a task corresponding to a remaining-stage algorithm post-processing process of the first image data; if a target operation is detected or the camera application is running in the background or is closed, triggering an image acquisition request, the image acquisition request comprising information of the first image data; in response to the image acquisition request, acquiring segmented metadata of the first image data from the nonvolatile storage medium according to the information of the first image data; triggering the target task according to the segmented metadata of the first image data to perform remaining-stage algorithm post-processing on the first image data to obtain second image data, the second image data being image data for which algorithm post-processing is completed, the target operation being an operation of viewing, editing or sharing an image corresponding to the first image data, and the remaining-stage algorithm post-processing comprising at least one stage of algorithm post-processing.

2. The method of claim 1, wherein, The target operation is an operation of viewing a thumbnail, the thumbnail being displayed in a viewfinder interface of the camera application, and the thumbnail being a thumbnail corresponding to the to-be-processed image data.

3. The method of claim 1, wherein, If the camera application is running in the background or is closed, performing remaining-stage algorithm post-processing on the first image data to obtain second image data comprises: if the camera application is running in the background or is closed and a preset triggering condition is detected, performing remaining-stage algorithm post-processing on the first image data to obtain the second image data.

4. The method of claim 3, wherein, The preset triggering condition comprises at least one of the following: an operation of opening an image access application, an operation of viewing an image, an operation of sharing an image, an operation of editing an image, the system being in an idle state, the screen being turned off or the screen being in a sleep state, the running memory being higher than a first threshold value, the device temperature being lower than a first temperature threshold value, and the processor load being lower than a second threshold value.

5. The method according to any one of claims 1 to 4, characterized in that, Performing remaining-stage algorithm post-processing on the first image data to obtain second image data comprises: monitoring system state data; determining a scheduling strategy according to the system state data; performing the target task in a task queue according to the scheduling strategy to perform the remaining-stage algorithm post-processing on the first image data to obtain the second image data; The task queue comprises tasks corresponding to remaining-stage algorithm post-processing processes of respective image data, and the target task is generated according to a task identifier of the remaining-stage algorithm post-processing process of the first image data.

6. The method of claim 1, wherein, Before the second image data is obtained, the method further comprises: detecting a viewing operation for the first image data; In response to the viewing operation, a first target image is displayed, the first target image being an image corresponding to the first image data; After the second image data is obtained, the method further includes: The first target image is refreshed as a second target image, the second target image being an image corresponding to the second image data.

7. The method of claim 1, wherein, After the to-be-processed image data is obtained, the method further includes: The to-be-processed image data is cut from the volatile storage medium to the non-volatile storage medium; The to-be-processed image data is first-stage algorithm post-processed to obtain first image data, including: The to-be-processed image data is read from the non-volatile storage medium, and the to-be-processed image data is first-stage algorithm post-processed to obtain first image data, and the first image data is stored to the non-volatile storage medium.

8. The method of claim 1, wherein, Before the to-be-processed image data is first-stage algorithm post-processed to obtain first image data, the method further includes: If it is determined that the condition for entering segmented photographing is met, the step of first-stage algorithm post-processing the to-be-processed image data to obtain first image data is entered.

9. The method of claim 8, wherein, The condition for entering segmented photographing includes at least one of the following: a continuous shooting operation is detected, the running memory is lower than a third threshold, the device temperature is higher than a second temperature threshold, and the post-processing algorithm time consumption is greater than a preset time threshold.

10. An electronic device, comprising: The algorithm post-processing module and the image acquisition module are included; The image acquisition module is configured to: in response to a photographing operation, acquire to-be-processed image data; The algorithm post-processing module is configured to: when the camera application is in foreground running, first-stage algorithm post-process the to-be-processed image data to obtain first image data, the first image data being image data for which algorithm post-processing is not completed; The first image data and segmentation metadata are stored to a non-volatile storage medium, the segmentation metadata including all or part of data of photographing metadata of the first image data and auxiliary device information, and a task identifier of a target task; The target task is a task corresponding to a remaining-stage algorithm post-processing process of the first image data; if a target operation is detected, or the camera application is in background running or is closed, an image acquisition request is triggered, the image acquisition request including information of the first image data; in response to the image acquisition request, the segmentation metadata of the first image data is acquired from the non-volatile storage medium according to the information of the first image data; According to the segmentation metadata of the first image data, the target task is triggered to perform remaining-stage algorithm post-processing on the first image data to obtain second image data, the second image data being image data for which algorithm post-processing is completed, the target operation being an operation for viewing, editing or sharing an image, and the remaining-stage algorithm post-processing including at least one stage of algorithm post-processing.

11. The electronic device of claim 10, wherein, The target operation is an operation for viewing a thumbnail, the thumbnail being displayed on a viewfinder interface of the camera application, and the thumbnail being a thumbnail corresponding to the to-be-processed image data.

12. The electronic device of claim 10, wherein, The algorithm post-processing module is specifically configured to: If the camera application is in the background or closed, and a preset trigger condition is detected, the first image data is processed by a remaining stage algorithm post-processing to obtain the second image data.

13. The electronic device of claim 12, wherein, The preset trigger condition includes at least one of the following: an operation of opening an image access application, an operation of viewing an image, an operation of sharing an image, an operation of editing an image, a system being in an idle state, a screen being off or a screen being in a breath state, a running memory being higher than a first threshold, a device temperature being lower than a first temperature threshold, and a processor load being lower than a second threshold.

14. The electronic device of any of claims 10 to 13, wherein, The algorithm post-processing module includes a system state sensing submodule and a scheduling submodule. The system state sensing submodule is configured to monitor system state data. The scheduling submodule is configured to determine a scheduling strategy according to the system state data, and execute a target task in a task queue according to the scheduling strategy to process the first image data by the remaining stage algorithm post-processing to obtain the second image data. The task queue includes tasks corresponding to remaining stage algorithm post-processing processes of respective image data, and the target task is generated according to a task identifier of the remaining stage algorithm post-processing process of the first image data.

15. The electronic device of claim 10, wherein, The refresh module is further configured to detect a viewing operation for the first image data, display a first target image in response to the viewing operation, the first target image being an image corresponding to the first image data, and refresh the first target image to a second target image after the second image data is obtained, the second target image being an image corresponding to the second image data.

16. The electronic device of claim 10, wherein, The algorithm post-processing module is further configured to: Cut the to-be-processed image data from the volatile storage medium to the non-volatile storage medium. Read the to-be-processed image data from the non-volatile storage medium, process the to-be-processed image data by a first stage algorithm post-processing to obtain first image data, and store the first image data to the non-volatile storage medium.

17. The electronic device of claim 10, wherein, The segmented photographing trigger module is configured to enter the step of processing the to-be-processed image data by the first stage algorithm post-processing to obtain the first image data if it is determined that a condition of entering segmented photographing is met.

18. The electronic device of claim 17, wherein, The condition of entering segmented photographing includes at least one of the following: detecting a continuous shooting operation, a running memory being lower than a third threshold, a device temperature being higher than a second temperature threshold, and a post-processing algorithm time consumption being greater than a preset time threshold.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 9.

20. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 9.

Images