Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium, and is applied to terminal equipment, and the method comprises the steps: obtaining original image data and a corresponding reference image in response to a shooting instruction; performing image processing on the reference image to generate first intermediate data and a first processing image; under the condition that the current state of the terminal equipment meets a first condition, generating a second processing image based on the first intermediate data and the original image data, and updating the first processing image into the second processing image; wherein the resolution of the second processed image is higher than that of the first processed image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the continuous development of mobile imaging, terminal devices equipped with camera applications can usually use the terminal's built-in camera module to take photos to meet user photography needs. However, due to the hardware limitations of the mobile terminal's lens size, subsequent algorithm processing is required to minimize the gap between the device image and the target image.

[0003] However, processing image data takes time, and the specific duration often depends on the size of the algorithm. As algorithms become increasingly complex, processing time increases, potentially requiring users to wait for optimized images, reducing image processing efficiency and impacting the real-time performance of the terminal device. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device, and storage medium, which can avoid some repeated calculations by reusing intermediate results across stages, thereby improving image processing efficiency and enhancing the real-time performance of terminal devices.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method, applied to a terminal device, the method comprising:

[0007] In response to a shooting instruction, acquiring original image data and a corresponding reference image;

[0008] performing image processing on the reference image to generate first intermediate data and a first processed image;

[0009] When the current state of the terminal device meets the first condition, a second processed image is generated based on the first intermediate data and the original image data, and the first processed image is updated to the second processed image; wherein the resolution of the second processed image is higher than the resolution of the first processed image.

[0010] In a second aspect, an embodiment of the present application provides an image processing method, applied to a cloud server, the method comprising:

[0011] Receiving an image optimization request sent by a terminal device;

[0012] Based on the image optimization request, the image to be processed received by the cloud server is optimized to generate a target image;

[0013] Send the target image to the terminal device.

[0014] In a third aspect, an embodiment of the present application provides an image processing apparatus, applied to a terminal device, the image processing apparatus comprising:

[0015] an acquisition unit, configured to acquire raw image data and a corresponding reference image in response to a shooting instruction;

[0016] a processing unit configured to perform image processing on the reference image to generate first intermediate data and a first processed image;

[0017] A generation unit is configured to generate a second processed image based on the first intermediate data and the original image data when the current state of the terminal device meets the first condition, and update the first processed image to the second processed image; wherein the resolution of the second processed image is higher than the resolution of the first processed image.

[0018] In a fourth aspect, an embodiment of the present application provides an image processing device, which is applied to a cloud server. The image processing device includes:

[0019] a receiving unit configured to receive an image optimization request sent by a terminal device;

[0020] a processing unit configured to perform image optimization processing on the image to be processed received by the cloud server based on the image optimization request to generate a target image;

[0021] The sending unit is configured to send the target image to the terminal device.

[0022] In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method described in the first aspect or the second aspect is implemented.

[0023] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method described in the first or second aspect above is implemented.

[0024] The present application provides an image processing method, apparatus, device, and storage medium. The method is applied to a terminal device, and in response to a capture instruction, obtains original image data and a corresponding reference image; performs image processing on the reference image to generate first intermediate data and a first processed image; and, when the current state of the terminal device satisfies a first condition, generates a second processed image based on the first intermediate data and the original image data, and updates the first processed image to the second processed image; wherein the resolution of the second processed image is higher than the resolution of the first processed image. Thus, through this progressive image processing method, the terminal device first processes the reference image to generate the first intermediate data and the first processed image, and then, when the terminal device satisfies the first condition, combines the first intermediate data with the original image data to generate and store the second processed image. Thus, through the image processing and intermediate result reuse mechanism, not only can repeated calculations in the image processing process be avoided, significantly reducing computing power and energy consumption, but also, by triggering the generation of the second processed image on demand, system resource scheduling can be optimized, thereby improving image processing efficiency and further enhancing the real-time performance of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the image processing method implementation process provided in the embodiment of the application Figure 1 ;

[0026] Figure 2 A schematic diagram of an image of standard size provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a layered multiplexing architecture for hot and cold data provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a visual control for starting / stopping the second shooting mode provided in an embodiment of the present application;

[0029] Figure 5 A full-size image schematic diagram provided for an embodiment of the present application;

[0030] Figure 6 A schematic diagram of a first user operation to start / close a visual control provided in an embodiment of the present application;

[0031] Figure 7 Schematic diagram of the image processing method implementation process provided in the embodiment of the application Figure 2 ;

[0032] Figure 8 A schematic diagram of a cloud server optimization processing strategy provided in an embodiment of the present application;

[0033] Figure 9A schematic diagram of image optimization processing performed by a cloud server provided in an embodiment of the present application;

[0034] Figure 10 Schematic diagram of the implementation framework for cross-stage reuse of intermediate data provided in an embodiment of the present application;

[0035] Figure 11 A schematic diagram of the phased processing flow provided for an embodiment of the present application;

[0036] Figure 12 A schematic diagram of the end-cloud collaborative architecture process provided in an embodiment of the present application;

[0037] Figure 13 This is a schematic diagram comparing the reuse rates of intermediate results of this application and the control group;

[0038] Figure 14 This is a comparison diagram of the local storage space occupied by a single photo taken by this application and the control group;

[0039] Figure 15 Schematic diagram of the composition structure of the image processing device proposed in the embodiment of the present application Figure 1 ;

[0040] Figure 16 Schematic diagram of the composition structure of the image processing device proposed in the embodiment of the present application Figure 2 ;

[0041] Figure 17 This is a schematic diagram of the structure of the electronic device proposed in the embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. It should be understood that the specific embodiments described herein are only used to explain the different applications and are not intended to limit the application. It should also be noted that for ease of description, the drawings only show the parts that differ from the related applications.

[0043] With the continuous development of mobile imaging, due to the hardware limitations of the lens size of mobile devices such as mobile phones, it is necessary to rely on subsequent algorithm processing to bridge the gap between mobile phone imaging and SLR and photo editing as much as possible.

[0044] However, due to differences in the corresponding algorithms for different shooting modes, the time required to synthesize the target image varies. For example, in beauty shooting mode, synthesizing the target image may require invoking algorithms such as facial feature detection, eye enlargement, and skin smoothing. In background blur shooting mode, synthesizing the target image may require invoking algorithms such as foreground-background segmentation and background blur.

[0045] However, as algorithms become increasingly complex, processing time increases, potentially requiring users to wait a long time before seeing their newly optimized images. At the same time, increasingly clear images typically mean larger image sizes, taking up more and more storage space on mobile phones.

[0046] In one possible implementation, a single-shot processing solution is used. This means that as soon as the user takes a photo, the mobile device immediately begins processing the current image. The user cannot see the generated image until processing is complete. It is certain that synthesizing the target image in each shooting mode requires a certain amount of time. If the user triggers the command to view the target image before synthesis is complete, a waiting interface will be displayed. The user must wait until synthesis is complete before viewing the target image. This can result in a poor user experience. If this waiting period can be avoided, this problem can be resolved.

[0047] In one possible implementation, multi-stage fusion technology is used to improve the imaging speed through a staged process of synthesizing a preview frame → a quick image (Quick image) → a final image (Final image). Then, when the user triggers an instruction for viewing a target image, the preview image can be displayed to the user first. After the target image is synthesized, the target image is displayed to cover the preview image, thus avoiding the problem of waiting.

[0048] That is to say, although the relevant technology can select the preview image in time and display it to the user, each stage independently calculates intermediate data such as facial feature points and segmentation masks, resulting in more than 15% computational redundancy.

[0049] Specifically, the relevant technologies have the following problems:

[0050] (1) Intermediate data management defects.

[0051] Repeated calculation problem: In the related technology, facial features need to be re-extracted and segmentation results need to be independently calculated at each stage (taking about 120ms per image), resulting in an increase of about 15% in energy consumption for 12-megapixel (MP) image processing.

[0052] Data isolation: Most images can only be processed at the moment they are taken. The user has to wait as long as the processing time. Unprocessed data is lost, which may cause resource congestion when a large number of tasks need to be processed in a short period of time. There is no cross-time reuse and peak-shifting processing mechanism.

[0053] (2) Insufficient cloud collaboration efficiency.

[0054] Real-time processing bottleneck: Android's native camera function can only be triggered in real time on the client side. Taking multiple photos causes severe resource competition and congestion. The need for real-time processing of multiple tasks can easily lead to platform bottlenecks.

[0055] Single storage strategy: Android native camera data is stored locally on the phone by default, taking up the user's phone storage space.

[0056] (3) Lack of resource scheduling mechanism.

[0057] Waste of computing resources: The current image processing framework for mobile devices does not dynamically schedule hardware computing power and adjust hardware computing power processing priority based on the device status (whether it is charging or whether the hardware computing power resources are idle). The potential of heterogeneous computing power such as the central processing unit (CPU), graphics processing unit (GPU) and neural network processing unit (NPU) is not fully stimulated.

[0058] (4) User experience pain points.

[0059] Storage space pressure: Related technologies require local storage of full-size original images (e.g., larger than 18.6MB), resulting in a reduction in the actual available capacity of storage devices. Limited storage capacity, such as 128GB and 256GB, is even more limited. In Related Technology 1, a single image takes up approximately 18.6MB of storage space. After a user takes 100 photos, the local space is reduced by 1.86GB.

[0060] Cloud access delay: Although similar cloud album services have enabled cloud access, accessing large cloud images requires re-downloading the entire file. Pre-loading based on user behavior predictions is not implemented, resulting in a poor image viewing experience.

[0061] To address the above-mentioned problems, embodiments of the present application provide an image processing method, apparatus, device, and storage medium. In response to a capture instruction, the method obtains original image data and a corresponding reference image; performs image processing on the reference image to generate first intermediate data and a first processed image; and, when the current state of the terminal device satisfies a first condition, generates a second processed image based on the first intermediate data and the original image data, and updates the first processed image to the second processed image; wherein the resolution of the second processed image is higher than the resolution of the first processed image. Thus, through this progressive image processing method, the terminal device first processes the reference image to generate the first intermediate data and the first processed image, and then, when the terminal device satisfies the first condition, combines the first intermediate data with the original image data to generate and store the second processed image. Thus, through the image processing and intermediate result reuse mechanism, not only can repeated calculations in the image processing process be avoided, significantly reducing computing power and energy consumption, but also, by triggering the generation of the second processed image on demand, system resource scheduling can be optimized, thereby improving image processing efficiency and, in turn, enhancing the real-time performance of the terminal device.

[0062] The embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

[0063] In one embodiment of the present application, an image processing method is provided. The image processing method can be applied to an image processing device or electronic device, and can also be applied to any terminal device including the image processing device or electronic device.

[0064] In the embodiments of this application, Figure 1 Schematic diagram of the image processing method implementation process provided in the embodiment of the application Figure 1 ,like Figure 1 As shown, the image processing method may include the following steps:

[0065] Step 101: In response to a shooting instruction, obtain original image data and a corresponding reference image.

[0066] Among them, the shooting instruction can be an instruction generated based on the user's shooting operation (such as touch screen operation, trigger control operation, voice control instruction, gesture control instruction, etc.), an automatic shooting instruction generated in response to a timed shooting function, etc., and the embodiments of this application are not limited.

[0067] In some embodiments, raw image data refers to image data directly captured by a camera module that has not been processed or has only undergone basic processing (such as white balance, exposure adjustment, etc.).

[0068] Exemplarily, the original image data may be an image in the Raw format or an image in the YUV420 format, which is not limited in the embodiments of the present application.

[0069] YUV is a color encoding system, where Y represents luminance (Luma) and U and V represent chrominance (Chroma). It is primarily used in video and graphics processing pipelines. Compared to the RGB color space representing red, green, and blue, the YUV format facilitates encoding and transmission, reducing bandwidth usage and information error rates. Therefore, the YUV format is generally used in this application.

[0070] In one embodiment, the original image data may be obtained in such a way that a camera of the terminal device may continuously capture image frames at a certain frame rate, and these continuously captured image frames are the original image data.

[0071] In another embodiment, the method for obtaining the original image data can also be: collecting the original image data based on the shooting time of the shooting instruction, there is a collection order between the original image data, and the collection time of the original image data is within a unit time range (such as 1s).

[0072] It should be noted that the amount of original image data can be set according to actual needs. For example, the amount of original image data can be 7 frames or 30 frames, etc., which is not limited in the embodiments of the present application.

[0073] In an embodiment of the present application, the reference image refers to a reference image (also referred to as a reference frame) selected from multiple original image data, which is used for subsequent image alignment, feature extraction and other operations, and is the basis for generating the first processed image and the second processed image.

[0074] In one embodiment, a method for obtaining a reference image corresponding to the raw image data may include: after obtaining the raw image data, caching the raw image data in a fixed-size cache queue via shared memory, wherein each preview frame in the raw image data is timestamped. Based on the timestamp of receiving the first capture instruction, the terminal device may select the preview frame closest to the timestamp as the reference frame (i.e., the reference image). For example, an image frame within a certain time range before and after the capture instruction may be selected as the reference image.

[0075] It should be noted that the original image data and reference images can be obtained through interfaces such as the Camera2 API and cached through a shared memory mechanism to improve access efficiency and reduce resource overhead.

[0076] In another embodiment, the reference image may be determined from the raw image data by evaluating the quality (e.g., clarity, exposure, focus, etc.) of each raw image frame in the raw image data and selecting the image frame with the highest quality as the reference frame. This may be achieved through image processing algorithms, such as calculating the sharpness or contrast value of the image.

[0077] In another embodiment, the method for determining the reference image from the original image data can also be: in some application scenarios, the user can explicitly specify a preview frame as the reference frame through a specific operation (such as long pressing the screen, double-clicking, etc.).

[0078] It should be noted that the reference image can be the first frame, the last frame, the frame with the highest resolution or other frames in the original image data, which is not limited in the embodiments of the present application.

[0079] It should also be noted that after the reference image is determined, the reference image can be displayed in the photo album of the terminal device for the user to view.

[0080] In an embodiment of the present application, the terminal device can directly obtain the original image data and the corresponding reference image; the terminal device can also respond to the shooting instruction after receiving the first shooting instruction, thereby obtaining the original image data and the corresponding reference image.

[0081] In some embodiments, the first shooting instruction refers to an instruction generated based on the user's shooting operation when the shooting mode of the terminal device is in the default mode (also known as the high-definition mode); the user's shooting operation may include but is not limited to touch screen operation, button operation, voice instruction, gesture control instruction, etc.

[0082] Of course, the first shooting instruction can also generate an automatic shooting instruction based on the timed shooting function, which is not limited in the embodiment of the present application.

[0083] It should be understood that the first shooting instruction can be used to control the hardware layer to generate preview frames and photo frames corresponding to the shooting instruction based on the image frames captured by the camera module, and return the preview frames and photo frames to the application layer.

[0084] It should be noted that the first shooting instruction responds to the first shooting mode, and the activation / deactivation of the first shooting mode can be controlled by a visual control, and the visual control can be displayed in the user interface.

[0085] Step 102: Perform image processing on the reference image to generate first intermediate data and a first processed image.

[0086] In some embodiments, the first intermediate data refers to temporary data generated during the image quantization process, which can be used for direct calling of subsequent processing steps to avoid repeated calculations.

[0087] Exemplarily, the first intermediate data may include image feature data and / or scene segmentation data.

[0088] In some embodiments, the image processing may be a lightweight processing for quickly extracting key features of the reference image and generating a first processed image without losing core information.

[0089] Specifically, lightweight processing can include but is not limited to face detection, scene segmentation, basic noise reduction, and the 3A algorithm (autofocus, auto white balance, and auto exposure). These processes are usually performed by lightweight algorithm models on the terminal device, can be completed in a very short time, and generate intermediate data that can be used for subsequent processing.

[0090] For example, the image feature data may be the coordinates of the facial region, key point positions, etc., while the scene segmentation data may be the object outline, background mask, etc.

[0091] In some embodiments, the first processed image may also be referred to as a Quick image, which is a low-resolution, preliminarily optimized image used to replace traditional album thumbnails, allowing users to view captured images more quickly.

[0092] It should be noted that the first processed image can be an image in YUV format, an image in RGB format, etc., which is not limited in the embodiment of the present application.

[0093] It should also be noted that the size of the generated first processed image can be set according to actual needs. For example, the size of the first processed image can be 12MB or 16MB, etc., which is not limited in the embodiments of the present application.

[0094] It should be noted that the lightweight processing process is usually completed on the device side, such as running on the NPU, to increase processing speed and save CPU / GPU resources.

[0095] It should also be noted that the terminal device can persistently store the processed first intermediate data in a local disk or database for on-demand reading in subsequent processing stages.

[0096] It should also be noted that, after generating the first processed image, the terminal device may save the first processed image for display or viewing.

[0097] Step 103: When the current state of the terminal device satisfies the first condition, generate a second processed image based on the first intermediate data and the original image data, and update the first processed image to the second processed image.

[0098] In some embodiments, the second processed image refers to an image with higher definition but not full resolution that is generated after further processing based on the first processed image.

[0099] It should be noted that the resolution of the second processed image is higher than the resolution of the first processed image.

[0100] It should be noted that the second processed image may also be referred to as a standard size image, a standard image, or a medium size image, which are equivalent or interchangeable. In other words, the standard size image may be a medium size image with a higher resolution obtained by calling a standard algorithm model and performing standard algorithm processing.

[0101] In some embodiments, the terminal device can obtain first intermediate data and original image data when the current state meets the first condition, generate a second processed image based on the first intermediate data and the original image data, and then store the second processed image to a preset storage location.

[0102] In some embodiments, after the terminal device generates the second processed image, the Quick image in the album can also be updated to the processed second processed image through a Uniform Resource Identifier (URI) replacement mechanism for display or user viewing.

[0103] In some embodiments, the current state of the terminal device satisfies the first condition, including at least one of the following: responding to a picture viewing instruction; the temperature of the terminal device is less than a preset temperature threshold; the load of the terminal device is less than a preset load threshold.

[0104] In some embodiments, an image viewing instruction refers to an operation request for viewing an image, triggered by a user through an operating interface (such as an album application) or a voice command. The image viewing instruction can be generated by the user clicking a specific icon, scrolling through the album, or issuing a voice command (such as opening an album). After receiving the instruction, the system will initiate the image loading process and obtain the second processed image.

[0105] For example, in a real-world scenario, when a user enters the photo album interface and clicks on a picture, the terminal device will first load the picture at the current browsing location from the storage area for processing. The system then reads the original image data from the disk directory and reuses the first intermediate data of the Quick image (i.e., the first processed image) to generate the second processed image.

[0106] It should be noted that, based on the image viewing instruction, the terminal device can pre-load the target image and one image before and after it for processing, so that pre-processing can be performed, thereby improving the user experience.

[0107] In some embodiments, the terminal device can preload images based on user access behavior predictions. For example, an LSTM network can be used to analyze the speed and direction of the album scrolling to predict the range of images that may be accessed within the next 3 seconds, and a preset number of adjacent second processed images can be preloaded.

[0108] In some embodiments, the temperature of the terminal device refers to the operating temperature of key components inside the device (such as CPU, GPU), which can be collected through a built-in temperature sensor.

[0109] In some embodiments, the preset temperature threshold is a temperature upper limit set based on the device's hardware characteristics and safe operating standards, used to determine whether the device is suitable for performing certain energy-intensive tasks. When the actual operating temperature of the terminal device is below this threshold, the system considers the device to be within the normal operating range and suitable for performing computationally intensive tasks. If the temperature exceeds this threshold, the hardware overheating protection mechanism may be triggered, resulting in task suspension or downgrade.

[0110] In some embodiments, the load of the terminal device refers to the current CPU usage, reflecting the amount of tasks the device is performing and the proportion of computing resources it occupies.

[0111] The preset load threshold is the upper limit of computing resource usage set by the system to ensure smooth operation. It is usually expressed as a percentage and reflects the utilization of core components such as the CPU and GPU. When the device load is below the threshold, the current system resources are sufficient to safely execute processing tasks. Conversely, if the load approaches or exceeds the threshold, the execution of non-critical tasks should be restricted or delayed.

[0112] It should be noted that the terminal device can predict the CPU / GPU load based on a load prediction algorithm and dynamically adjust the number of concurrent tasks. For example, the load prediction algorithm can be an autoregressive integrated moving average (ARIMA) model.

[0113] It should also be noted that when the temperature of the terminal device is greater than a preset temperature threshold, the idle algorithm post-processing can be suspended, and the CPU can only perform critical tasks (such as Quick map generation); when the battery level of the terminal device is less than a preset battery level threshold, the idle algorithm post-processing and other non-essential background tasks can be suspended. For example, the preset temperature threshold can be 45°C; the preset battery level threshold can be 20%, which is not limited in the embodiments of this application.

[0114] It should also be noted that any one of the above-mentioned first conditions can be met to generate the second processed image, or all three conditions can be met simultaneously to generate the second processed image. This is not limited in the embodiments of the present application.

[0115] It should also be noted that if the first intermediate data exists, the terminal device can generate the second processed image based on the first intermediate data and the original image data. If the first intermediate data does not exist, the terminal device can generate the second processed image directly based on the original image data, that is, regenerate the first intermediate data or directly reuse any previously existing related data. This flexible processing method can adapt to different scenarios and needs, ensuring that standard size processing can be carried out smoothly.

[0116] In some embodiments, the terminal device can ensure the consistency of the first intermediate data and the lightweight algorithm version based on a dual mechanism of hash value verification + timestamp marking.

[0117] For example, hash value verification can generate a unique hash value (e.g., SHA-256) for each first intermediate data file (e.g., face coordinates, segmentation mask), which is tied to the algorithm version. The hash value is verified before processing. If it does not match the current algorithm version, the old first intermediate data is discarded and regenerated.

[0118] For example, the timestamp can record the generation time of the first intermediate data, giving priority to the latest version of the data. In other words, when the terminal device rolls back to the old algorithm, the intermediate data of the corresponding time period can be automatically selected to avoid compatibility issues.

[0119] In some embodiments, when the terminal device detects a model upgrade (such as MobileNet V3→V4), it automatically triggers an incremental update and retains the old version compatible interface, allowing the user to complete the transition without noticing.

[0120] Compared with the isolated processing flow in related technologies, the intermediate data reuse rate of this application is increased from 0 to more than 30%, and the computing power overhead is reduced.

[0121] In an embodiment of the present application, the terminal device performs lightweight processing on the reference image to generate first intermediate data and a first processed image. Then, when the terminal device meets the first condition, the first intermediate data and the original image data are combined to generate a second processed image and store it. On the one hand, through the lightweight processing and intermediate result reuse mechanism, repeated calculations are avoided, and computing power consumption and energy consumption are significantly reduced; on the other hand, by triggering the generation of the second processed image on demand, system resource scheduling is optimized, thereby improving image processing efficiency and user experience. In an embodiment of the present application, when the current state of the terminal device meets the second condition, a third processed image is generated based on the second intermediate data and the original image data, and the second processed image is updated to the third processed image for display or viewing.

[0122] Among them, the second intermediate data is intermediate data in the process of generating the second processed image based on the first intermediate data and the original image data; and the feature resolution of the second intermediate data is higher than the feature resolution of the first intermediate data, and the resolution of the third processed image is higher than the resolution of the second processed image.

[0123] It can be understood that feature resolution refers to the fineness or complexity of the image features corresponding to the first and second intermediate data. For example, if the image feature is a facial feature, the facial feature of the first intermediate data may be 64*64 pixels, while the facial feature of the second intermediate data may be 128*128 pixels. In other words, the second intermediate data has a higher feature resolution and can capture more image details.

[0124] It can be understood that the third processed image refers to the use of a more complex full-scale algorithm model (such as multi-frame fusion, super-resolution reconstruction, deep learning enhancement, etc.) to process the original image data to generate higher resolution and higher quality images.

[0125] It is understood that the third processed image refers to an image with higher pixel density and richer details than the second processed image. For example, the third processed image may be an ultra-high-definition image that has been semantically enhanced using the ViT-Large model.

[0126] It should be noted that the third processed image may also be referred to as a full-size image, and the two may be equivalent to or replace each other.

[0127] In some embodiments, the second intermediate data refers to new intermediate result data generated during a standard size processing process based on the first intermediate data and the original image data.

[0128] In some embodiments, the second intermediate data may include but is not limited to segmentation mask data and color parameters.

[0129] The segmentation mask data may be generated by performing semantic segmentation on the image through a neural network. For example, the neural network may be a U-net.

[0130] Specifically, segmentation mask data is a binary matrix used to distinguish foreground and background areas during image processing. It is typically generated by deep learning models (such as U-Net). It plays a key role in multi-stage image processing and can be reused by multiple subsequent algorithms to avoid repeated calculations.

[0131] For example, in portrait mode, segmentation masks can be used to blur the background, enhance hairline definition, or guide style transfer. This solution persistently stores segmentation masks, enabling direct access for mid-scale processing, full-scale processing, and even cloud-based optimization, significantly reducing computational redundancy.

[0132] Specifically, color parameters refer to a set of numerical values ​​that describe the color characteristics of an image, including but not limited to white balance coefficients, color temperature, saturation, and tone mapping curves. These color parameters determine the overall color style and visual appearance of the image and are crucial for maintaining consistency between images at different stages.

[0133] It should be noted that the color profile (ICC Profile) extracted when the Quick image is generated can be used as a reference input in subsequent cloud processing to ensure that the image style after cloud-based refinement is consistent with the user's expectations and avoid color distortion caused by platform differences.

[0134] In some embodiments, the second intermediate data can be based on a dual mechanism of hash value verification + timestamp marking to ensure the consistency of the second intermediate data with the standard algorithm version. The specific implementation method refers to the above embodiment of the first intermediate data, and for the sake of brevity, it is not repeated here.

[0135] For example, Figure 2 A schematic diagram of a standard size image provided in an embodiment of the present application, such as Figure 2 It can be seen that the size of the standard size image is 3072*4096 pixels and the size is 3.8MB.

[0136] In some embodiments, standard size processing refers to calling a standard algorithm model to perform standard algorithm processing to generate a medium standard size image of 12M or 16M.

[0137] For example, the standard algorithm may include but is not limited to a raw domain multi-frame optimization algorithm such as a turboraw algorithm and a hybridraw algorithm.

[0138] In some embodiments, during the medium-sized image generation stage, the face detection algorithm may be skipped and the face coordinate data generated in the Quick image stage may be directly used, thereby saving approximately 40% of the computation time.

[0139] In some embodiments, a method in which a terminal device performs standard size processing based on the first intermediate data and the original image data to generate second intermediate data and a standard size image (that is, a second processed image) may include the following steps:

[0140] The terminal device can align each preview frame in the original image data with the reference frame to obtain aligned original image data; and perform standard size processing on the aligned original image data based on the first intermediate data to obtain processed original image data and second intermediate data; thereafter, the terminal device can fuse the processed original image data into a standard size image.

[0141] It should be understood that in image processing, differences between images may occur due to slight displacement, rotation, or perspective changes when capturing multiple frames. To accurately process these images (such as fusion), they can be aligned with a selected reference frame.

[0142] In some embodiments, the alignment process can be achieved through an image registration algorithm (such as feature point matching, affine transformation, etc.), which analyzes the feature points in the image (such as corners, edges, etc.), calculates the transformation relationship between the images (such as translation, rotation, scaling and other parameters), and then transforms the image according to these parameters so that multiple original image frames are spatially aligned.

[0143] It should be noted that after aligning the original image data with the reference frame, the original image data can be pre-processed such as noise reduction, and then fused into the target image, so as to ensure the display effect of the target image.

[0144] In some embodiments, if the first intermediate data exists, the first intermediate data is directly reused. The terminal device can use a standard algorithm model to generate processed raw image data and second intermediate data based on the raw image data and the first intermediate data. If the first intermediate data does not exist, the second intermediate data can be directly generated based on the standard algorithm model and the raw image data.

[0145] In some embodiments, image fusion refers to combining useful information from multiple standard-processed image frames to generate a standard image that contains more information and has higher quality.

[0146] For example, after receiving multiple processed standard-size image frames, the terminal device fuses them using a fusion algorithm. The fusion algorithm combines the frames into a single image based on pixel values, feature information, and other factors, according to certain rules (such as weighted averaging or region-based fusion).

[0147] In some embodiments, after generating a standard size image, the terminal device replaces the first processed image with the standard size image, which means that the previously generated low-quality or temporary image (such as a preview image or a Quick image) is replaced with a high-quality standard size image, and the corresponding storage path or index information is updated.

[0148] This replacement method ensures that users see more finely processed images on subsequent visits, while avoiding the storage redundancy problem caused by saving different versions of the same image multiple times.

[0149] Next, the replacement storage mechanism is introduced in the embodiments of this application.

[0150] In some embodiments, a logical association exists between the replacement storage mechanism and the standard-size image. As the final image to be displayed, the storage path or URI of the standard-size image needs to be written into the system's image database or file system to overwrite the originally stored low-quality image.

[0151] For example, when the terminal device completes the generation of the Quick image, it will store it as / images / quick / 123.jpg. After the standard size processing is completed, the system will store the new standard size image as / images / standard / 123.jpg and point the reference of the original Quick image to the new standard size image by updating the database record.

[0152] This replacement storage mechanism significantly reduces local storage usage. For example, a Quick image is approximately 4.2MB in size, while a standard-sized image is approximately 12MB. However, since the Quick image serves only as a transitional image, once the standard-sized image is generated and stored, the Quick image can be deleted or only the index information can be retained, freeing up storage resources. Furthermore, the URI replacement mechanism allows for seamless switching, without the user being aware of changes in the image processing process.

[0153] In an embodiment of the present application, by reusing the feature information in the first intermediate data to generate a standard size image and update the storage content, the waste of time and computing power caused by re-extracting the same features is avoided, thereby further improving the image processing efficiency and the overall performance of the system.

[0154] In an embodiment of the present application, after generating the first intermediate data and the second intermediate data, the terminal device may store the original image data and the associated intermediate data in a preset storage location.

[0155] In some embodiments, the intermediate data includes first intermediate data and / or second intermediate data.

[0156] In some embodiments, the intermediate results of the preceding algorithm (including but not limited to the first intermediate data, the second intermediate data, the blurred depth map, and the color parameters) use a unified agreed format to encapsulate the feature data, supporting heterogeneous inputs of lightweight models on the terminal side (such as MobileNet V3) and large cloud models (such as ViT).

[0157] In some embodiments, the preset storage location may refer to a data storage path or logical partition pre-configured by the system, which is used to store data of different priorities or purposes.

[0158] Exemplarily, the preset storage location may include but is not limited to a user album, a database, etc.; or it may be a terminal device's memory, a terminal device's disk, or a cloud server, etc., without any limitation here.

[0159] In some embodiments, the terminal device can dynamically choose to write intermediate data and original image data into the terminal device's memory, the terminal device's disk, or a cloud server based on the device status (such as memory capacity, storage space size) and user behavior (such as album access frequency).

[0160] For example, when the phone is idle and has sufficient power, the system may cache more data to the local disk; when storage space is insufficient or the user does not frequently access certain photos, the data will be uploaded to cloud storage.

[0161] In the embodiment of the present application, the first intermediate data, the second intermediate data and the original image data are uniformly stored in a preset storage location to achieve persistent management of the intermediate data. This facilitates direct access to the existing intermediate data in subsequent processing stages, reducing repeated calculations.

[0162] In another embodiment of the present application, during the process of the terminal device storing the original image data and the associated intermediate data in a preset storage location, the terminal device may determine the access frequency of the original image data and the associated intermediate data.

[0163] In some embodiments, access frequency refers to the number of read or write operations performed on a particular piece of data (e.g., raw image data, intermediate processing results) within a specific time period. This metric is used to measure the user's active use of that data. A higher access frequency indicates that the data is more important or more frequently accessed. For example, if a piece of intermediate data was accessed five times in the past 72 hours, its access frequency is 5 times / 72 hours.

[0164] In some embodiments, intermediate data refers to temporary calculation results generated during image processing, such as facial feature point coordinates, segmentation masks, and color parameters. This data can be reused between different processing stages, avoiding repeated calculations and thus saving computing resources. Depending on the data type and purpose, intermediate data may include first intermediate data and second intermediate data.

[0165] By calculating access frequency, terminal devices can identify which data requires frequent access and prioritize storage in memory, disk, or the cloud. This allows for dynamic assessment of intermediate data usage, enabling the rational allocation of storage resources and improving overall system response speed and energy efficiency.

[0166] In one implementation, when the access frequency is greater than or equal to the first frequency, the original image data and the associated intermediate data are stored in the memory of the terminal device.

[0167] In some embodiments, the first frequency is a threshold used to distinguish frequently accessed data from other data. When the access frequency of data exceeds the threshold, it indicates that the data is in a high-frequency state (also known as hot data) and is suitable for storage in a memory with fast access speed but limited capacity.

[0168] In some embodiments, the memory is a storage unit in the terminal device used to temporarily store high-speed access data. It has extremely low access latency (usually less than 10ms), but a small capacity and is suitable for storing the most commonly used data.

[0169] In the embodiment of the present application, storing high-frequency data in memory can significantly reduce the reliance on disk or network in subsequent processing stages, thereby improving processing efficiency. This ensures that key intermediate data is quickly available, thereby reducing system latency and improving the smoothness of the user's photography experience.

[0170] In another implementation, when the access frequency is less than the first frequency and greater than or equal to the second frequency, the original image data and the associated intermediate data are stored in a disk of the terminal device.

[0171] The first frequency is higher than the second frequency.

[0172] In some embodiments, the second frequency is another set threshold lower than the first frequency and is used to identify medium-frequency access data (also known as warm data). Although this type of data is not frequently accessed, it still has some use value and is suitable for storage on a disk with a larger capacity but slightly slower access speed.

[0173] In some embodiments, the disk is a medium used for persistent storage in a terminal device, and has a large storage capacity but a slow access speed (usually between a few milliseconds and tens of milliseconds).

[0174] In the embodiment of the present application, by storing the data accessed by the intermediate frequency to the disk, the long-term availability of the data can be guaranteed without occupying too much memory resources. In this way, the use of storage resources can be balanced, thereby avoiding excessive memory usage and improving the stability of system operation and resource utilization.

[0175] In another implementation, when the access frequency is less than the second frequency, the original image data and the associated intermediate data are sent to a cloud server for storage.

[0176] Accordingly, the cloud server can receive the original image data and associated intermediate data sent by the terminal device.

[0177] In some embodiments, when the access frequency of data is lower than the second frequency, it indicates that the data is accessed at a low frequency (also referred to as cold data). At this time, the system uploads the data to the cloud server for storage.

[0178] In some embodiments, the cloud server is a remote node deployed in the cloud with large-scale storage and computing capabilities, suitable for storing infrequently accessed data or performing deep processing.

[0179] In some embodiments, after uploading infrequently accessed data to a cloud server, the local device can free up storage space while still being able to obtain required data on demand through the network.

[0180] Furthermore, the cloud server also supports a variety of advanced image optimization algorithms, such as dehazing, color enhancement, dynamic range expansion, etc., which help to further improve image quality.

[0181] It should be noted that when the terminal device sends the original image data and associated intermediate data to the cloud server for storage, it can use Protocol Buffers (Protobuf) + binary stream, that is, the first intermediate data and the second intermediate data are encapsulated into a standardized format, supporting seamless docking of the terminal-side TensorFlow Lite and the cloud-side PyTorch model.

[0182] It should also be noted that the terminal device can also mark the data format with a version number, and the old version system can automatically downgrade and use historical intermediate results or recalculate on demand without using intermediate results.

[0183] It should also be noted that when the terminal device sends the original image data and the associated intermediate data to the cloud server for storage, a secure transmission mechanism can also be used for uploading.

[0184] Specifically, the terminal device may use a preset encryption algorithm (AES-256-GCM) to encrypt the original image data and the associated intermediate data.

[0185] In some embodiments, the terminal device can also fragment the original image data and the associated intermediate data, and the fragment size can be dynamically adjusted (1MB / fragment in a weak network environment, 10MB / fragment under Wi-Fi).

[0186] In some embodiments, the terminal device can also generate a unique hash fingerprint for each image and upload it to the chain (Hyperledger Fabric), so that the processing process can be traced and the results cannot be tampered with.

[0187] In some embodiments, the terminal device can also automatically adjust the data segment size (1MB to 10MB) according to the network quality (5G / Wi-Fi), and use FEC forward error correction coding in weak network environments to improve the transmission success rate.

[0188] It should be noted that the terminal device can also perform zero-trust verification on the original image data and associated intermediate data. For example, the terminal device can generate a data hash fingerprint based on blockchain technology to ensure the consistency of the cloud server processing results with the original image data and associated intermediate data of the terminal device.

[0189] Compared to the plaintext transmission strategy in related technologies, this application reduces the risk of data leakage by 99.8%, thereby improving transmission efficiency. This also enables collaborative processing between local and cloud, thereby reducing the storage pressure on terminal devices and providing a more flexible image management solution.

[0190] In an embodiment of the present application, a multi-level storage strategy based on access frequency dynamically determines the access popularity of original image data and intermediate data, and stores them in memory, disk or cloud respectively. In this way, it can flexibly adapt to the user's actual usage scenarios, thereby optimizing the utilization of local storage space, thereby reducing device power consumption and improving overall performance.

[0191] For example, Figure 3 This is a schematic diagram of a layered multiplexing architecture for hot and cold data provided in an embodiment of the present application. Figure 3As shown in the figure, this architecture adopts a three-tier storage model, including hot data in memory, warm data on disk, and cold data in the cloud. Hot data in memory refers to only the user's most recent photo data, using the extremely high bandwidth of memory to accelerate processing. Warm data on disk refers to the full-size images that users frequently access (such as photos from the past week), which are stored on disk and can be accessed locally without latency. Cold data in the cloud refers to photos that users rarely access, uploaded to the cloud for raw image storage, with only Quick images retained locally (for example, the cloud uses Glacier low-power storage for data that has not been accessed for a long time (>1 month)).

[0192] It should be noted that the terminal device can also synchronize the photo collection in advance based on the user's geographical location (such as home Wi-Fi), and the access success rate in a weak network environment can be increased by 70%.

[0193] In one embodiment, Quick images (4.2MB / image) and the 10 most recent full-size images are retained locally, and data that has not been accessed for 7 days is automatically migrated to the cloud-based Glacier cold storage. When users access it, they can quickly return to the source through the Content Delivery Network (CDN) edge node.

[0194] In one embodiment, an improved Least Recently Used-K (LRU-K) strategy is used to manage the local cache, identify infrequently accessed images, and trigger cloud backup, thereby improving local storage space utilization.

[0195] Compared with the native local full-data storage solution in related technologies, this solution reduces storage costs by 77%, and predicts loading based on user access behavior, greatly improving cloud access efficiency.

[0196] In an embodiment of the present application, the current state of the terminal device satisfies the second condition, including at least one of the following: responding to a target shooting instruction; responding to a picture viewing instruction; the temperature of the terminal device is less than a preset temperature threshold; the load of the terminal device is less than a preset load threshold.

[0197] In some embodiments, the target shooting instruction refers to an instruction generated by the terminal device based on the user's shooting operation in the second shooting mode (also known as ultra-high-definition mode).

[0198] It should be noted that the target shooting instruction responds to the second shooting mode, and the activation / deactivation of the second shooting mode can be controlled by a visual control, which can be displayed in the user interface. For example, Figure 4 A schematic diagram of a visual control for starting / stopping the second shooting mode provided in an embodiment of the present application.

[0199] refer to Figure 4 The user can click "Format" in the user interface, and then click "Ultra HD" (that is, the target shooting instruction in the embodiment) in the next user interface, so that the terminal device can generate a third processed image in response to the target shooting instruction.

[0200] It should be noted that the picture viewing instruction, the temperature of the terminal device being less than the preset temperature threshold, and the load of the terminal device being less than the preset load threshold have been introduced in detail in the above embodiments. For the sake of brevity, they will not be repeated here.

[0201] In one embodiment, after receiving a target shooting instruction, the terminal device can respond to the shooting instruction to determine whether the terminal device meets any one of the conditions of a picture viewing instruction, the temperature of the terminal device is less than a preset temperature threshold, and the load of the terminal device is less than a preset load threshold. If the current state of the terminal device meets any one of the above conditions, a third processed image is generated based on the second intermediate data and the original image data, and the third processed image is stored.

[0202] It should be noted that the terminal device can generate a third processed image when any one of the following conditions is met: an image viewing instruction, the temperature of the terminal device is less than a preset temperature threshold, and the load of the terminal device is less than a preset load threshold; or it can generate a third processed image when all three conditions are met. This is not limited in the embodiments of the present application.

[0203] It should also be noted that the terminal device can pre-load images based on user access behavior predictions. For example, a long short-term memory (LSTM) network can be used to analyze the speed and direction of the album scrolling to predict the range of images that may be accessed within the next three seconds, and a preset number of adjacent third processed images can be pre-loaded.

[0204] In another embodiment of the present application, in the process of the terminal device generating a third processed image based on the second intermediate data and the original image data and storing the third processed image, the terminal device can perform full-size processing based on the second intermediate data and the original image data, generate a full-size image, and use the full-size image to replace the standard-size image for storage.

[0205] For example, Figure 5 A full-size image schematic diagram provided for an embodiment of the present application, such as Figure 5 It can be seen that the size of the full-size image is 4416*5888 pixels and the size is 6.9MB.

[0206] It should be noted that full-size images can be compressed and stored in JPEG or HEIF format to balance image quality and file size.

[0207] In this way, LZ4 compression and Advanced Encryption Standard (AES) encryption can be used when uploading full-size images. Combined with cloud-based distributed storage (such as HDFS), the probability of data loss is reduced by two orders of magnitude; illustratively, the distributed storage can be the Hadoop Distributed File System (HDFS).

[0208] It should also be noted that after compression and encryption, the full-size image can be fragmented and uploaded. Accordingly, the cloud server can verify the integrity of the fragments based on MD5 hash and automatically retry in weak network conditions.

[0209] In one possible implementation, when a user enters the photo album interface and clicks on an image, the terminal device prioritizes loading the image at the current browsing location from the storage area for processing. The system then reads the original image data from the disk directory, reuses the secondary intermediate data of the standard-sized image, and invokes the full-scale algorithm model to process the original image data, generating a full-size, clear image of 25MB or 100MB.

[0210] Furthermore, the terminal device can also update the standard-size image in the album to the processed full-size image through the URI replacement mechanism, ensuring that the user sees the full-size processed image.

[0211] It should be noted that, based on the image viewing instruction, the terminal device can pre-load the target image and one image before and after it for processing, so that pre-processing can be performed, thereby improving the user experience.

[0212] In another possible implementation, the terminal device determines whether the current system is in an idle state (the real-time temperature of the terminal device is less than a preset temperature or the load of the terminal device is less than a preset load). If it is in an idle state, the terminal device can batch process pictures that have not yet completed full-size processing.

[0213] It should be noted that if there is second intermediate data, the terminal device can perform full-size processing based on the second intermediate data and the original image data to generate a full-size image. If there is no second intermediate data, the terminal device can directly perform full-size processing on the original image data, that is, regenerate the second intermediate data.

[0214] For example, in actual use, when a user enters an album to view a photo and the user has set the ultra-high-definition mode, the system will check whether the current device meets the second condition. If so, the system will read the second intermediate data generated in the previous processing stage (such as face segmentation masks, color profiles, etc.) and use this data as input to call a more advanced full-scale image processing algorithm to generate a full-size image. Ultimately, the full-size image will replace the original standard-size image and become the final version viewed by the user.

[0215] For example, the full image processing algorithm may include but is not limited to Generative Adversarial Network for Super-Resolution (GAN) and ViT-Large model.

[0216] In an embodiment of the present application, by responding to target shooting instructions when the terminal device meets specific conditions and reusing existing intermediate calculation results for full-size image processing, performance waste caused by repeated calculations can be avoided. At the same time, higher quality image output can be provided according to user needs, thereby achieving dual optimization of image quality and efficiency and improving user experience.

[0217] In another embodiment of the present application, after the terminal device obtains and stores the third processed image, the terminal device can determine the time difference between the last access time of the third processed image and the current time; when the time difference is greater than the preset time, the third processed image is sent to the cloud server and the third processed image stored in the terminal device is deleted.

[0218] Correspondingly, the cloud server can receive the third processed image sent by the terminal device; and store the third processed image in a preset storage area of ​​the cloud server.

[0219] In some embodiments, the most recent access time refers to the time point when the user last viewed or operated the third processed image.

[0220] In some embodiments, the time difference value refers to the interval between the time point when the third processed image was last accessed and the current system time, and is used to measure whether the image has not been used for a long time. The time difference value is typically obtained through access timestamps in system log records or file metadata.

[0221] It should be noted that the time difference is calculated by subtracting the last access time from the current time, and the result is expressed in minutes, hours, or days. For example, if a picture was last viewed at the same time three days ago, the time difference is 72 hours.

[0222] In some embodiments, when the time difference is greater than a preset time, the terminal device may also store the first processed image (ie, the Quick image) corresponding to the third processed image and the index information corresponding to the third processed image in a storage area of ​​the terminal device.

[0223] It should be noted that the preset time can be set based on actual needs, for example, 7 days. If the time difference is greater than 7 days, it indicates that the third processed image has not been accessed for a long time. To facilitate the user's quick access next time, its preview image and index information are stored locally. In this way, the next time the user views the image, the preview image can be loaded directly from the local computer without having to obtain it from the cloud server, which improves loading speed.

[0224] In some embodiments, the index information refers to information containing key metadata about the third processed image.

[0225] Exemplarily, the index information may include but is not limited to a cloud Uniform Resource Locator (URL), a hash value, a thumbnail URI, and the like.

[0226] The cloud URL is used to obtain the full-size image from the cloud server when needed. The terminal device can send a request to the server through the URL to obtain the data of the full-size image.

[0227] The hash value is used to verify the integrity and consistency of the full-size image. For example, when a terminal device obtains a full-size image from a cloud server, it can calculate its hash value and compare it with the pre-stored hash value to ensure that the image has not been damaged or tampered with during transmission.

[0228] Thumbnail URIs are used to display thumbnails locally. Thumbnails quickly provide a general overview of an image, making it easier for users to identify images while browsing. Devices can use thumbnail URIs to retrieve thumbnail data locally or from the cloud and display it on the user interface.

[0229] In other words, the index information provides a unique identifier and access path for the first processed image on the cloud server. Together, they form a lightweight caching mechanism, allowing users to quickly view the image content locally while still being able to access the complete, full-size image at any time through the index information.

[0230] It should also be noted that the terminal device can determine whether to send the full-size image to the cloud server by implementing intelligent cache management based on the Least Recently Used (LRU) algorithm, ensuring that frequently accessed images are retained locally first, while infrequently accessed images are migrated to the cloud server for storage, thereby optimizing the efficiency of device storage space utilization.

[0231] Furthermore, the terminal device can upload the full-size image to the cloud server for storage and management. Meanwhile, the cloud server can receive and store the full-size image sent by the terminal device.

[0232] In some embodiments, the cloud server may use a distributed storage system to store the full-size image and processing metadata.

[0233] Among them, processing metadata refers to key information in the image processing process, which may include feature points, color adjustment parameters, etc.

[0234] It should be noted that to ensure the security and privacy of user data, full-size images and metadata stored in the cloud are encrypted. Encrypted data can only be decrypted and accessed by authorized users. Furthermore, encrypted data is linked to the user's private cloud space, allowing users to conveniently manage and access their image data using their own account and password.

[0235] In an embodiment of the present application, by sending the full-size image to the server for storage when the time difference is greater than the preset time, local storage space can be saved, and the actual available capacity of the 128GB device is increased by 37%, realizing a collaborative storage mechanism between local and cloud, which not only ensures the persistence and security of data, but also avoids excessive occupation of local storage space, thereby supporting larger-scale photo management and long-term preservation needs.

[0236] In an embodiment of the present application, the terminal device may send original image data and associated intermediate data to a cloud server; wherein the intermediate data may include first intermediate data and / or second intermediate data. If the current state of the terminal device satisfies a third condition, an image optimization request is sent to the cloud server.

[0237] Accordingly, the cloud server receives and stores the original image data and associated intermediate data sent by the terminal device. After receiving the image optimization request sent by the terminal device, the cloud server performs image optimization processing on the image to be processed received by the cloud server based on the image optimization request to generate a target image, and then sends the target image to the terminal device.

[0238] Furthermore, the terminal device receives the target image sent by the cloud server based on the cloud server's response to the image optimization request, and stores the target image for display or viewing.

[0239] In some embodiments, the target image may be an image generated by the cloud server performing image optimization processing on the full-size image (third processed image) based on the intermediate data, or may be an image generated by the cloud server performing image optimization processing on the original image data based on the intermediate data.

[0240] It can be understood that an image optimization request refers to a request message sent by a terminal device to a cloud server, the purpose of which is to request the cloud server to perform image optimization processing on a specific image (such as an image to be processed or original image data).

[0241] It should be noted that the resolution of the target image is higher than the resolution of the third processed image.

[0242] In some embodiments, the current state of the terminal device satisfies a third condition, including at least one of the following: responding to a first user operation; the terminal device is in a charging state and the power of the terminal device is greater than a preset power threshold; the terminal device is connected to a wireless network.

[0243] The first user action refers to an action taken by the user on the terminal device to enable or activate the "Cloud Optimization" feature. This feature allows the terminal device to delegate image optimization processing to a cloud server, rather than performing it locally on the terminal device.

[0244] It should be noted that the activation / deactivation of the first user operation can be controlled by a visual control, which can be displayed in the user interface. For example, Figure 6 A schematic diagram of a first user operation to start / close a visual control provided in an embodiment of the present application.

[0245] refer to Figure 6 The user can click "Album" in the user interface, and then click "Auto-sync Album" in the next user interface (that is, the first user operation in the embodiment), so that the terminal device can send an image optimization request to the cloud server in response to the first user operation.

[0246] In some embodiments, a wireless network refers to a type of network that accesses the Internet via Wi-Fi or other non-cellular data communication methods.

[0247] Compared to mobile data networks, wireless networks offer higher bandwidth, lower latency, and more stable connections, making them particularly suitable for data-intensive operations like large file transfers and cloud processing. This means that cloud-based image optimization tasks will only be initiated when the device is connected to a wireless network, avoiding high mobile data charges and impacting the user's other network experiences.

[0248] For example, the terminal device can detect whether it has received a first user operation; secondly, determine whether the current terminal device battery level has reached a preset level; and finally, confirm whether the terminal is connected to a wireless network. Only when the above three conditions are met at the same time will the image optimization task be initiated.

[0249] In another exemplary embodiment, after any one of the above third conditions is met, the terminal device will start the image optimization task.

[0250] In this embodiment, by setting one of the three conditions mentioned above as a trigger, the system can dynamically adjust the task execution strategy based on the current environment and user intent, achieving optimal resource allocation and utilization. This improves image processing efficiency, reduces power consumption, and enhances end-cloud collaboration without affecting the user experience.

[0251] In another embodiment of the present application, an image processing method is provided, which can be applied to a cloud server.

[0252] In the embodiments of this application, Figure 7 Schematic diagram of the image processing method implementation process provided in the embodiment of the application Figure 2 ,like Figure 7 As shown, the image processing method may include the following steps:

[0253] Step 701: Receive an image optimization request sent by a terminal device.

[0254] In some embodiments, an image optimization request refers to a request message sent by a terminal device to a cloud server, the purpose of which is to request the cloud server to perform image optimization processing on a specific image (such as an image to be processed or original image data).

[0255] Step 702: Based on the image optimization request, perform image optimization processing on the image to be processed received by the cloud server to generate a target image.

[0256] In some embodiments, image optimization processing refers to the process in which a cloud server, based on a request from a terminal device, calls a large model algorithm or computing resources to automatically enhance, repair, or stylize an image to generate a target image with higher quality or that better meets the requirements.

[0257] Among them, the large model algorithm can be an artificial intelligence (AI) large model, such as using the ViT-Large model to perform dehazing, color calibration and dynamic range expansion.

[0258] Exemplarily, after receiving the image optimization request sent by the terminal device, the cloud server performs image optimization processing on the image to be processed received by the cloud server in response to the image optimization request, thereby generating a target image.

[0259] Step 703: Send the target image to the terminal device.

[0260] In an embodiment of the present application, after generating the target image, the cloud server may send the target image to the terminal device so that the terminal device can use the target image to replace the full-size image for storage.

[0261] In the embodiment of the present application, a cloud server is introduced. With the help of the powerful computing resources and advanced large-model algorithms of the cloud server, complex image optimization tasks can be completed efficiently and accurately, which greatly improves the efficiency and quality of image processing and effectively makes up for the shortcomings of the terminal device in computing power and algorithm complexity; at the same time, the terminal device can directly use the optimized target image to replace the full-size image for storage, saving local storage space.

[0262] It should be noted that steps 201 to 203 are explained in detail in the above-mentioned application embodiment and will not be repeated here for the sake of brevity.

[0263] In an embodiment of the present application, there are two ways to perform image optimization processing on the image to be processed received by the cloud server to generate a target image. The following describes these two implementation methods in detail.

[0264] In a possible implementation, a cloud server receives original image data and associated intermediate data sent by a terminal device; performs image optimization processing on the original image data based on the intermediate data to generate a target image.

[0265] Among them, the intermediate data includes first intermediate data and / or second intermediate data. The first intermediate data is obtained by the terminal device performing image processing based on the reference image, and the second intermediate data is obtained by the terminal device performing image processing based on the first intermediate data and the original image data.

[0266] In some embodiments, after the terminal device writes the original image data and the intermediate data into the storage area, the cloud server receives the original image data and the intermediate data sent by the terminal device and stores them accordingly.

[0267] Furthermore, after the cloud server receives the image optimization request sent by the terminal device, it can obtain the original image data and intermediate data from the storage area of ​​the cloud server, and perform image optimization processing on the original image data based on the intermediate data to generate a target image.

[0268] That is to say, since the previous algorithm has written the original image data and intermediate data into the storage area for persistent storage, the cloud server can directly read the original image data from the storage area, and perform full-scale algorithm and cloud-based large-model algorithm optimization on the original image data directly on the cloud server to generate the target image.

[0269] In this way, the terminal device only needs to perform lightweight processing and offload computing-intensive tasks to the cloud. The cloud server can directly use the original image data and intermediate data to perform full algorithm optimization in the cloud, giving full play to the advantages of the cloud's powerful computing power and large model algorithms, ensuring that the generated target image is of higher quality and better effect.

[0270] In another possible implementation, the cloud server receives the third processed image sent by the terminal device; stores the third processed image in a preset storage area of ​​the cloud server; and performs image optimization processing on the third processed image based on the intermediate data to generate a target image.

[0271] In some embodiments, after the terminal device generates the third processed image, if the time difference between the last access time of the third processed image and the current time is greater than a preset time, the third processed image can be sent to the cloud server. Accordingly, the cloud server receives the third processed image sent by the terminal device and stores it.

[0272] Furthermore, after the cloud server receives the image optimization request sent by the terminal device, it can obtain the third processed image from the storage area of ​​the cloud server, and perform image optimization processing on the third processed image based on the intermediate data to generate a target image.

[0273] In other words, the cloud server can use the cloud-based large model to perform refined processing (including but not limited to defogging, de-reflection, and dynamic range expansion) based on the third processed image generated on the terminal side. It should be noted that the terminal device can retain color parameters for reference by the cloud server.

[0274] In this way, the cloud server uses the cloud large model to perform refined processing based on the third processed image, which can deeply optimize the details and overall effect of the image and further improve the image quality; and the terminal device retains the color parameters for cloud reference, so that the cloud optimization processing can better combine the processing intentions of the terminal device and user preferences to generate a target image that better meets user needs.

[0275] For example, Figure 8 This is a schematic diagram of a cloud server optimization processing strategy provided in an embodiment of the present application. Figure 8 As shown, the following steps may be included:

[0276] S801. The camera application layer APP sends preview and photo taking requests to the camera hardware layer.

[0277] S802: The camera hardware layer sends a preview frame and a photo frame to the camera application layer in response to the preview and photo request.

[0278] S803: The terminal device stores the original image data (also referred to as the original photographed frame) and the intermediate data (ie, related parameters) into a storage area for persistent storage.

[0279] S804: compress the original image data and the intermediate data and upload them to the corresponding storage area of ​​the cloud server (ie, the user's private cloud space).

[0280] S805. The cloud server performs image optimization processing on the full-size image based on the intermediate data.

[0281] S806. The cloud server can generate a target image based on the generated high-resolution full-size image and the incremental computing power of the cloud large model, and store it in the album.

[0282] S807: The cloud server performs image optimization processing on the original image data based on the intermediate data.

[0283] S808. The cloud server can recalculate based on the original image data and the cloud large model computing power, generate the target image, and store it in the album.

[0284] S809: When the time difference between the most recent access time of the full-size image and the current time is greater than a preset time, the full-size image is sent to the cloud server, and the full-size image stored in the terminal device is deleted.

[0285] It should be noted that the terminal device only saves the Quick image, which is then downloaded from the cloud server when the user views it.

[0286] It should be noted that, when the cloud server processes and generates the target image, a processing waiting diagram will be displayed on the corresponding image. For example, Figure 9 Schematic diagram of image optimization processing performed by a cloud server provided in an embodiment of the present application; Figure 9 As shown, it can be seen that at this time, the cloud server is performing image optimization processing on the received image to be processed to generate a target image.

[0287] In an embodiment of the present application, the cloud server receives an image access instruction sent by a terminal device; based on the image access instruction, the first full-size image currently being accessed is sent to the terminal device, and at least two adjacent images of the first full-size image are cached at the edge node.

[0288] In some embodiments, the image access instruction refers to a request triggered by a user operation or automatically by the system, and is used to indicate which image needs to be accessed currently.

[0289] For example, the image access instruction may include identification information of the target image (such as file name, URI, etc.), as well as possible additional operation parameters (such as zoom ratio, preloading adjacent images, etc.). For example, when a user clicks on a photo in an album, the client generates an image access instruction and sends it to the server or edge node to obtain the corresponding full-size image.

[0290] In some embodiments, the first full-size image refers to a high-quality original image that the user currently requests to access, and its resolution is much higher than that of the Quick image or the medium-size image.

[0291] In some embodiments, edge nodes refer to computing and storage resources deployed at the edge of the network, close to terminal devices, responsible for caching commonly used data and performing lightweight processing tasks to reduce dependence on central cloud servers and improve access speed.

[0292] It should be noted that edge nodes not only perform data caching functions, but also support preloading strategies based on user behavior predictions to improve the overall access experience.

[0293] For example, before sending the first full-size image, the cloud server can search for the corresponding storage location (local, cloud, or edge node) based on the user's request path, read it, and transmit it to the terminal device via the network. At the same time, to improve the efficiency of subsequent access, the system will cache at least two images adjacent to the current image (i.e., the previous and next images) at the edge node, so that the user can quickly respond when scrolling and browsing, reducing latency.

[0294] In the embodiment of the present application, by receiving an image access instruction from a terminal device and sending the first full-size image currently being accessed to the terminal device based on the instruction, at the same time, at least two adjacent images are cached at the edge node. This significantly reduces the waiting time for users to access adjacent images, thereby improving the smoothness of album browsing and further optimizing the user experience.

[0295] The present application provides an image processing method, which is applied to a terminal device. The terminal device performs lightweight processing on a reference image to generate first intermediate data and a first processed image. Then, if the terminal device meets a first condition, the first intermediate data is combined with the original image data to generate and store a second processed image. On the one hand, the lightweight processing and intermediate result reuse mechanism avoids repeated calculations, significantly reducing computing power and energy consumption. On the other hand, by triggering the generation of the second processed image on demand, system resource scheduling is optimized, thereby improving image processing efficiency and user experience.

[0296] Based on the above embodiments, another embodiment of the present application proposes an image processing method. To address the above problems, an image processing flow optimization solution for cross-stage reuse of intermediate data is designed.

[0297] In the embodiments of this application, Figure 10 A schematic diagram of the implementation framework for cross-stage reuse of intermediate data provided in the embodiment of the present application is shown in FIG. Figure 10 As shown, the following steps are included:

[0298] S1001. The camera application layer APP sends preview and photo taking requests to the camera hardware layer.

[0299] S1002: The camera hardware layer sends a preview frame and a photo frame to the camera application layer in response to the preview and photo request.

[0300] S1003: The terminal device captures the preview frame at the time corresponding to the reference frame as a Temp image (ie, the reference image in the above embodiment), and stores it in the album database.

[0301] It should be noted that the terminal device may also select a preset algorithm to process the Temp image as needed. For example, the preset algorithm may be a superimposed watermark algorithm.

[0302] S1004: The terminal device sends the photographed reference frame to a lightweight algorithm for processing in real time, which can also be called lightweight algorithm post-processing.

[0303] S1005: The terminal device performs a lightweight algorithm post-processing on the reference image to obtain a Quick image (ie, the first processed image in the above embodiment) and first intermediate data, and stores them in the album database in place of the Temp image.

[0304] It should be noted that the terminal device may also select a preset algorithm to process the Quick image as needed. For example, the preset algorithm may be a superimposed watermark algorithm.

[0305] S1006: The terminal device stores the original image data and related parameter data into a storage area for persistent storage.

[0306] S1007: When the user is viewing the corresponding image or the mobile phone is idle, image-related data is read from the storage medium and the algorithm is actively activated for post-processing optimization.

[0307] S1008: When the current state of the terminal device satisfies the first condition, read the original image data and the first intermediate data from the storage area and perform standard algorithm post-processing.

[0308] S1009. The terminal device performs standard algorithm post-processing based on the original image data and the first intermediate data to obtain a standard size image (that is, the second processed image in the above embodiment) and the second intermediate data, and replaces the Quick image in the album database.

[0309] It should be noted that the terminal device stores the second intermediate data in the storage area for persistent storage.

[0310] S1010: When the current state of the terminal device satisfies a second condition, read the original image data and the second intermediate data from the storage area and perform full-size algorithm post-processing.

[0311] S1011. The terminal device performs full-size algorithm post-processing based on the original image data and the second intermediate data to obtain a full-size image (that is, the third processed image in the above embodiment), and replaces the standard-size image in the album database.

[0312] S1012: When the current state of the terminal device satisfies the third condition, the original image data and intermediate data are read from the storage area and processed in a large model in a cloud server (ie, the user's private cloud space).

[0313] Exemplarily, an image optimization request is sent to a cloud server; based on the cloud server's response to the image optimization request, a target image sent by the cloud server is received, and the target image is stored.

[0314] S1013. Based on the image optimization request, the cloud server performs image optimization processing (i.e., large model refinement processing) on ​​the received image to be processed, generates a target image (i.e., the final full-size and full-algorithm image), and sends the target image to the terminal device and stores it in the album database.

[0315] S1014: If the user has accessed the corresponding image within a short period of time (e.g., 7 days), the final image will be stored in the local space of the device; if the user has not viewed the corresponding image for a long period of time (e.g., more than 7 days), the full-size and full-algorithm final image will be stored in the user's private cloud space.

[0316] Only Quick images that take up very little storage space are retained locally, and the current and adjacent photos can be loaded from the cloud when the user accesses them again; this can be achieved by combining the LRU algorithm.

[0317] It should be noted that in the embodiments of this application, a unified heterogeneous computing resource pool (CPU / GPU / NPU / cloud) can be constructed through hardware resource pooling to support dynamic task allocation. Specifically, on the terminal device side, the NPU performs high-real-time tasks (such as face detection), the GPU handles computationally intensive tasks (such as GAN super-resolution), and the CPU manages I / O and scheduling;

[0318] For the cloud server side, during idle periods, the cloud-based large model is called to combine the intermediate results of the end-side algorithm and the migratable parameters for fine-grained optimization.

[0319] It's also worth noting that when the device temperature exceeds 50°C, image quality enhancement and algorithm post-processing automatically cease to maintain a basic user experience. When the battery level drops below 15%, non-critical cloud tasks are suspended, prioritizing communication and display functions. Compared to the real-time image processing solution in Related Technology 1, peak phone temperatures are reduced by approximately 3°C.

[0320] It should also be noted that the terminal device can also set a priority queue; for example, user-active viewing tasks (priority 1) > idle batch processing (priority 2) > cloud-based asynchronous tasks (priority 3).

[0321] For example, when the user scrolls through the album, the large-size processing of the remaining images in the background is suspended, and the current user's focused photo and the two photos on the left and right are loaded and processed from the storage medium first; when the device is idle, unfinished tasks are batch processed.

[0322] according to Figure 10 As shown in the schematic diagram of the framework, the beneficial effects achieved by the embodiment of the present application include at least the following:

[0323] 1. Significantly reduce computing redundancy and energy consumption.

[0324] Through the cross-stage reuse mechanism of intermediate results (including but not limited to the persistent storage of original image data parameters, facial feature points, and segmentation masks), there is no need to repeatedly execute the feature extraction algorithm in the subsequent processing stages. For example, medium-sized processing directly calls the facial coordinate data and segmentation results of the Quick image, reducing the computational time by 30%. Compared with traditional solutions (such as multi-stage fusion technology that does not reuse intermediate data), this solution can reduce the peak temperature of the device by 3°C when processing 50 images in batches.

[0325] 2. Optimize storage space occupancy and access efficiency.

[0326] For historical images that users are unlikely to access, only Quick images (average 4.2MB / image) and index files are stored locally, while full-size images (18.6MB+) are stored in the cloud. Compared to Related Technology 1 (images must be retained locally, 18.6MB / image), the storage space savings rate reaches 77.4% (18.6-4.2) / 18.6) (standard deviation ±8%). At the same time, the mechanism of preloading two adjacent full-size images reduces the delay in sliding through the album from the traditional 500ms to less than 200ms, improving access fluency by 60%.

[0327] 3. Dynamic resource scheduling improves system efficiency.

[0328] An intelligent trigger algorithm post-processing strategy based on device status (select strategies include temperature <45°C, CPU load <80%, and charging status) dynamically allocates task priorities. Compared to the static scheduling strategy of Related Technology 1, CPU peak performance is reduced by over 50%, and heterogeneous computing power utilization, such as CPUs and NPUs, is improved. Due to the reduction of batch congestion and multi-threaded scheduling wait times for runnables, the actual task completion speed is increased by more than 2 times.

[0329] In the embodiments of this application, Figure 11 This is a schematic diagram of the phased processing flow provided in the embodiment of this application. Figure 11 As shown, the following processes are involved:

[0330] S1101, capturing the moment of taking a photo and displaying a preview.

[0331] (1) Preview frame capture: Use the Camera2 API to capture the YUV420 preview frame output by the sensor (delay < 50ms), cache the preview buffer in shared memory, and match the corresponding preview frame with the timestamp of the reference frame;

[0332] (2) Memory optimization: Maintain a fixed-size cache queue. Preview buffers that exceed the cache are released directly. Only the preview buffer corresponding to the timestamp of the base frame will be used for further processing.

[0333] S1102: Generate a Quick image based on the reference frame:

[0334] (1) Lightweight algorithm: A lightweight algorithm is used to process the reference photo frame and perform face detection, scene segmentation, basic noise reduction and 3A algorithm;

[0335] (2) Intermediate result storage: write the facial feature point coordinates and segmentation mask (such as U-Net output) into the database and disk cache;

[0336] (3) Output optimization: Generate 12M or 16M Quick images, replace the album thumbnail URI, and users can view them quickly after clicking to take a photo.

[0337] S1103, standard size drawing generation (12M / 16M, triggered on demand)

[0338] (1) Trigger conditions:

[0339] User-triggered: When entering the album to view a specific picture, the current browsing position and one picture before and after it are loaded and processed from the storage medium first;

[0340] System Idle Trigger: Batch process unfinished images when the device is idle (e.g., temperature < 45°C, CPU load < 80%).

[0341] (2) Data processing:

[0342] Read raw data from the disk directory, reuse the intermediate results of the Quick graph (such as face region coordinates, image segmentation results), and skip repeated detection steps (reducing computational time by 30%);

[0343] Algorithm: Call the standard algorithm model to perform standard algorithm processing (such as turboraw, hybridraw and other raw domain multi-frame optimization algorithms) to generate a medium standard size image of 12M or 16M;

[0344] Storage update: Updated the album Quick images to medium-sized images through the URI replacement mechanism.

[0345] S1104, full-size image generation (25M / 100M, triggered on demand)

[0346] (1) Triggering scenario:

[0347] The user selected "Ultra HD" shooting in the camera settings;

[0348] When the system is idle, it will automatically process all medium-sized images and then continue to process full-sized images;

[0349] When the user pauses to view the corresponding photos, the full-size processing of the corresponding number of photos will be directly triggered.

[0350] (2) Data processing:

[0351] Reuse of intermediate results: Load the segmentation mask for medium-sized processing and use the full-scale algorithm for optimization;

[0352] Resource scheduling: Pause tasks when CPU or GPU load exceeds 90%, giving priority to ensuring smooth user interaction.

[0353] Storage strategy: Generate a 25MB or 100MB full-size, clear image and update the album URI to point to the full-size image.

[0354] S1105, cloud optimization processing (intelligent triggering)

[0355] (1) Trigger conditions:

[0356] The user has enabled the "Cloud Optimization" function and the device is charging (battery level > 80%) and connected to Wi-Fi (or mobile network enabled);

[0357] (2) Cloud Optimization:

[0358] Model call: Deploy large AI models (such as the ViT-Large model) for dehazing, color calibration, and dynamic range expansion;

[0359] Intermediate data synchronization: The client uploads original image data, segmentation masks, and image color parameters to ensure that the cloud output is consistent with the local style.

[0360] (3) Preloading strategy: The edge node caches two adjacent full-size images in the user's photo album, with an access hit rate of >90%.

[0361] In an embodiment of the present application, the time difference between the last access time of the full-size image and the current time is determined; if the time difference is greater than the preset time, the full-size image is sent to the cloud server and the full-size image stored in the terminal device is deleted.

[0362] In the above embodiment, since the reference frames in each stage are fixed and the intermediate algorithm processing results are reused, the user can switch from Temp image - Quick image - standard size image - full-size HD image - full-size full-algorithm large model optimized image without any jumps in the entire link, and can feel the improvement in image quality and details.

[0363] In some embodiments, the cloud server uses distributed storage (such as HDFS) to save the full-size image and processing metadata (feature points, color parameters), which are then encrypted and associated with the user's private cloud space;

[0364] In some embodiments, the terminal device only retains the Quick image (4.2MB / image) and index file (including cloud URL, hash value, and thumbnail URI).

[0365] At the same time, when users change their terminal devices (such as mobile phones) to cloud-stored user data, they will still consider the same brand of terminal devices to improve user retention rate, considering that a large amount of cloud-stored data will still be limited.

[0366] In some embodiments, the full-size image is retained locally for a specific time (such as seven days) and then automatically deleted, while the cloud version is permanently stored; when a user accesses a cloud image, the current and two adjacent high-definition images (JPEG or HEIF format) are automatically downloaded, and the LRU algorithm manages the local cache.

[0367] Next, the LRU algorithm is introduced in the embodiments of this application.

[0368] A. Core idea of ​​the algorithm.

[0369] LRU (Least Recently Used) is a data elimination strategy based on access history. The core idea is that if data has been accessed recently, it is more likely to be accessed in the future. Conversely, if data has not been accessed for a long time, it may be eliminated to free up space.

[0370] In local cache management, LRU tracks the access time sequence of data to ensure that hot data with high frequency of access is retained in the cache, while cold data with low frequency of access is eliminated, thereby improving the cache hit rate and optimizing the use of storage resources.

[0371] B. How the algorithm works.

[0372] LRU implements data management through the following steps:

[0373] Access record maintenance: Use a bidirectional linked list to maintain the data access order. The head of the linked list stores the most recently accessed data, and the tail stores the data that has not been accessed for the longest time.

[0374] Combined with the hash table (Key-Value structure), fast data positioning is achieved, and the Value of the hash table points to the corresponding node in the linked list.

[0375] Data access and update: Reading data (Get operation): If the data exists, quickly locate the node through the hash table and move it to the head of the linked list (marked as the most recently used). Writing data (Put operation): If the data exists, update the value and move it to the head; if it does not exist and the cache is not full, insert the new node to the head; if the cache is full, eliminate the tail node (the least recently used) and insert the new data.

[0376] Time complexity: The hash table achieves O(1) search speed, the doubly linked list achieves O(1) insertion and deletion operations, and the hash table + doubly linked list achieves O(1) time complexity, which is suitable for high concurrency scenarios.

[0377] C. Advantages of LRU application in this application.

[0378] Efficient use of storage space: Local cache capacity is limited. LRU prioritizes high-frequency data by eliminating low-frequency data, reducing redundant storage. For example, only thumbnails (Quick images) are retained in the mobile device album, and full-size images are loaded or uploaded to the cloud on demand.

[0379] Adapting to the principle of locality: Programs exhibit temporal locality (recently accessed data is likely to be accessed again) and spatial locality (nearby accessed data is likely to be accessed consecutively). LRU adheres to this principle by maintaining access order, improving cache hit rates.

[0380] Dynamic resource scheduling: Trigger processing on demand based on device status (such as CPU load and temperature). For example, batch-generate medium-sized images when the phone is idle, and use cloud computing power to optimize images while charging to avoid resource contention.

[0381] End-to-end collaborative optimization: Only lightweight data (such as Quick Images) is stored locally, while full-size data is uploaded to the cloud. This data is downloaded on demand when users access it, and latency is reduced by preloading adjacent data (such as the left and right images in an album).

[0382] In the embodiments of this application, Figure 12 The terminal device includes a real-time processing layer and an idle processing layer, and the cloud server includes a storage layer and a computing layer. Figure 12 As shown, the execution steps are as follows:

[0383] S1201. The terminal device monitors the device status in real time.

[0384] For example, the device status may include click status, temperature, power, memory, CPU load, hard disk space, etc.

[0385] S1202. If the device state is the user click state, the terminal device determines the moment when the user takes the photo, and then executes step S1204.

[0386] S1203. If the device status is the time when the user is viewing photos or the device is idle, execute step S1207.

[0387] S1204: The terminal device captures the preview frame corresponding to the time of the reference frame as a Temp image based on the moment when the user takes the photo, and stores it in the album database.

[0388] S1205: The terminal device performs a lightweight algorithm post-processing on the reference image to obtain a Quick image (ie, the first processed image in the above embodiment) and first intermediate data, and stores them in the album database in place of the Temp image.

[0389] S1206: The terminal device stores the original image data and the intermediate algorithm processing results into a storage area (ie, a storage medium) for persistent storage.

[0390] Exemplarily, the intermediate algorithm processing result includes first intermediate data and / or second intermediate data.

[0391] S1207. When the user views the photo or the terminal device is idle, the original image data and intermediate data are read from the storage area for algorithm post-processing.

[0392] S1208. The terminal device performs standard algorithm post-processing based on the original image data and the first intermediate data to obtain a standard size image (that is, the second processed image in the above embodiment) and the second intermediate data, and replaces the Quick image in the album database.

[0393] It should be noted that the terminal device stores the second intermediate data in the storage area for persistent storage.

[0394] S1209: The terminal device performs full-size algorithm post-processing based on the original image data and the second intermediate data to obtain a full-size image (that is, the third processed image in the above embodiment), and replaces the standard-size image in the album database.

[0395] S1210: compress the original image data and the intermediate data and upload them to the corresponding storage area of ​​the cloud server.

[0396] It should be noted that the cloud-side model input layer is compatible with the client-side algorithm processing intermediate results; that is, the color profile and tone mapping parameters (TMC parameters) can be synchronously updated to the cloud server.

[0397] S1211. Determine whether the current state of the terminal device satisfies the third condition. When the current state of the terminal device satisfies the third condition, read the original image data and intermediate data from the storage area and perform large model refinement processing on the computing layer.

[0398] S1212: Send the target image to the terminal device.

[0399] Correspondingly, the terminal device receives the target image sent by the cloud server and stores the target image.

[0400] It should be noted that, from Figure 12 As can be seen from the figure, the cloud server sends the first full-size image currently being accessed to the terminal device based on the image access instruction, and caches at least two adjacent images of the first full-size image at the edge node.

[0401] Next, the examples of the present application are introduced to verify the superiority of the method proposed in the present application through comparative experiments.

[0402] In some embodiments, the experimental environment parameters include but are not limited to test model, Android version, test scenario, test method and environment parameters.

[0403] Specifically, the test model for this application is test model A (Snapdragon 8Gen3, 16GB RAM); the test model for the control group is also test model A (Snapdragon 8Gen3, 16GB RAM); the Android version is Android 15; the test scene is 50 continuous shots (High Dynamic Range (HDR) + Night Scene mode mixed load); the test method is to use Android Profiler to track CPU / GPU / NPU utilization and Perfetto to record task scheduling time. The environmental parameters are room temperature of 25°C, screen brightness is fixed at 50%, and irrelevant background processes are closed.

[0404] For example, Figure 13 This is a comparison diagram of the reuse rate of intermediate results of this application and the control group. The control group includes related technology 1 and related technology 2. Figure 13 It can be seen that the intermediate result reuse rate of the present application is significantly higher than that of the related technology 1 and the related technology 2.

[0405] Among them, related technology 1 adopts a single-end processing solution, that is, the current image is processed immediately on the mobile device side after the user takes a photo, and the user can only see the image generation after the processing is completed.

[0406] Related technology 2 adopts multi-stage fusion technology to improve the imaging speed through a staged process of synthesizing preview frame → quick image (Quick image) → final image (Final image). Furthermore, when the user triggers the instruction for viewing the target image, the preview image can be displayed to the user first. After the target image is synthesized, the target image is displayed to cover the preview image.

[0407] For example, Figure 14 This is a schematic diagram comparing the local storage space occupied by a single photo taken by this application and the control group. The control group includes related technology 1 and related technology 2. Figure 14 It can be seen that the local storage space occupied by a single photo taken in this application is significantly lower than the local storage space occupied by a single photo taken in related technologies 1 and 2 at the time of shooting, 1 day after shooting, and 7 days after shooting.

[0408] For example, Table 1 is a comparison result of various quantitative indicators of the present application and related technologies 1 and 2. As can be seen from Table 1, various indicators of the present application are significantly better than those of related technologies 1 and 2.

[0409] Table 1

[0410]

[0411] For example, Table 2 shows the comparison results of various quantitative indicators of the present application and Related Art 1. Table 2 provides the test methods used for various indicators of the present application and Related Art 1, as well as the comparison results of various quantitative indicators. As can be seen from Table 2, under the same test methods, various indicators of the present application are significantly superior to those of Related Art 1.

[0412] Table 2

[0413]

[0414] For example, Table 3 shows the comparison results of various quantitative indicators of the present application and related technology 2. As can be seen from Table 3, various indicators of the present application are significantly better than those of related technology 2.

[0415] Table 3

[0416]

[0417] It can be seen from Tables 1, 2 and 3 above that this application achieves a balance between image quality, efficiency and cost in mobile photography scenarios through the core architecture of intermediate result reuse, phased dynamic scheduling based on user behavior prediction and device status, peak-shifting post-processing, and end-cloud collaboration.

[0418] In some embodiments, the present application adopts a three-level trigger logic, specifically including the following steps:

[0419] (1) Real-time layer: Preview frame capture and Quick image generation are forced to be completed in real time (<50ms), and only the latest matching preview frame data is retained in the memory.

[0420] (2) Idle time layer: Dynamically schedules medium / large-scale processing tasks based on device temperature (<45°C), battery level (>30%), and network status (Wi-Fi / 5G), increasing NPU utilization to 85%.

[0421] (3) Intent prediction layer: Through user behavior analysis (such as album sliding speed and dwell time), preload three adjacent full-size images (hit rate > 90%).

[0422] In some embodiments, the present application also employs an energy optimization model, specifically a task scheduler based on user behavior prediction, dynamically balancing image quality and power consumption. Experiments have shown that when taking 50 consecutive photos, the peak temperature can be reduced by approximately 3°C. However, related technologies do not incorporate user intent prediction and rely solely on waiting for real-time image processing to complete, resulting in poor user perception.

[0423] In some embodiments, a heterogeneous architecture combining end-side NPU / GPU / CPU collaboration and cloud computing clusters is used to build a multi-layer computing resource pooling scheduling framework.

[0424] In some embodiments, the end-side hardware is divided into two parts: the NPU is dedicated to real-time tasks (such as face detection and preview frame processing), the GPU processes medium-sized image super-resolution reconstruction (U-Net model acceleration), the CPU is responsible for I / O scheduling, and the memory, hard disk and network disk are responsible for persistent storage of intermediate results.

[0425] In some embodiments, cloud computing power is integrated, and under charging / high power conditions, the cloud ViT-Large model is called for refined processing, and a lightweight inference engine (such as a TensorRT optimization model) is deployed on the edge node to reduce processing latency.

[0426] In some embodiments, dynamic energy efficiency control dynamically adjusts the computing power allocation strategy based on real-time feedback of device temperature (threshold 45°C) and power consumption (<20% frequency reduction).

[0427] In some embodiments, an intermediate data association model based on a graph neural network (GNN) is constructed to achieve dynamic reuse of feature points, segmentation masks, and optical flow parameters:

[0428] In some embodiments, version-aware storage uses hash check (SHA-256) + timestamp to mark the intermediate result version, is compatible with algorithm iterative upgrades (such as MobileNet V3→V4), automatically triggers incremental updates and retains the old version interface.

[0429] In some embodiments, semantic association retrieval uses a GNN model to establish a mapping relationship between feature points and scene semantics (such as the portrait mode prioritizes the reuse of a 68-point face model) to improve the retrieval hit rate.

[0430] In some embodiments, memory-disk hierarchical caching is used, high-frequency access data (such as the segmentation results of the last five images) is stored in the memory device (response <10ms), and historical data is stored in the hard disk device by time partition, supporting millisecond-level retrieval.

[0431] In some embodiments, a preloading engine based on user behavior prediction and spatial topology awareness is designed to achieve dual optimization of storage efficiency and access latency: sliding intention modeling, analyzing the album sliding speed and direction through the LSTM network, predicting the range of pictures that may be accessed in the next 3 seconds, and loading 5 adjacent full-size pictures in advance.

[0432] In some embodiments, hot and cold data are layered, Quick images (4.2MB / image) and the 10 most recent full-size images are retained locally, and data that has not been accessed for 7 days is automatically migrated to the cloud-based Glacier cold storage and quickly returned to the source through the CDN edge node when accessed.

[0433] In some embodiments, the defragmentation algorithm uses an improved LRU-K strategy to manage the local cache, identify infrequently accessed images and trigger cloud backup, thereby improving local storage space utilization.

[0434] In some embodiments, a multi-stage jointly optimized end-cloud processing chain is established to achieve a graded leap in image quality, end-side preprocessing, and the ISP image processing chip is used to perform RAW data noise reduction and dynamic range expansion, and metadata (color profile, exposure parameters) are extracted simultaneously when generating 12M Quick images.

[0435] In some embodiments, cloud-based refinement uses the ViT-Large model for semantic segmentation-guided super-resolution reconstruction (4x magnification), combined with the segmentation mask uploaded on the client side to achieve local enhancement (such as hair detail restoration).

[0436] In some embodiments, style consistency is guaranteed by deploying a generative adversarial network (GAN) to align the color styles of end-cloud processing, ensuring that the visual transition between the Quick image and the cloud-based refined image is natural and imperceptible.

[0437] In some embodiments, secure and reliable data encapsulation and transmission protocols build a cross-platform data security system under a zero-trust architecture, covering full life cycle protection.

[0438] In some embodiments, blockchain evidence is stored, a unique hash fingerprint is generated for each image and uploaded to the chain (Hyperledger Fabric), so that the processing process is traceable and the results cannot be tampered with.

[0439] In some embodiments, a dynamic fragmentation strategy automatically adjusts the data fragment size (1MB to 10MB) according to the network quality (5G / Wi-Fi), and adopts FEC forward error correction coding in weak network environments to improve the transmission success rate.

[0440] In summary, the embodiments of the present application may include the following aspects:

[0441] (1) Reuse of intermediate results across stages: The original image data and intermediate calculation data such as facial feature points and segmentation masks are persistently stored, supporting subsequent read calls for reuse;

[0442] (2) Persistent storage: The original image data and meta parameter information from the sensor are written to the storage medium for persistent storage, which is used for subsequent on-demand intelligent triggering processing.

[0443] (3) Intelligent triggering mechanism: Dynamically schedule tasks based on device status (temperature, load, power, hardware computing power status, etc.) and user behavior;

[0444] (4) User-unaware image refinement in stages: from a preview Temp image based on the preview timestamp → a photo Quick image based on the photo reference frame → a standard large image based on a standard size → a refined large image based on a full size → a full-size, full-algorithm final image refined based on a large model with cloud computing power.

[0445] To ensure that users can see the pictures they take immediately, we continuously superimpose the required algorithms, and utilize the advantageous computing power of each module (CPU, GPU, NPU, and large cloud models) in a staggered manner to perform refined grading, polishing, and optimization for the final formation of the pictures.

[0446] (5) End-cloud collaborative storage and processing strategy: For photos that users will not view in the short term (e.g., photos that have not been accessed for more than 7 days), only Quick images are retained locally, and full-size images are stored on demand in the cloud. When the user subsequently accesses the current frame, the image is loaded from the cloud and adjacent frames are preloaded. If the user views the image multiple times (e.g., more than twice), the image is directly stored locally). Cloud computing power can be used to process the user's photos.

[0447] Thus, the embodiment of the present application proposes a hierarchical storage architecture that combines intermediate result reuse, algorithm peak-shifting post-processing, phased dynamic scheduling based on user behavior prediction and device status, and end-cloud collaborative storage to achieve the following effects:

[0448] ① Reduced computing redundancy, and the intermediate data reuse rate increased from an average of 0% to 30%;

[0449] ② Reduced end-side storage costs, with the storage space occupied by a single image reduced by 77%;

[0450] ③ End-cloud collaboration efficiency: Expand from existing pure end-side real-time processing to end-cloud collaboration, and optimize cloud access latency through behavior prediction;

[0451] ④Power consumption optimization: Under the same task load, the peak temperature of the device is reduced by 3°C;

[0452] ⑤Algorithm processing bottleneck: Existing solutions all rely on real-time processing on the end-side, which is constrained by the computing power bottleneck of mobile phone hardware. This solution introduces algorithm staggered post-processing to reduce peaks and fill valleys, making full use of the computing power of mobile devices during idle time to avoid congestion. At the same time, it connects to a large cloud model and can use the massive computing power of the cloud to optimize images, breaking through the computing power bottleneck of mobile devices on the end-side.

[0453] In other words, this application has built a multi-level progressive image processing framework, combined with real-time processing on the end side, on-demand staggered triggering and cloud-based collaborative optimization, to realize the dynamic generation and storage management process from the user clicking to take a photo, from seeing the preview image to the final full-size high-definition image, and adopts progressive processing + cloud-based linkage architecture to solve the current technical problems.

[0454] To sum up, the embodiment of the present application proposes an image processing method. Through this progressive image processing method, the terminal device first performs lightweight processing on the reference image to generate first intermediate data and a first processed image, and then, when the terminal device meets the first condition, combines the first intermediate data with the original image data to generate a second processed image and stores it; in this way, through the lightweight processing and intermediate result reuse mechanism, not only can repeated calculations in the image processing process be avoided, and computing power consumption and energy consumption are significantly reduced; but by triggering the generation of the second processed image on demand, system resource scheduling can also be optimized, thereby improving image processing efficiency and further improving the real-time performance of the terminal device.

[0455] Furthermore, the embodiments of the present application, with the help of the powerful computing resources and advanced large-model algorithms of the cloud server, can efficiently and accurately complete complex image optimization tasks, greatly improve the efficiency and quality of image processing, and effectively make up for the shortcomings of terminal devices in computing power and algorithm complexity; at the same time, the terminal device can directly use the optimized target image to replace the full-size image for storage, saving local storage space.

[0456] Based on the above embodiment, in another embodiment of the present application, Figure 15 Schematic diagram of the composition structure of the image processing device proposed in the embodiment of the present application Figure 1 , the image processing device is applied to a terminal device, such as Figure 15 As shown, the image processing device 150 proposed in the embodiment of the present application may include:

[0457] An acquisition unit 1501 is configured to acquire original image data and a corresponding reference image in response to a shooting instruction;

[0458] The processing unit 1502 is configured to perform image processing on the reference image to generate first intermediate data and a first processed image;

[0459] The generation unit 1503 is configured to generate a second processed image based on the first intermediate data and the original image data when the current state of the terminal device meets the first condition, and update the first processed image to the second processed image; wherein the resolution of the second processed image is higher than the resolution of the first processed image.

[0460] In some embodiments, the current state of the terminal device satisfies the first condition, including at least one of the following: responding to a picture viewing instruction; the temperature of the terminal device is less than a preset temperature threshold; the load of the terminal device is less than a preset load threshold.

[0461] In some embodiments, the generation unit 1503 is further configured to generate a third processed image based on the second intermediate data and the original image data and update the second processed image to the third processed image when the current state of the terminal device satisfies the second condition; wherein the feature resolution of the second intermediate data is higher than the feature resolution of the first intermediate data, and the resolution of the third processed image is higher than the resolution of the second processed image.

[0462] In some embodiments, the current state of the terminal device satisfies the second condition, including at least one of the following: responding to a target shooting instruction; responding to a picture viewing instruction; the temperature of the terminal device is less than a preset temperature threshold; the load of the terminal device is less than a preset load threshold.

[0463] In some embodiments, see Figure 15 , the image processing apparatus 150 may further include a determining unit 1504 and a sending unit 1505, wherein:

[0464] The determining unit 1504 is configured to determine a time difference between a last access time of the third processed image and a current time;

[0465] The sending unit 1505 is configured to send the third processed image to the cloud server and delete the third processed image stored in the terminal device when the time difference is greater than a preset time.

[0466] In some embodiments, the image processing apparatus may further include a receiving unit 1506, wherein:

[0467] The sending unit 1504 is further configured to send the original image data and the associated intermediate data to the cloud server; wherein the intermediate data includes the first intermediate data and / or the second intermediate data;

[0468] The sending unit 1504 is further configured to send an image optimization request to the cloud server when the current state of the terminal device meets the third condition;

[0469] The receiving unit 1506 is configured to receive a target image sent by the cloud server based on the cloud server's response to the image optimization request, and store the target image; wherein the target image is generated based on the original image data and associated intermediate data, and the resolution of the target image is higher than the resolution of the third processed image.

[0470] In some embodiments, the current state of the terminal device satisfies a third condition, including at least one of the following: responding to a first user operation; the terminal device is in a charging state and the power of the terminal device is greater than a preset power threshold; the terminal device is connected to a wireless network.

[0471] In some embodiments, see Figure 15, the image processing apparatus 150 may further include a storage unit 1507, wherein:

[0472] The determining unit 1504 is further configured to determine the access frequency of the original image data and the associated intermediate data; wherein the intermediate data includes the first intermediate data and / or the second intermediate data;

[0473] The storage unit 1507 is configured to store the original image data and the associated intermediate data in the memory of the terminal device when the access frequency is greater than or equal to a first frequency; store the original image data and the associated intermediate data in the disk of the terminal device when the access frequency is less than the first frequency and greater than or equal to a second frequency; and send the original image data and the associated intermediate data to the cloud server for storage when the access frequency is less than the second frequency; wherein the first frequency is higher than the second frequency.

[0474] Based on the above embodiment, in another embodiment of the present application, Figure 16 Schematic diagram of the composition structure of the image processing device proposed in the embodiment of the present application Figure 2 , the image processing device is applied to a cloud server, such as Figure 16 As shown, the image processing device 160 proposed in this embodiment of the application may include:

[0475] Receiving unit 1601, configured to receive an image optimization request sent by a terminal device;

[0476] The processing unit 1602 is configured to perform image optimization processing on the image to be processed received by the cloud server based on the image optimization request to generate a target image;

[0477] The sending unit 1603 is configured to send the target image to the terminal device.

[0478] In some embodiments, the receiving unit 1601 is further configured to receive original image data and associated intermediate data sent by the terminal device;

[0479] The processing unit 1602 is further configured to perform image optimization processing on the original image data based on the intermediate data to generate a target image; wherein the intermediate data includes first intermediate data and / or second intermediate data, the first intermediate data is obtained by the terminal device performing image processing based on the reference image, and the second intermediate data is obtained by the terminal device performing image processing based on the first intermediate data and the original image data.

[0480] In some embodiments, see Figure 16 , the image processing device 160 may further include a storage unit 1604, wherein:

[0481] The receiving unit 1601 is further configured to receive a third processed image sent by the terminal device;

[0482] The storage unit 1604 is configured to store the third processed image in a preset storage area of ​​the cloud server.

[0483] In some embodiments, the processing unit 1602 is further configured to perform image optimization processing on the third processed image based on the intermediate data to generate a target image.

[0484] In some embodiments, the receiving unit 1601 is further configured to receive an image access instruction sent by a terminal device;

[0485] The sending unit 1603 is further configured to send the currently accessed first full-size image to the terminal device based on the image access instruction, and cache at least two adjacent images of the first full-size image at the edge node.

[0486] In the embodiments of this application, Figure 17 This is a schematic diagram of the structure of the electronic device proposed in the embodiment of this application. Figure 17 As shown, the electronic device 170 proposed in the embodiment of the present application may include a processor 1701, a memory 1702, a communication interface 1703, and a bus 1704 for connecting the processor 1701, the memory 1702 and the communication interface 1703.

[0487] In an embodiment of the present application, the processor 1701 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor function may also be other, and the embodiment of the present application is not specifically limited. The electronic device 170 may further include a memory 1702, which may be connected to the processor 1701, wherein the memory 1702 is used to store executable program code, the program code including computer operating instructions, and the memory 1702 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0488] In the embodiment of the present application, the bus 1704 is used to connect the communication interface 1703, the processor 1701 and the memory 1702, as well as the mutual communication between these devices.

[0489] In actual applications, the above-mentioned memory 1702 can be a volatile memory (volatile memory), such as random-access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as read-only memory (Read-Only Memory, ROM), flash memory (flash memory), hard disk drive (Hard Disk Drive, HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 1701.

[0490] Furthermore, in an embodiment of the present application, the processor 1701 is used to: obtain original image data and a corresponding reference image in response to a shooting instruction; perform image processing on the reference image to generate first intermediate data and a first processed image; when the current state of the terminal device meets the first condition, generate a second processed image based on the first intermediate data and the original image data, and store the second processed image; wherein the resolution of the second processed image is higher than the resolution of the first processed image.

[0491] In another embodiment of the present application, the processor 1701 is further used to: receive an image optimization request sent by a terminal device; based on the image optimization request, perform image optimization processing on the image to be processed received by the cloud server to generate a target image; and send the target image to the terminal device.

[0492] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0493] If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0494] An embodiment of the present application also provides a computer-readable storage medium for storing computer programs or instructions.

[0495] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program or instructions enable the processor or electronic device to execute the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0496] The embodiment of the present application also provides a computer program product.

[0497] In some embodiments, the computer program product may include a computer program or instructions.

[0498] In some embodiments, the computer program product can be applied to the computer device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the computer device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0499] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a processor or electronic device, the processor or electronic device executes the various methods of the embodiments of the present application. For the sake of brevity, they are not described here in detail.

[0500] It should be noted that the descriptions of the electronic device, storage medium, computer program product, and computer program embodiments described above are similar to the descriptions of the method embodiments described above and have similar beneficial effects as the method embodiments. For technical details not disclosed in the electronic device, storage medium, computer program product, and computer program embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0501] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0502] The term "and / or" in this article is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, object A and / or object B can mean: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0503] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0504] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0505] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0506] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0507] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0508] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0509] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments or device embodiments.

[0510] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: Applied to a terminal device, the method includes: In response to a shooting instruction, acquiring original image data and a corresponding reference image; performing image processing on the reference image to generate first intermediate data and a first processed image; When the current state of the terminal device satisfies the first condition, a second processed image is generated based on the first intermediate data and the original image data, and the first processed image is updated to the second processed image, wherein the resolution of the second processed image is higher than the resolution of the first processed image.

2. The method according to claim 1, characterized in that The current state of the terminal device satisfies a first condition, including at least one of the following: responding to an image viewing instruction; The temperature of the terminal device is lower than a preset temperature threshold; The load of the terminal device is less than a preset load threshold.

3. The method according to claim 1, characterized in that The method further comprises: If the current state of the terminal device satisfies a second condition, generating a third processed image based on the second intermediate data and the original image data, and updating the second processed image to the third processed image; The feature resolution of the second intermediate data is higher than that of the first intermediate data, and the resolution of the third processed image is higher than that of the second processed image.

4. The method according to claim 3, characterized in that The current state of the terminal device satisfies the second condition, including at least one of the following: responding to a target shooting instruction; responding to an image viewing instruction; The temperature of the terminal device is lower than a preset temperature threshold; The load of the terminal device is less than a preset load threshold.

5. The method according to claim 4, characterized in that After generating the third processed image, the method further includes: determining a time difference between a last access time of the third processed image and a current time; When the time difference is greater than a preset time, the third processed image is sent to a cloud server, and the third processed image stored in the terminal device is deleted.

6. The method according to claim 3, characterized in that The method further comprises: Sending the original image data and associated intermediate data to a cloud server; wherein the intermediate data includes the first intermediate data and / or the second intermediate data; When the current state of the terminal device satisfies the third condition, sending an image optimization request to the cloud server; receiving a target image sent by the cloud server based on a response of the cloud server to the image optimization request, and storing the target image; The target image is generated based on the original image data and associated intermediate data, and the resolution of the target image is higher than the resolution of the third processed image.

7. The method according to claim 6, characterized in that The current state of the terminal device satisfies the third condition, including at least one of the following: In response to a first user operation; The terminal device is in a charging state and the power level of the terminal device is greater than a preset power threshold; The terminal device is connected to a wireless network.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Determining access frequencies of the original image data and associated intermediate data; wherein the intermediate data includes first intermediate data and / or second intermediate data; When the access frequency is greater than or equal to a first frequency, storing the original image data and the associated intermediate data in a memory of the terminal device; When the access frequency is less than the first frequency and greater than or equal to the second frequency, storing the original image data and the associated intermediate data to a disk of the terminal device; When the access frequency is less than the second frequency, the original image data and the associated intermediate data are sent to a cloud server for storage; wherein the first frequency is higher than the second frequency.

9. An image processing method, characterized in that: Applied to a cloud server, the method includes: Receiving an image optimization request sent by a terminal device; Based on the image optimization request, performing image optimization processing on the image to be processed received by the cloud server to generate a target image; The target image is sent to the terminal device.

10. The method according to claim 9, characterized in that The performing image optimization processing on the image to be processed received by the cloud server to generate a target image includes: Receiving original image data and associated intermediate data sent by a terminal device; performing image optimization processing on the original image data based on the intermediate data to generate the target image; Among them, the intermediate data includes first intermediate data and / or second intermediate data, the first intermediate data is obtained by the terminal device performing image processing based on the reference image, and the second intermediate data is obtained by the terminal device performing image processing based on the first intermediate data and the original image data.

11. The method according to claim 9, characterized in that The method further comprises: receiving a third processed image sent by the terminal device; The third processed image is stored in a preset storage area of ​​the cloud server.

12. The method according to claim 11, characterized in that The performing image optimization processing on the image to be processed received by the cloud server to generate a target image includes: The third processed image is subjected to image optimization processing based on the intermediate data to generate the target image.

13. The method according to claim 11, characterized in that The method further comprises: receiving an image access instruction sent by the terminal device; Based on the image access instruction, the first full-size image currently being accessed is sent to the terminal device, and at least two adjacent images of the first full-size image are cached at the edge node.

14. An image processing device, characterized in that: Applied to a terminal device, the image processing device includes: an acquisition unit, configured to acquire raw image data and a corresponding reference image in response to a shooting instruction; a processing unit configured to perform image processing on the reference image to generate first intermediate data and a first processed image; A generation unit is configured to generate a second processed image based on the first intermediate data and the original image data when the current state of the terminal device satisfies a first condition, and update the first processed image to the second processed image; wherein the resolution of the second processed image is higher than the resolution of the first processed image.

15. An image processing device, characterized in that: Applied to a cloud server, the image processing device includes: a receiving unit configured to receive an image optimization request sent by a terminal device; a processing unit configured to perform image optimization processing on the image to be processed received by the cloud server based on the image optimization request to generate a target image; A sending unit is configured to send the target image to the terminal device.

16. An electronic device, characterized in that: The electronic device includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 8 or the method according to any one of claims 9 to 13 is implemented.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 or the method according to any one of claims 9 to 13 is implemented.

Images