Image processing methods, apparatuses, electronic devices, chips, and storage media

By introducing visual selection and neural network co-optimization into the image processing chip, the problem of automated design of the image processing chip in complex environments is solved, achieving high-quality image output and personalized user experience.

CN123123709APending Publication Date: 2026-11-17BEIJING X RING TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510630366.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies struggle to automate the design of image processing chip workflows in complex and ever-changing environments, failing to adapt to diverse needs across different scenarios, and resulting in insufficient image processing quality and user experience.

Method used

By visually prompting users to select parameter combinations, and combining the collaborative work of neural networks and image processing modules, the image processing chip can be autonomously optimized and personalized to improve image quality and user experience.

Benefits of technology

It improves image processing quality and overall visual appeal, enhances user engagement and experience, and caters to the diverse needs and personalized preferences of different user groups.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123709A_ABST
    Figure CN123123709A_ABST
Patent Text Reader

Abstract

This application proposes an image processing method, apparatus, electronic device, chip, and storage medium, relating to the field of image processing. The method includes: displaying a prompt message in response to a shooting operation; wherein the prompt message prompts the selection of a parameter combination from at least one parameter combination, the parameter combination including control parameters of each image processing module in the target processing link of the image processing chip; determining the target parameter combination selected by the triggering object of the shooting operation in response to a selection operation triggered by the prompt message; and processing a first captured image based on the target parameter combination through each image processing module in the target processing link to obtain and display a first target image with optimized image quality. Thus, providing diverse parameter combinations for users to select not only adapts to the diverse needs and personalized preferences of different user groups but also improves image processing quality and overall visual appeal, enhancing the user's shooting experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, chip, and storage medium. Background Technology

[0002] Whether it's drones capturing fleeting moments, autonomous vehicles requiring high-precision visual recognition, or smartphones pursuing the ultimate user experience, all demand image processing chips with exceptional performance. These chips must maintain stable, high-performance image processing capabilities under varying lighting conditions, complex backgrounds, and different shooting angles. This not only enhances the user experience but also provides a solid foundation for further applications such as machine learning, augmented reality, and real-time monitoring. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, this application proposes an image processing method, apparatus, electronic device, chip, and storage medium to provide users with a variety of parameter combinations to choose from. This not only adapts to the diverse needs and personalized preferences of different user groups, but also improves image processing quality and overall visual appeal, thereby enhancing the user's shooting experience.

[0005] One embodiment of this application proposes an image processing method, including:

[0006] In response to the shooting operation, a prompt message is displayed; wherein the prompt message is used to prompt the selection of a parameter combination from at least one parameter combination, the parameter combination including the control parameters of each image processing module in the target processing link of the image processing chip;

[0007] In response to the selection operation triggered by the prompt information, determine the target parameter combination selected by the triggering object of the shooting operation;

[0008] The image processing modules in the target processing chain process the first captured image based on the target parameter combination to obtain and display the first target image with optimized image quality.

[0009] Another embodiment of this application provides an image processing apparatus, including:

[0010] The display module is used to display prompt information in response to the shooting operation; wherein the prompt information is used to prompt the selection of a parameter combination from at least one parameter combination, the parameter combination including the control parameters of each image processing module in the target processing link of the image processing chip;

[0011] The determination module is used to determine the target parameter combination selected by the triggering object of the shooting operation in response to the selection operation triggered by the prompt information;

[0012] The processing module is used to process the first captured image based on the target parameter combination through each image processing module in the target processing link, so as to obtain and display the first target image after image quality optimization.

[0013] Another embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the image processing method as described in the foregoing aspect.

[0014] Another aspect of this application provides a chip including an interface circuit and a processing circuit coupled to each other, the interface circuit being used to input or output signals, and the processing circuit being configured to perform the image processing method as described in the foregoing aspect.

[0015] In another aspect of this application, a non-transitory computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the image processing method as described in the foregoing aspect.

[0016] Another aspect of this application provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the image processing method as described in the foregoing aspect.

[0017] The image processing method, apparatus, electronic device, chip, and storage medium proposed in this application, through a visual demonstration, prompt or guide users to select their desired parameter combination from at least one such combination. This allows for image processing based on the selected parameter combination, enhancing user engagement and experience. Furthermore, it provides a diverse range of parameter combinations to suit the varied needs and personalized preferences of different user groups. Based on the user-selected target parameter combination, the various image processing modules in the target processing chain of the image processing chip can work collaboratively to achieve higher quality image output. This not only improves image processing quality and overall visual appeal but also enhances the user's shooting experience.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0020] Figure 1 A schematic flowchart illustrating the first image processing method provided in the embodiments of this application;

[0021] Figure 2 A schematic flowchart illustrating the second image processing method provided in this application embodiment;

[0022] Figure 3 This is a schematic diagram illustrating the implementation of the third processing module provided in an embodiment of this application;

[0023] Figure 4 A schematic flowchart illustrating the third image processing method provided in this application embodiment;

[0024] Figure 5 This is a schematic flowchart illustrating the fourth image processing method provided in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram illustrating the generation method of the output image in each iteration update process provided in the embodiments of this application;

[0026] Figure 7 A schematic flowchart illustrating the fifth image processing method provided in this application embodiment;

[0027] Figure 8 This is a schematic diagram illustrating the implementation principle of any embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the structure of a chip proposed in an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0032] With the widespread adoption of smart devices such as smartphones, artificial intelligence (AI) glasses, drones, smart security systems, and autonomous vehicles, user demand for high-quality image and video processing is rapidly increasing. These devices rely on image processing chips to achieve fast and accurate image signal processing in various complex environments to ensure the clarity and realism of the output images.

[0033] The design of image processing chips (such as image signal processors (ISPs) and display processing units (DPUs) faces at least the following challenges:

[0034] 1. Complex parameter tuning: Taking image processing chips, including ISPs, as an example, the image processing algorithms executed by ISPs, such as denoising, demosaicing, and automatic white balance, depend on the setting of multiple control parameters (such as denoising granularity, brightness compensation, color compensation, and gamma value in gamma correction). Manual tuning is time-consuming, laborious, and difficult to adapt to complex and ever-changing environments.

[0035] II. Hardware Resource Constraints: In some portable devices, high-performance image processing functions need to be achieved under limited hardware resources and power consumption conditions.

[0036] Therefore, the key to improving the image processing performance of image processing chips and enhancing display quality lies in how to efficiently design the processing pipeline of image processing chips and find the most suitable control parameters for the current scenario (including the control parameters of each image processing module in the pipeline of image processing chips).

[0037] In related technologies, the following methods are mainly used to determine the control parameters of each image processing module in the image processing chip pipeline:

[0038] Option 1: Real-time Automatic Parameter Optimization Method Based on Neural Networks: This method constructs a parameter network and an image processing chip surrogate model. The parameter network directly predicts the control parameters of the image processing chip from the original image, while the surrogate model learns the mapping relationship between the control parameters and the image processing modules within the chip. Through steps such as simulating non-differentiable image processing modules, mapping the original image to the control parameters of the image processing chip, and online tuning, real-time optimization of the chip's control parameters is achieved, improving image quality and processing speed.

[0039] Option 2: Scene-Adaptive Automatic Optimization Method: A database is created within the camera, containing various scenes and a set of control parameters configured for each scene. After the camera captures video footage, features of the current scene are extracted and compared with those features in the database to determine the scene type. The corresponding set of control parameters is then invoked, automatically adjusting the control parameters in the image processing chip based on different scenes, thus enhancing the camera's adaptability.

[0040] Option 3: Automated Testing and Optimization Method Based on Image Quality Assessment: This method acquires real-time values ​​of parameters affecting the image processing algorithm, such as YUV values, shutter speed, and gain. When these parameters change, the image processing algorithm interface is invoked to capture a frame of data and save it as a JPEG file, while simultaneously recording the current YUV values. By analyzing this data and the corresponding image quality, the control parameters in the image processing chip can be optimized. This automates image quality testing and provides data support for algorithm optimization.

[0041] YUV is a color space (or color space). Y: represents luminance (Luma), which is the brightness information of the image and does not contain color information. U and V: represent blue difference (Cb) and red difference (Cr) respectively, which are chromaticity information and describe the specific hue of the color.

[0042] JPEG is an image compression technology standard (Joint Photographic Experts Group).

[0043] In summary, the above solution has at least the following problems:

[0044] 1. It can only adjust parameters of a preset processing pipeline and cannot automatically design the processing flow of an image processing chip; or it can only optimize for a preset scenario and cannot automatically design the processing flow of an image processing chip; or it can only search for a specific set of control parameters for a single scenario and cannot automatically design the processing flow of an image processing chip.

[0045] 2. Post-processing algorithms can only be used for optimization in specific scenarios.

[0046] Therefore, in view of at least one of the problems existing in the above-mentioned related technologies, this application proposes an image processing method, apparatus, electronic device, chip and storage medium.

[0047] The image processing method, apparatus, electronic device, chip, and storage medium of this application are described below with reference to the accompanying drawings. Before specifically describing the embodiments of this application, commonly used technical terms are first introduced for ease of understanding:

[0048] The Image Processing Unit (ISP) is responsible for processing the raw images acquired from the image sensor, executing a series of image processing algorithms to improve image quality and prepare it for subsequent processing or display. These image processing algorithms include, but are not limited to: Noise Reduction: reducing random noise in the image and improving image sharpness; Demosaicing: converting the color information captured by the Bayer filter array into a complete RGB image; Automatic White Balance (AWB): adjusting the image's colors to accurately reproduce true colors under various lighting conditions; Exposure Control: optimizing image brightness to ensure details are neither overexposed nor underexposed; Color Correction: correcting color deviations in the image sensor output using techniques such as color correction matrices, making image colors more realistic and natural; and Edge Enhancement: enhancing edge details in the image and improving visual sharpness.

[0049] RGB is a color space based on three primary colors: red (R), green (G), and blue (B). Different degrees of superposition are used to produce a rich and wide range of colors, commonly known as the three-primary-color model.

[0050] The DPU is responsible for processing the image data that will be displayed to the user. The image processing algorithms executed by the DPU include, but are not limited to, image scaling, color space conversion, frame rate adjustment, and high dynamic range (HDR) processing, to ensure that the image can be presented on the display screen with the best quality and efficiency.

[0051] A differentiable module, also known as a differentiable module, is a module whose input and output relationship can be described by a differentiable function. In other words, the output of a differentiable module is continuous and differentiable with respect to its input, and can be optimized using gradient descent: because a differentiable module is differentiable, the gradient of the loss function with respect to the module parameters can be calculated using the backpropagation algorithm, and these gradients can be used to update the parameters.

[0052] A non-differentiable module is a module whose input-output relationship cannot be described by a differentiable function. That is, a non-differentiable module may have discontinuities, discrete operations, or non-smooth behavior, making gradients impossible to calculate directly.

[0053] LTM, short for Local Tone Mapping, is used to adjust the local contrast and brightness of an image to optimize the presentation of HDR content on low dynamic range or standard dynamic range (SDR) displays.

[0054] GTM, short for Global Tone Mapping, is used to uniformly adjust the brightness and contrast of an image, typically applied in HDR to SDR conversion. Unlike LTM, GTM applies the same transformation rules to the entire image.

[0055] 3DLUT, short for 3D Look-Up Table, is a technique used for color correction and color space conversion. It stores the mapping between input and output color values ​​using a three-dimensional array and is commonly used in video post-production and display device calibration.

[0056] Figure 1 This is a schematic flowchart of the first image processing method provided in the embodiments of this application.

[0057] It should be noted that the image processing method of this application embodiment can be applied to an image processing device. In some possible embodiments, the image processing device can be configured in an electronic device or a chip so that the electronic device or chip can perform image processing functions. In addition, in some possible embodiments, the image processing device can also be software in an electronic device.

[0058] In any embodiment of this application, the chip can be integrated into an electronic device. The chip includes a Central Processing Unit (CPU), an Image Signal Processor (ISP), an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), a System-on-A-Chip (SOC), a Reduced Instruction Set Computer (RISC), etc., which will not be listed here.

[0059] The electronic devices mentioned include, but are not limited to, terminals, personal computers (PCs), etc. A terminal is a user-side entity used to receive or transmit signals, such as a mobile phone. A terminal can also be called a terminal device, user equipment (UE), mobile station (MS), mobile terminal device (MT), etc. Terminals can be communication-enabled vehicles, smart cars, mobile phones, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of this application do not limit the specific technology or device form used in the terminal.

[0060] like Figure 1 As shown, the image processing method may include the following steps S101 to S103:

[0061] Step S101: In response to the shooting operation, a prompt message is displayed; wherein the prompt message is used to prompt the selection of a parameter combination from at least one parameter combination, the parameter combination including the control parameters of each image processing module in the target processing link of the image processing chip.

[0062] The shooting operations include, but are not limited to: taking photos, previewing, and recording videos.

[0063] Image processing chips include, but are not limited to, ISP and DPU.

[0064] The target processing pipeline of the image processing chip includes multiple image processing modules connected in series. These modules execute different image processing algorithms and include differentiable and non-differentiable image processing modules.

[0065] Each parameter combination includes control parameters for each image processing module in the target processing chain. Taking an image processing chip including an ISP as an example, the parameter combinations may include: noise reduction granularity, brightness compensation, color compensation, gamma value, etc.

[0066] In this embodiment of the application, in response to a user-triggered shooting operation, a prompt message can be displayed in a visual manner, wherein the prompt message is used to prompt or guide the user to select a parameter combination from at least one parameter combination.

[0067] For example, the prompt information can be used to indicate the names of different parameter combinations: such as original parameters, search update parameters, and photo editing (photo retouching) preference parameters. Users can select the parameter combination that meets their personalized needs from different parameter combinations based on the names of the different parameter combinations.

[0068] Step S102: In response to the selection operation triggered by the prompt information, determine the target parameter combination selected by the triggering object of the shooting operation.

[0069] The trigger for the shooting operation includes the user currently using the electronic device.

[0070] In this embodiment of the application, the target parameter combination selected by the user (i.e. the triggering object of the shooting operation) can be determined in response to the user's selection operation triggered by the prompt information.

[0071] As an example, multiple controllable controls can be displayed visually, where the name of the controllable control can be the name of the parameter combination. In this application, the user can select the target parameter combination by selecting a controllable control.

[0072] Using the example above, three controllable controls can be displayed visually, such as control 1, control 2, and control 3. Control 1 is named the original parameter, control 2 is named the search update parameter, and control 3 is named the P-image preference parameter. If the user selects control 3, such as by clicking control 3, the target parameter combination can be determined to be the parameter combination corresponding to the P-image preference parameter.

[0073] Step S103: The first captured image is processed by each image processing module in the target processing link based on the target parameter combination to obtain and display the first target image with optimized image quality.

[0074] The first captured image includes the original captured image acquired or captured by the image sensor, or a new image obtained by processing the original captured image.

[0075] In any embodiment of this application, when the image processing chip includes an ISP, the first captured image may include the original captured image captured by the image sensor.

[0076] In any embodiment of this application, when the image processing chip includes a DPU, the first captured image may include an image processed from the original captured image. For example, the original captured image may be input to an ISP for image processing, and the output image of the ISP may be used as the first captured image; alternatively, the output image of the ISP may be further processed to obtain the first captured image.

[0077] Therefore, by using different image processing chips to process captured images in a targeted manner, both image processing quality and practical application requirements can be taken into account.

[0078] In this embodiment of the application, multiple image processing modules connected in series in the target processing link can process the first captured image based on the combination of target parameters to obtain a first target image with optimized image quality, and then display the first target image.

[0079] As an example, the following formula can be used to obtain the first target image:

[0080] IPC(I) = M N (M N-1 (...M1(I,θ1)...,θ N-1 ),θ N (1)

[0081] In this context, IPC refers to the Image Processing Chip, I refers to the first captured image, and M refers to... i This refers to the i-th image processing module in the target processing chain, θ i represents the control parameters for the i-th image processing module, where i is a positive integer less than or equal to N, and N is the number of image processing modules in the target processing chain. That is, the first captured image serves as the input to the first image processing module in the target processing chain, and the output of the i-th image processing module in the target processing chain serves as the input to the (i+1)-th image processing module.

[0082] The image processing method of this application embodiment, through a visual display, prompts or guides the user to select the desired parameter combination from at least one parameter combination. The image is then processed based on the selected parameter combination. This not only enhances the user's sense of participation and experience but also provides a variety of parameter combinations for users to choose from, adapting to the diverse needs and personalized preferences of different user groups. Based on the target parameter combination selected by the user, the various image processing modules in the target processing chain of the image processing chip can work collaboratively to achieve higher quality image output. This not only improves image processing quality and overall visual appeal but also enhances the user's shooting experience.

[0083] This application provides another image processing method. Figure 2 This is a schematic flowchart of the second image processing method provided in the embodiments of this application.

[0084] It should be noted that the image processing method can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in related technologies. The embodiments of this application do not limit this.

[0085] like Figure 2 As shown, the image processing method may include the following steps S201 to S203:

[0086] Step S201: In response to the shooting operation, a prompt message is displayed; wherein the prompt message is used to prompt the selection of a parameter combination from at least one parameter combination, the parameter combination including at least one of the following: original parameter combination, first parameter combination and second parameter combination.

[0087] The parameter combination includes the control parameters of each image processing module in the target processing link of the image processing chip.

[0088] It should be noted that explanations of the shooting operation and prompts can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0089] The original parameter combination includes the original control parameters of multiple image processing modules connected in series in the target processing link. The multiple image processing modules connected in series include a non-differentiable first processing module and a differentiable second processing module. The original control parameters of each first processing module are determined based on the original control parameters of a differentiable third processing module associated with the first processing module. The input data and output data of the first processing module and its associated third processing module are the same, that is, the third processing module can be a substitute for the first processing module.

[0090] The input data includes the input image and the input control parameters.

[0091] As an example, the j-th non-differentiable first processing module in the target processing chain of an image processing chip is labeled as... Where j is a positive integer, and The associated differentiable third processing module is in, It can be implemented using neural networks, and and The input and output are the same, assuming The control parameter is θ j Then we have:

[0092]

[0093] Among them, I j for The input image.

[0094] For example, The implementation method can be as follows Figure 3 As shown, It mainly consists of two parts: the input control parameters are converted into parameter vectors through a fully connected neural network, and the input image is processed by a neural network (such as UNet (a type of convolutional neural network)) and then fused with the parameter vectors at each encoder / decoder stage to obtain the output vector.

[0095] in, Training methods include supervised training, where the input image, control parameters, and output image are all related to... Alignment, loss function, for example, is L1 (1-norm or L1-norm) loss.

[0096] The first parameter combination is obtained by searching and updating the original parameter combination. For example, a heuristic algorithm (such as ant colony algorithm, genetic algorithm, etc.) can be used to search and update the original parameter combination to obtain the first parameter combination.

[0097] The second parameter combination is obtained by adjusting or fine-tuning the first parameter combination based on the image editing preferences (such as photo editing preferences) of the triggering object of the shooting operation.

[0098] In this embodiment of the application, in response to a user-triggered shooting operation, a prompt message can be displayed in a visual manner to prompt or guide the user to select a parameter combination from at least one parameter combination (such as the original parameter combination, the first parameter combination, and the second parameter combination).

[0099] Step S202: In response to the selection operation triggered by the prompt information, determine the target parameter combination selected by the triggering object of the shooting operation.

[0100] Step S203: The first captured image is processed by each image processing module in the target processing link based on the target parameter combination to obtain and display the first target image with optimized image quality.

[0101] It should be noted that the explanations of steps S202 to S203 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0102] The image processing method of this application embodiment allows the user to independently select a parameter combination from at least one parameter combination (such as an original parameter combination, a first parameter combination, and a second parameter combination), which can balance image processing quality and the user's personalized customization needs. The original parameter combination ensures basic image processing quality. Parameter searching and fine-tuning of the original parameter combination yields a first parameter combination, which optimizes the control parameters of each image processing module in the target processing chain of the image processing chip, thereby further improving the image quality output by the image processing chip. Based on this, fine-tuning the first parameter combination according to the user's image editing preferences (such as photo editing preferences) can enhance the user's personalized shooting experience while improving imaging effects.

[0103] This application provides another image processing method. Figure 4 This is a schematic flowchart of the third image processing method provided in the embodiments of this application.

[0104] It should be noted that the image processing method can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in related technologies. The embodiments of this application do not limit this.

[0105] like Figure 4 As shown, based on any embodiment of this application, the second parameter combination can be obtained by the following steps S401 to S403:

[0106] Step S401: Determine a second target image with editing records from a plurality of candidate images stored locally; wherein the second target image represents image editing preferences.

[0107] The number of second target images can be at least one.

[0108] For example, whenever a user edits a locally stored image, the editing result (such as a photo editing effect) can be automatically recorded. Therefore, in this application, images with editing results or editing records can be queried and used as second target images. For instance, a second target image with editing records can be obtained from a local photo album.

[0109] Step S402: Query the second captured image associated with the second target image; wherein the second target image is generated based on the second captured image.

[0110] The second captured image associated with the second target image includes: a raw image (RAW image) captured by an image sensor. The second target image is an image obtained by processing the second captured image; for example, the second target image may be a final RGB format image.

[0111] In this embodiment of the application, a second captured image associated with each second target image can be queried.

[0112] Step S403: Based on the second captured image and the second target image, at least some parameters in the first parameter combination are fine-tuned to obtain the second parameter combination.

[0113] In this embodiment of the application, at least some parameters in the first parameter combination can be fine-tuned using the associated second captured image and the second target image to obtain the second parameter combination.

[0114] For example, the second captured image can be used as the input image of the image processing chip, and the second target image can be used as the ground truth (i.e., the labeled image). In this application, the second captured image can be processed by multiple image processing modules connected in series in the target processing link based on the first parameter combination to obtain the output image of the image processing chip. Based on the difference between the output image and the second target image, at least some parameters in the first parameter combination can be fine-tuned to obtain the second parameter combination.

[0115] In any embodiment of this application, at least some of the parameters that need to be adjusted may include: control parameters of the differentiable second processing module in the image processing chip. That is, in this application, the parameters to be adjusted can be determined from the first parameter combination, wherein the parameters to be adjusted include the control parameters of the differentiable second processing module in the image processing chip, so that the parameters to be adjusted can be updated according to the second captured image and the second target image to obtain the second parameter combination.

[0116] For example, the second captured image can be used as the input image of the image processing chip, and the second target image can be used as the ground truth (i.e., the labeled image). In this application, the second captured image can be processed by multiple image processing modules connected in series in the target processing link based on the first parameter combination to obtain the output image of the image processing chip. Based on the difference between the output image and the second target image, the parameters to be adjusted in the first parameter combination can be fine-tuned to obtain the second parameter combination.

[0117] It should be noted that the control parameters in the first parameter combination are optimized basic configurations applicable to most scenarios. Directly overriding these control parameters may lead to a decrease in the performance of the image processing chip or poor adaptability. Considering this issue, in any embodiment of this application, a small learning rate can be used for parameter fine-tuning, that is, a small adjustment can be made based on the first parameter combination. This can retain the core advantages of the original parameter tuning information, while making personalized adjustments for different users' image editing preferences to improve personalized adaptability.

[0118] That is, in order to avoid overwriting the optimized original parameter tuning information (i.e., each control parameter in the first parameter combination), in this application, a relatively small learning rate (denoted as the target learning rate in this application) can be used to update the parameters to be adjusted based on the second captured image and the second target image to obtain the second parameter combination, wherein the target learning rate is lower than the set learning rate threshold.

[0119] For example, a target learning rate can be used to update the parameters to be adjusted once based on the difference between the output image of the image processing chip and the second target image, so as to avoid overwriting the original parameter tuning information and ensure basic performance.

[0120] The image processing method of this application embodiment uses a second target image representing the user's personalized image editing preferences and a second captured image associated with the second target image to fine-tune at least some parameters in the first parameter combination, which can improve the user's personalized shooting experience while improving the imaging effect.

[0121] This application provides another image processing method. Figure 5 This is a schematic flowchart illustrating the fourth image processing method provided in the embodiments of this application.

[0122] It should be noted that the image processing method can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in related technologies. The embodiments of this application do not limit this.

[0123] like Figure 5As shown, based on any embodiment of this application, the target processing link of the image processing chip can be determined or designed using the following steps S501 to S504:

[0124] Step S501: Generate the initial processing link of the image processing chip based on the differentiable second processing module and the differentiable third processing module; wherein, the initial processing link includes N differentiable image processing modules connected in series.

[0125] The second processing module is a differentiable image processing module in the image processing chip, and the third processing module is a differentiable image processing module associated with the non-differentiable first processing module in the image processing chip.

[0126] It should be noted that the explanations of the first processing module, the second processing module, and the third processing module in the foregoing embodiments also apply to this embodiment, and will not be repeated here.

[0127] As an example, taking an image processing chip including an ISP as an example, we can identify all the image processing modules contained in the ISP, denoted as set S. isp The final processing link (referred to as the target processing link in this application) of the ISP is designed as a serial pipeline, wherein each image processing module M in the pipeline... i ∈S isp That is, M i Only from S isp Choose from.

[0128] First, we can work on S isp The image processing module in the code is divided into two subsets based on whether it is differentiable or not: and in, Including S isp All differentiable image processing modules, i.e. The control parameters of the image processing module can be fine-tuned using the gradient descent method. Including S isp All non-differentiable image processing modules, for example, The image processing module in the code contains operators such as statistics, maximum / minimum / sorting, which cannot perform gradient backpropagation. The control parameters of the image processing module cannot be adjusted using the gradient descent method.

[0129] After that, All image processing modules are approximated by a neural network to obtain a replacement set. Among them, for Each image processing module (i.e., the first processing module) in the process. exist There exists one and only one corresponding third processing module. Both modules have the same input and output. Can be derived from Figure 3 The neural network shown is implemented as follows, assuming The control parameter is θ j Then it should be:

[0130]

[0131] Therefore, differentiable replacement of non-differentiable modules can be achieved.

[0132] Then, according to and The initial processing link of the image processing chip is generated, wherein the initial processing link is a serial pipeline containing N differentiable image processing modules connected in series, and the input of each image processing module is the output of the previous image processing module.

[0133] Step S502: Assign learnable weight vectors to the N differentiable image processing modules connected in series.

[0134] For example, a weight vector can be initialized to generate N differentiable image processing modules in series.

[0135] Step S503: Based on the weight vector, perform N rounds of iterative update process on the initial processing link to obtain candidate processing links.

[0136] In this embodiment, N rounds of iterative update process can be performed on the initial processing link based on the weight vectors of N differentiable image processing modules connected in series, so as to obtain candidate processing links.

[0137] In any embodiment of this application, the nth iteration update process in the Nth iteration update process is used to determine the image processing module with the processing order n in the candidate processing link, that is, to determine the nth image processing module in the candidate processing link, where n is a positive integer less than or equal to N. In other words, the Nth iteration update process is used to redetermine the rearrangement order of each differentiable image processing module.

[0138] In this application, for the nth iteration update process, firstly, an input image can be acquired and input in parallel to N image processing modules in the initial processing link to obtain the output images of N image processing modules. Then, the output images of the N image processing modules can be fused to obtain a fused image. Specifically, when n=1, the weight vector initially allocated to the N image processing modules can be used to fuse the output images of the N image processing modules to obtain a fused image. When n is greater than 1, the weight vector of the N image processing modules output in the (n-1)th iteration update process can be used to fuse the output images of the N image processing modules to obtain a fused image.

[0139] Subsequently, based on the fused image and the expected image associated with the input image, the control parameters and weight vectors corresponding to the N image processing modules can be iteratively updated to obtain the control parameters and weight vectors corresponding to the N image processing modules output in the nth iteration update process. Thus, the image processing module with the largest weight vector output in the nth iteration update process can be selected as the image processing module with the processing order n in the candidate processing chain.

[0140] In the case of n=1, the weight vector of the N image processing modules is the initially assigned weight vector. In the case of n>1, the weight vector of the N image processing modules is the weight vector of the N image processing modules output in the (n-1)th iteration update process.

[0141] The expected image is the image obtained after processing the input image, and this expected image can be used as the ground truth (i.e., the labeled image).

[0142] As an example, one could use, such as Figure 6 The illustrated two-layer optimization method or other Neural Architecture Search (NAS) methods are used for network architecture updates: The image processing chip pipeline consists of N iterations. For the nth iteration, a learnable weight vector W is introduced. n ∈R 1*(|S|+1) Where |S| represents the number of image processing modules in the image processing chip, and +1 indicates that no module is set in the nth iteration update process. Each iteration update process includes all |S| parallel image processing modules and one skip connection. The output images of all image processing modules are weighted to obtain the output image (i.e., the fused image) of that iteration update process. The weighting weight of each image processing module can be softmax(W). n ).

[0143] For example, the fused image in the nth iteration update process can be adopted as follows: Figure 6 The process shown calculates that, during the nth iteration update, the fused image (i.e. Figure 6 The control parameters and weight vectors of |S| image processing modules are iteratively updated based on the difference between the output image and the expected image associated with the input image, until the network converges. Then, based on the weight vector W of the |S| image processing modules, the image processing module with the largest weight vector is determined and used as the image processing module with processing order n in the pipeline.

[0144] In summary, it is possible to design and update the pipeline of image processing chips, such as by using the NAS method to search for the optimal pipeline design, i.e., adaptively searching for the best pipeline design across all scenarios, which can improve image quality.

[0145] Step S504: Generate the target processing link based on the candidate processing links.

[0146] In this embodiment, the target processing link of the image processing chip can be generated based on the candidate processing links.

[0147] As an example, the differentiable third processing module in the candidate processing link can be replaced with the associated non-differentiable first processing module to obtain the target processing link.

[0148] For example, firstly, the N differentiable image processing modules in the candidate processing chain can be trained using the first training sample to obtain the original control parameters of the N differentiable image processing modules. The first training sample includes sample images as input data and labeled images (i.e., ground truth) associated with the sample images.

[0149] In other words, N differentiable image processing modules connected in series in the candidate processing link of the image processing chip can be used to process the sample images in the first training sample to obtain the output image of the image processing chip. Based on the difference between the output image and the labeled image in the first training sample, the control parameters of the N differentiable image processing modules in the candidate processing link can be adjusted to obtain the original control parameters of the N differentiable image processing modules.

[0150] Subsequently, each third processing module in the candidate processing link can be replaced with the associated first processing module to obtain the target processing link of the image processing chip. Based on the parameter constraints of the associated first processing module, the original control parameters of the third processing module are transformed (such as quantization, Contrastive Language–Image Pre-training (CLIP) operations, etc.) to obtain the original control parameters of the associated first processing module in the target processing link.

[0151] The original parameter combination includes the original control parameters of N image processing modules in the target processing link.

[0152] Therefore, not only can the pipeline of the image processing chip be optimized, but also the control parameters of N image processing modules in the pipeline can be optimized, which can further improve the image quality output by the image processing chip and improve the user's shooting experience.

[0153] The image processing method of this application embodiment performs differentiable replacement on non-differentiable modules during the pipeline design stage, and adaptively searches for the best pipeline design for the entire scene using the NAS method to obtain candidate processing links for the image processing chip. Then, it uses differentiable modules (such as neural networks) to search for parameters of the candidate processing links to obtain the optimal original control parameters for the entire scene, and derives them to non-differentiable modules, which can further improve the image quality.

[0154] This application provides another image processing method. Figure 7 This is a schematic flowchart illustrating the fifth image processing method provided in the embodiments of this application.

[0155] It should be noted that the image processing method can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in related technologies. The embodiments of this application do not limit this.

[0156] like Figure 7 As shown, based on any embodiment of this application, the first parameter combination can be obtained by the following steps S701 to S702:

[0157] Step S701: Perform at least one round of search and update process on the original parameter combination.

[0158] Each round of the search and update process is used to search for and update the parameters of the original parameter combination.

[0159] In any embodiment of this application, for the first search update process in at least one round of search update process, firstly, the original control parameters of the differentiable second processing module in the target processing link can be fixed, and a heuristic algorithm (such as ant colony algorithm, genetic algorithm, etc.) can be used to perform a heuristic search on the original control parameters of the non-differentiable first processing module to obtain the updated control parameters of the first processing module output in the first round of search update process.

[0160] Subsequently, a third processing module associated with the first processing module can be used to replace the first processing module in the target processing link to obtain the update processing link of the first round of search update process. Then, the control parameters of the associated third processing module in the update processing link of the first round of search update process can be updated according to the update control parameters of the first processing module output by the first round of search update process, and the control parameters of the associated third processing module in the update processing link of the first round of search update process can be kept unchanged.

[0161] Finally, the control parameters of the differentiable second processing module in the update processing link of the first round of search update process can be updated using the second training sample to obtain the update control parameters of the second processing module output by the first round of search update process.

[0162] The second training sample includes a sample image that serves as input data, and a labeled image (i.e., ground truth) associated with that sample image.

[0163] In other words, multiple image processing modules connected in series in the update processing link of the first round of search and update process can be used to process the sample images in the second training samples to obtain the output image of the image processing chip. Based on the difference between the output image and the labeled image in the second training samples, the control parameters of the differentiable second processing module in the update processing link of the first round of search and update process can be adjusted to obtain the update control parameters of the second processing module output by the first round of search and update process.

[0164] In any embodiment of this application, for the k-th round (k is a positive integer greater than 1) of the search and update process in at least one round of search and update, firstly, it can be determined whether a set stop update condition is met. If the stop update condition is met, the search and update process ends. The stop update condition includes, but is not limited to: network convergence, and k reaching a preset maximum number of iterations.

[0165] If the stop-update condition is not met, the associated third processing module in the update processing chain of the (k-1)th round of search and update process is updated to the first processing module, and the update control parameters of the second processing module output by the (k-1)th round of search and update process are kept unchanged. A heuristic algorithm (such as ant colony algorithm, genetic algorithm, etc.) is used to perform a heuristic search on the update control parameters of the first processing module to obtain the update control parameters of the first processing module output by the kth round of search and update process.

[0166] Then, a third processing module associated with the first processing module can be used to replace the first processing module to obtain the update processing link of the k-th search update process. Then, the control parameters of the third processing module associated with the update processing link of the k-th search update process can be updated according to the update control parameters of the first processing module output by the k-th search update process, and the control parameters of the associated third processing module in the update processing link of the k-th search update process can be kept unchanged.

[0167] Finally, the control parameters of the differentiable second processing module in the update processing link of the k-th round search update process can be updated using the third training sample to obtain the update control parameters of the second processing module output by the k-th round search update process.

[0168] The third training sample includes a sample image that serves as input data, and a labeled image (i.e., ground truth) associated with that sample image.

[0169] In other words, multiple image processing modules connected in series in the update processing link of the k-th round search update process can be used to process the sample images in the third training sample to obtain the output image of the image processing chip. Based on the difference between the output image and the labeled image in the third training sample, the control parameters of the differentiable second processing module in the update processing link of the k-th round search update process can be adjusted to obtain the update control parameters of the second processing module output by the k-th round search update process.

[0170] In summary, a heuristic algorithm can be used to perform multiple rounds of parameter search and update on the original parameter combination to determine the optimal first parameter combination for the entire scene. For example, the optimal control parameters corresponding to different scenes can be searched to obtain the first parameter combination, which can improve the flexibility of the method and improve the imaging effect.

[0171] Step S702: Generate a first parameter combination based on the update control parameters corresponding to the first processing module and the second processing module output in the last round of search and update process.

[0172] That is, the first parameter combination includes: the update control parameters corresponding to the first processing module output by the last round of search and update process, and the update control parameters corresponding to the second processing module.

[0173] The image processing method of this application performs multiple rounds of parameter search and update on the original parameter combination to obtain a first parameter combination, which can improve the reliability and accuracy of each control parameter in the first parameter combination, thereby improving the image quality.

[0174] In any embodiment of this application, the implementation principle of any embodiment of this application can be as follows: Figure 8 As shown, it mainly includes the following three parts: Pipeline design, parameter search, and user preference fine-tuning.

[0175] Part One: Pipeline Design.

[0176] (1) Differentiating non-differentiable modules:

[0177] 1.1 Taking the image processing module including the ISP as an example, before designing the pipeline, it is necessary to determine the set S of all modules included in the pipeline. isp (such as LTM, GTM, 3DLUT, etc.), where the pipeline is designed as a serial pipeline, and each module M in the pipeline i ∈S isp That is, M i Only from S isp Choose from.

[0178] 1.2, Regarding S isp The modules in the code are divided into two subsets based on whether they are differentiable or not: and in, Including S isp All differentiable modules, i.e., the control parameters of differentiable modules, can be fine-tuned using the gradient descent method; Including S isp All non-differentiable modules, especially those containing operators such as statistical, maximum / minimum / sorting operators, cannot undergo gradient backpropagation. Therefore, the control parameters of non-differentiable modules cannot be adjusted using gradient descent. and S constitutes isp The set partitioning.

[0179] 1.3, Regarding For all non-differentiable modules, design a neural network to approximate them and obtain an alternative set. right Each non-differentiable module in exist There exists one and only one corresponding differentiable module. Both modules have the same input and output. Can be derived from Figure 3 The neural network shown is implemented. That is, assuming... The control parameter is θ j Then it should be:

[0180]

[0181] In summary, the differentiable replacement of non-differentiable modules has been achieved.

[0182] (2) Use neural network search to determine the optimal pipeline design:

[0183] 2.1 This application makes the following two strong assumptions: the ISP is a serial pipeline, the input of each image processing module is the output of the previous image processing module, and the total number of modules is fixed; the image processing modules in the ISP can only be selected from a fixed set, and there are no other manually defined non-standard modules.

[0184] Based on the above two assumptions, a differentiable pipeline (denoted as the initial processing pipeline in this application) can be formally defined as follows:

[0185] ISP diff (I)=M N (M N-1 (...M1(I,θ1)...,θ N-1 ),θ N (3)

[0186] in, N represents the number of image processing modules included in the ISP.

[0187] 2.2, Use as follows Figure 6 The illustrated two-layer optimization method or other NAS method is used for network architecture updates: The pipeline update process consists of N stages in total. For the nth stage, a learnable weight vector W is introduced. n ∈R 1*(|S|+1) Where |S| is the set S isp The number of modules included, +1 indicates that no modules are set in this stage. Each stage contains all |S| parallel modules and one skip. The outputs of all modules are weighted to obtain the output image of this stage. The weight of each module is softmax(W). n ).

[0188] In the nth stage, the control parameters and weight vectors of the |S| modules can be iteratively updated based on the difference between the output image and the expected image associated with the input image, until the network converges.

[0189] 2.3 For the nth stage, based on the weight vector W of the |S| modules, the module with the largest weight vector can be determined from the |S| modules and used as the module with the nth processing order in the Pipeline.

[0190] (3) Perform a full-link differentiable module control parameter search to determine the initial values ​​of the control parameters (referred to as the original control parameters in this application):

[0191] 3.1. The differentiable pipeline obtained in step (2) (referred to as the candidate processing link in this application) is trained end-to-end using a neural network to obtain the control parameters of each module.

[0192] 3.2 After training, based on the parameter constraints of the non-differentiable modules, the control parameters of the learned differentiable modules are transformed (e.g., quantization, CLIP operation, etc.) to obtain the control parameters corresponding to the non-differentiable modules, thereby obtaining the original control parameters of the entire link.

[0193] Part Two: Parameter Search.

[0194] (1) Calculate the corresponding control parameters for non-differentiable modules: This step is the same as step 3.2 in the Pipeline design phase. Based on the parameter constraints of the non-differentiable modules, the control parameters of the learned differentiable modules are converted into the control parameters of the non-differentiable modules.

[0195] (2) Fix the control parameters of differentiable modules and perform heuristic search for the control parameters of non-differentiable modules: Since operations such as quantization and CLIP may introduce accuracy loss, heuristic algorithms (such as ant colony algorithm, genetic algorithm, etc.) can be used to fine-tune the control parameters of non-differentiable modules to improve the imaging effect.

[0196] (3) Convert the control parameters of the non-differentiable module obtained in step (2) into the control parameters of the associated differentiable module, and use the differentiable module to replace the non-differentiable module. This step has no loss of accuracy.

[0197] (4) Fix the control parameters of the non-differentiable module and perform gradient descent on the differentiable module to search for the optimal control parameters.

[0198] (5) Repeat steps 1-4 until the network converges or the number of iterations reaches the preset maximum number of iterations.

[0199] Part Three: Fine-tuning User Preferences.

[0200] (1) Record the user's photo editing results in the album: Each time the user edits a photo in the album, record the photo editing results (including the final image obtained after editing, as well as the RAW image captured by the image sensor before editing).

[0201] (2) Based on the P-graph results recorded in step (1), fine-tune the control parameters of the differentiable modules in the Pipeline. For example, for each P-graph result, update the control parameters of the differentiable modules once using a small learning rate, thereby avoiding overwriting the original parameter tuning information.

[0202] Optionally, the control parameters obtained after parameter tuning are saved locally until the number of edited images exceeds a certain threshold before being considered for application in the camera.

[0203] (3) During the user's photo taking process, the system prompts the user to choose whether to use the original camera parameters (such as the original parameter combination) or the finely adjusted camera parameters (including the first parameter combination and the second parameter combination), which ensures the accuracy of the ISP's output image and also takes into account the user's personalized customization needs.

[0204] In summary, by replacing the non-differentiable modules in the pipeline with neural networks and performing neural network architecture search to determine the optimal pipeline design, and by using neural networks to search for and fine-tune the optimal parameters to determine the optimal control parameters for the entire scene, and then fine-tuning the control parameters of the differentiable modules according to user preferences, the user's personalized shooting experience can be improved while enhancing the imaging effect.

[0205] To implement the above embodiments, this application also proposes an image processing apparatus.

[0206] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application.

[0207] like Figure 9 As shown, the image processing device 900 may include: a display module 910, a determination module 920, and a processing module 930.

[0208] The display module 910 is used to display prompt information in response to the shooting operation; the prompt information is used to prompt the selection of a parameter combination from at least one parameter combination, the parameter combination including the control parameters of each image processing module in the target processing link of the image processing chip;

[0209] The determination module 920 is used to determine the target parameter combination selected by the triggering object of the shooting operation in response to the selection operation triggered by the prompt information.

[0210] The processing module 930 is used to process the first captured image based on the combination of target parameters through the image processing modules in the target processing link, so as to obtain and display the first target image with optimized image quality.

[0211] Furthermore, in one implementation of the embodiments of this application, in response to the image processing chip including an image signal processor (ISP), the first captured image includes the original captured image captured by the image sensor; in response to the image processing chip including a display processing unit (DPU), the first captured image includes an image after processing the original captured image.

[0212] In one implementation of this application, the parameter combination includes at least one of the following:

[0213] The original parameter combination includes the original control parameters of each image processing module. Each image processing module includes a non-differentiable first processing module and a differentiable second processing module. The original control parameters of the first processing module are determined based on the original control parameters of the associated differentiable third processing module. The input and output data of the first processing module and the associated third processing module are the same.

[0214] The first parameter combination; where the first parameter combination is obtained by searching and updating the original parameter combination;

[0215] The second parameter combination is obtained by adjusting the first parameter combination based on the image editing preferences of the triggering object.

[0216] In one implementation of this application, the second parameter combination is obtained using the following module:

[0217] The first acquisition module is used to determine a second target image with editing records from a plurality of candidate images stored locally; wherein the second target image represents image editing preferences; query a second captured image associated with the second target image; wherein the second target image is generated based on the second captured image; and fine-tune at least some parameters in the first parameter combination based on the second captured image and the second target image to obtain a second parameter combination.

[0218] In one implementation of this application, the first acquisition module is configured to: determine parameters to be adjusted from a first parameter combination; wherein the parameters to be adjusted include control parameters of a differentiable second processing module in an image processing chip; and update the parameters to be adjusted based on a second captured image and a second target image to obtain a second parameter combination.

[0219] In one implementation of this application, the first acquisition module is used to: update the parameters to be adjusted based on the second captured image and the second target image using a target learning rate to obtain a second parameter combination; wherein the target learning rate is lower than a learning rate threshold.

[0220] In one implementation of this application, the target processing link is determined using the following modules:

[0221] The first generation module is used to generate the initial processing link of the image processing chip based on the differentiable second processing module and the differentiable third processing module; wherein the initial processing link includes N differentiable image processing modules connected in series.

[0222] The allocation module is used to assign learnable weight vectors to N differentiable image processing modules connected in series.

[0223] The execution module is used to perform N rounds of iterative update process on the initial processing link based on the weight vector to obtain candidate processing links;

[0224] The second generation module is used to generate the target processing link based on the candidate processing links.

[0225] In one implementation of this application, the nth iteration update process is used to determine the image processing module with processing order n in the candidate processing link. The execution module executes the nth iteration update process, specifically: inputting the input image in parallel to N image processing modules in the initial processing link to obtain the output images of N image processing modules; fusing the output images of N image processing modules to obtain a fused image; wherein, in response to n=1, the weight vectors allocated to the N image processing modules are used to fuse the output images of the N image processing modules; in response to n being a positive integer greater than 1, the weight vectors of the N image processing modules output in the (n-1)th iteration update process are used to fuse the output images of the N image processing modules; based on the fused image and the expected image associated with the input image, the control parameters and weight vectors corresponding to the N image processing modules are iteratively updated to obtain the control parameters and weight vectors corresponding to the N image processing modules output in the nth iteration update process; the image processing module with the largest weight vector output in the nth iteration update process is selected as the image processing module with processing order n in the candidate processing link.

[0226] In one implementation of this application, the second generation module is configured to: train N differentiable image processing modules in the candidate processing link using a first training sample to obtain the original control parameters of the N differentiable image processing modules; replace the third processing module in the candidate processing link with the associated first processing module to obtain the target processing link; and transform the original control parameters of the third processing module based on the parameter constraints of the associated first processing module to obtain the original control parameters of the associated first processing module in the target processing link.

[0227] In one implementation of this application, the first parameter combination is obtained using the following module:

[0228] The second acquisition module is used to perform at least one round of search and update process on the original parameter combination; and to generate the first parameter combination based on the update control parameters corresponding to the first processing module and the second processing module respectively output by the last round of search and update process.

[0229] In one implementation of this application, the second acquisition module executes the first round of search and update process in at least one round of search and update process, specifically: fixing the original control parameters of the differentiable second processing module in the target processing link, performing a heuristic search on the original control parameters of the non-differentiable first processing module to obtain the updated control parameters of the first processing module output in the first round of search and update process; replacing the first processing module in the target processing link with a third processing module associated with the first processing module to obtain the update processing link of the first round of search and update process; updating the control parameters of the third processing module associated in the update processing link of the first round of search and update process according to the updated control parameters of the first processing module output in the first round of search and update process; fixing the control parameters of the third processing module associated in the update processing link of the first round of search and update process, and using the second training sample to update the control parameters of the differentiable second processing module in the update processing link of the first round of search and update process to obtain the updated control parameters of the second processing module output in the first round of search and update process.

[0230] In one implementation of this application, the second acquisition module executes the k-th round of search and update process in at least one round of search and update process. Specifically, it determines whether a set stop update condition is met. If yes, the search and update process ends. If no, the associated third processing module in the update processing chain of the (k-1)-th round of search and update process is updated to the first processing module, and the update control parameters of the second processing module output by the (k-1)-th round of search and update process are fixed. Heuristic search is performed on the update control parameters of the first processing module to obtain the update control parameters of the first processing module output by the k-th round of search and update process. Here, k is a positive integer greater than 1. The process involves replacing the first processing module with a third processing module associated with it to obtain the update processing link for the k-th round of search and update. Based on the update control parameters of the first processing module output by the k-th round of search and update, the control parameters of the third processing module associated with it in the update processing link are updated. The control parameters of the third processing module associated with it in the update processing link are fixed, and the control parameters of the differentiable second processing module in the update processing link of the k-th round of search and update are updated using a third training sample to obtain the update control parameters of the second processing module output by the k-th round of search and update.

[0231] It should be noted that the foregoing explanation of the image processing method embodiments also applies to the image processing apparatus of this embodiment, and will not be repeated here.

[0232] In the image processing apparatus of this application embodiment, a visual display method prompts or guides the user to select the desired parameter combination from at least one parameter combination. The captured image is then processed based on the selected parameter combination. This not only enhances the user's sense of participation and experience but also provides a variety of parameter combinations for users to choose from, adapting to the diverse needs and personalized preferences of different user groups. Based on the target parameter combination selected by the user, the various image processing modules in the target processing chain of the image processing chip can work collaboratively to achieve higher quality image output. This not only improves image processing quality and overall visual appeal but also enhances the user's shooting experience.

[0233] To implement the above embodiments, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image processing method as described in any of the foregoing embodiments.

[0234] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0235] Reference Figure 10 The electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0236] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0237] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of such data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0238] Power component 1006 provides power to various components of electronic device 1000. Power component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0239] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0240] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0241] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0242] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0243] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), Bluetooth, and other technologies.

[0244] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0245] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0246] To implement the above embodiments, this application also proposes a chip, wherein the chip includes an interface circuit and a processing circuit coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is configured to perform the image processing method provided in any of the foregoing embodiments.

[0247] Figure 11 This is a schematic diagram of another chip structure proposed in an embodiment of this application. See also... Figure 11 The diagram shown is a schematic representation of the structure of chip 1100, but it is not limited to this.

[0248] Chip 1100 includes processing circuit 1101, which is configured to perform any of the above image processing methods.

[0249] In some embodiments, chip 1100 further includes one or more interface circuits 1102. Optionally, the interface circuit 1102 is connected to memory 1103, and the interface circuit 1102 can be used to receive signals from memory 1103 or other devices, and the interface circuit 1102 can be used to send signals to memory 1103 or other devices. For example, the interface circuit 1102 can read instructions stored in memory 1103 and send the instructions to processing circuit 1101.

[0250] In some embodiments, the interface circuit 1102 performs at least one of the communication steps such as sending and / or receiving in the above method, while the processing circuit 1101 performs other steps.

[0251] In some embodiments, the terms interface circuit, interface, transceiver pin, transceiver, etc., can be used interchangeably.

[0252] In some embodiments, chip 1100 further includes one or more memories 1103 for storing instructions. Optionally, all or part of the memories 1103 may be located outside of chip 1100.

[0253] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image processing method as described in any of the foregoing method embodiments.

[0254] To implement the above embodiments, this application also proposes a computer program product having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method as described in any of the foregoing method embodiments.

[0255] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0256] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0257] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0258] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0259] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0260] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0261] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0262] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An image processing method, characterized in that, include: In response to the shooting operation, a prompt message is displayed; wherein the prompt message is used to prompt the selection of a parameter combination from at least one parameter combination, the parameter combination including the control parameters of each image processing module in the target processing link of the image processing chip; In response to the selection operation triggered by the prompt information, determine the target parameter combination selected by the triggering object of the shooting operation; The image processing modules in the target processing chain process the first captured image based on the target parameter combination to obtain and display the first target image with optimized image quality.

2. The method according to claim 1, characterized in that, In response to the image processing chip including an image signal processor (ISP), the first captured image includes the original captured image captured by the image sensor; In response to the image processing chip including a display processing unit (DPU), the first captured image includes an image processed from the original captured image.

3. The method according to claim 1, characterized in that, The parameter combination includes at least one of the following: The original parameter combination includes the original control parameters of each image processing module. Each image processing module includes a non-differentiable first processing module and a differentiable second processing module. The original control parameters of the first processing module are determined based on the original control parameters of an associated differentiable third processing module. The input and output data of the first processing module and the associated third processing module are the same. The first parameter combination; wherein the first parameter combination is obtained by searching and updating the original parameter combination; The second parameter combination is obtained by adjusting the first parameter combination according to the image editing preferences of the triggering object.

4. The method according to claim 3, characterized in that, The second parameter combination is obtained using the following steps: A second target image with an editing record is determined from a plurality of candidate images stored locally; wherein the second target image represents the image editing preference; Query the second captured image associated with the second target image; wherein the second target image is generated based on the second captured image; Based on the second captured image and the second target image, at least some parameters in the first parameter combination are fine-tuned to obtain the second parameter combination.

5. The method according to claim 4, characterized in that, The step of fine-tuning at least some parameters in the first parameter combination based on the second captured image and the second target image to obtain the second parameter combination includes: From the first parameter combination, a parameter to be adjusted is determined; wherein, the parameter to be adjusted includes the control parameters of the differentiable second processing module in the image processing chip; Based on the second captured image and the second target image, the parameters to be adjusted are updated to obtain the second parameter combination.

6. The method according to claim 5, characterized in that, The step of updating the parameters to be adjusted based on the second captured image and the second target image to obtain the second parameter combination includes: Using a target learning rate, the parameters to be adjusted are updated based on the second captured image and the second target image to obtain the second parameter combination; The target learning rate is lower than the learning rate threshold.

7. The method according to claim 3, characterized in that, The target processing link is determined using the following steps: Based on the differentiable second processing module and the differentiable third processing module, an initial processing link of the image processing chip is generated; wherein, the initial processing link includes N differentiable image processing modules connected in series; Learnable weight vectors are assigned to the N differentiable image processing modules connected in series, and N rounds of iterative update process are performed on the initial processing link based on the weight vectors to obtain candidate processing links; The target processing link is generated based on the candidate processing link.

8. The method according to claim 7, characterized in that, The nth iteration update process is used to determine the image processing module with processing order n in the candidate processing link. The nth iteration update process includes: The input image is input in parallel to N image processing modules in the initial processing chain to obtain the output image of the N image processing modules; The output images of the N image processing modules are fused to obtain a fused image; wherein, in response to n=1, the weight vectors allocated to the N image processing modules are used to fuse the output images of the N image processing modules; in response to n being a positive integer greater than 1, the weight vectors of the N image processing modules output in the (n-1)th iteration update process are used to fuse the output images of the N image processing modules. Based on the fused image and the expected image associated with the input image, the control parameters and weight vectors corresponding to the N image processing modules are iteratively updated to obtain the control parameters and weight vectors corresponding to the N image processing modules output in the nth iteration update process. The image processing module with the largest weight vector output in the nth iteration update process is selected as the image processing module with the processing order n in the candidate processing link.

9. The method according to claim 7, characterized in that, The step of generating the target processing link based on the candidate processing links includes: The first training sample is used to train N differentiable image processing modules in the candidate processing link to obtain the original control parameters of the N differentiable image processing modules. The third processing module in the candidate processing link is replaced with the associated first processing module to obtain the target processing link; Based on the parameter constraints of the associated first processing module, the original control parameters of the third processing module are transformed to obtain the original control parameters of the associated first processing module in the target processing link.

10. The method according to claim 3, characterized in that, The first parameter combination is obtained using the following steps: Perform at least one round of search and update process on the original parameter combination; The first parameter combination is generated based on the update control parameters corresponding to the first and second processing modules output from the last round of search and update process.

11. The method according to claim 10, characterized in that, The first round of search update process in the at least one round of search update process includes: The original control parameters of the differentiable second processing module in the target processing link are fixed, and the original control parameters of the non-differentiable first processing module are heuristically searched to obtain the updated control parameters of the first processing module output in the first round of search update process. The first processing module in the target processing link is replaced by a third processing module associated with the first processing module to obtain the update processing link of the first round of search update process; Based on the update control parameters of the first processing module output in the first round of search and update process, update the control parameters of the associated third processing module in the update processing chain of the first round of search and update process; The control parameters of the associated third processing module in the update processing link of the first round search update process are fixed, and the control parameters of the differentiable second processing module in the update processing link of the first round search update process are updated using the second training sample to obtain the update control parameters of the second processing module output by the first round search update process.

12. The method according to claim 10, characterized in that, The k-th round search update process in the at least one round search update process includes: Determine whether the set stop update conditions are met; if so, end the search and update process. If not, the associated third processing module in the update processing link of the (k-1)th round search update process is updated to the first processing module, and the update control parameters of the second processing module output by the (k-1)th round search update process are fixed. A heuristic search is performed on the update control parameters of the first processing module to obtain the update control parameters of the first processing module output by the kth round search update process; where k is a positive integer greater than 1. The first processing module is replaced by a third processing module associated with the first processing module to obtain the update processing link of the k-th round of search update process; Based on the update control parameters of the first processing module output by the k-th round of search and update process, update the control parameters of the associated third processing module in the update processing link of the k-th round of search and update process; The control parameters of the associated third processing module in the update processing link of the k-th round search update process are fixed, and the control parameters of the differentiable second processing module in the update processing link of the k-th round search update process are updated using the third training sample to obtain the update control parameters of the second processing module output by the k-th round search update process.

13. An image processing apparatus, characterized in that, include: The display module is used to display prompt information in response to the shooting operation; wherein the prompt information is used to prompt the selection of a parameter combination from at least one parameter combination, the parameter combination including the control parameters of each image processing module in the target processing link of the image processing chip; The determination module is used to determine the target parameter combination selected by the triggering object of the shooting operation in response to the selection operation triggered by the prompt information; The processing module is used to process the first captured image based on the target parameter combination through each image processing module in the target processing link, so as to obtain and display the first target image after image quality optimization.

14. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1 to 12.

16. A chip, characterized in that, The chip includes an interface circuit and a processing circuit that are coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is used to implement the method of any one of claims 1 to 12.

17. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.