Image processing model training, image processing method, device, equipment and medium

Abstract

The application discloses an image processing model training method, an image processing method, an image processing device, an image processing equipment and a medium. The image processing model training method is applied to an electronic device and includes the following steps: obtaining a first image; obtaining operation information of a user operating a second image; the second image is obtained by processing the first image by using an image processing model of the electronic device; the operation information includes preference information of the user for the second image; and the image processing model is trained according to the operation information, the second image and the first image, so as to obtain an optimized image processing model. The embodiment of the application can improve the satisfaction degree of the user for the image processed by the electronic device.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing model training, image processing method, apparatus, device and medium. Background Technology

[0002] With the development of computer technology, electronic devices with camera functions have become widely used. Simultaneously, the camera performance of these devices has continuously improved. After the camera module captures the original image, the device can further process it using image processing models to optimize the image quality. However, users have varying needs for image optimization, so not all users may be satisfied with the images processed by the image processing models. Summary of the Invention

[0003] This application provides an image processing model training method, apparatus, device, and medium, which can improve user satisfaction with images processed by the image processing model. The technical solution is as follows:

[0004] In a first aspect, an image processing model training method is provided, applied to an electronic device, comprising: obtaining a first image; obtaining operation information of a user operating a second image; the second image being obtained by the image processing model of the electronic device processing the first image; the operation information including preference information of the user for the second image; and training the image processing model based on the operation information, the second image, and the first image to obtain an optimized image processing model.

[0005] In this embodiment, the electronic device can pre-set an image processing model, and the first image can be any image used as input to the image processing model. The first image can be captured by the electronic device's camera, obtained from a network, or acquired from another device. The operation information can be generated when the user expresses preference for the second image by clicking a control. The operation information can include information generated when the user deletes the second image if dissatisfied, information generated when the user saves the second image if satisfied, and editing information generated when the user edits the second image.

[0006] An image processing model can be used to adjust and process a first image output by an electronic device's ISP to obtain an image that the electronic device will display to the user. The image processing model performs at least one image processing sub-operation on the first image. An optimized image processing model can perform N sub-operations on the first image, where N is an integer greater than or equal to 0, including at least one image processing sub-operation and N sub-operations.

[0007] In one possible implementation, the trained image processing model can be used to disable at least some of the image processing functions in the pre-trained image processing model. Alternatively, the trained image processing model can be used to perform at least some of the image processing functions in the pre-trained image processing model.

[0008] When training an image processing model, if the operation information includes editing information generated when the user edits the second image, a third image that the user finds more satisfactory can be determined based on this information. The image processing model is then trained based on the differences between the third and first images, resulting in an optimized model. In this case, the third image can serve as a positive sample for training the image processing model.

[0009] When training an image processing model, if the operation information includes operation information generated from a user's collection of a second image, a third image can be determined based on this operation information. The third image is identical to the second image. The image processing model is then trained using both the second and first images. In this case, the second image serves as a positive sample for training the image processing model.

[0010] When training an image processing model, if the operation information includes editing information generated when a user deletes a second image, a negative sample identifier can be added to the second image, enabling the image processing model to recognize the second image as a negative sample during training. In this embodiment, a negative sample can represent a processing result that the image processing model cannot produce when processing an image. By adding negative samples, overfitting of the image processing model to positive samples can be prevented.

[0011] Using the above method, the electronic device can obtain operation information, which represents the user's preference information for the second image. Then, based on the operation information, the second image obtained by the electronic device, and the first image before image processing, the image processing model is trained, making the image processing model more in line with the user's preferences. In a possible implementation, the image processing model before training is a model that can implement preset image processing functions after learning from a certain number of training samples. The optimized image processing model can implement some or all of the preset image processing functions, or cancel at least some of the preset image processing functions. Therefore, when the user subjectively does not like the image processing effect of the image processing model, the user's preference information can be determined based on the user's operation information, and some image processing functions in the image processing model can be canceled, avoiding the inability to meet the personalized image processing preferences and needs of different users by pre-setting the same image processing model for all electronic devices of the same model.

[0012] In one implementation, training the image processing model based on the operation information, the second image, and the first image to obtain an optimized image processing model includes: sending the operation information, the second image corresponding to the operation information, and a downsampled image of the first image to a target server; receiving model update information returned by the target server; the model update information is used to determine the optimized image processing module, and the model update information is obtained by the server based on the second image, the operation information, and the downsampled image of the first image.

[0013] By downsampling the first image, we can preserve the content and features of the first image before the image processing model processes it, while also saving storage space on the electronic device and avoiding the need to consume more memory to store the information of the first image. Furthermore, this method allows the image processing model to be trained on the server side, thus utilizing the powerful computing and processing capabilities of the server and improving the training efficiency of the image processing model.

[0014] In one embodiment, sending the first image to a target server includes: sending a downsampled image of the first image to the target server, wherein the downsampled image of the first image is obtained by downsampling the first image according to a preset downsampling ratio.

[0015] The downsampling factor mentioned above can also be called the downsampling frequency, which can be used to represent the scaling factor between the downsampled image and the original image. When the downsampling factor is 1, the downsampled image of the first image is the first image.

[0016] In one embodiment, the image processing model training method further includes: downsampling the first image to obtain a downsampled image of the first image; and storing the downsampled image of the first image in the photo storage record of the second image.

[0017] The above implementation method enables automatic downsampling and storage of the first image after it is obtained, and timely recording of necessary information about the first image. This avoids the situation where the first image cannot be obtained because the image processing model processes the first image and then directly updates it with the second image.

[0018] In one embodiment, training the image processing model based on the operation information, the second image, and the first image to obtain an optimized image processing model includes: updating the second image based on the operation information to obtain a third image; training the image processing model based on the first image and the third image to obtain model update information; and obtaining the optimized image processing model based on the model update information.

[0019] The third image mentioned above can replace the second image as the result of being captured by an electronic device. Model update information may include: information on image processing functions retained in the image processing model, and / or information on image processing functions removed from the image processing model.

[0020] Using the above method, the server and electronic device can update the image processing model by sending model update information, thus eliminating the need to send the entire image processing model and reducing the amount of data sent.

[0021] In one embodiment, the photo storage record of the second image includes the second image and its metadata.

[0022] If the user's operation information on the second image includes editing information for changing the second image, the photo storage record of the second image may include a third image.

[0023] By using the metadata of the second image, the downsampled images of the second image and the first image can be quickly found.

[0024] In one implementation, training the image processing model based on the first image and the third image to obtain model update information includes: inputting a downsampled image of the first image and a downsampled image of the third image into the image processing model, and the image processing model obtaining the model update information based on the downsampled image of the first image and the downsampled image of the second image.

[0025] By using the above method, the image processing model can be trained using downsampled images of the first and third images. This allows for retraining of the image processing model with less data, reducing the amount of data processing required during retraining.

[0026] In one embodiment, the operation information of the second image includes: information for marking the second image, and / or image editing information for changing the second image.

[0027] By using the above methods, we can obtain user preference information for the second image from multiple perspectives, determine the learning materials for the image processing model from multiple perspectives, and collect as many images as possible that are beneficial to the improvement of the image processing model.

[0028] In one embodiment, the image processing model is a neural network model pre-configured in the terminal device.

[0029] In this embodiment, the neural network model pre-configured on the terminal device is a neural network model that has pre-learned a certain number of model training samples. The neural network model can refer to a model that uses algorithms to simulate human neural networks to process data.

[0030] The above method can utilize the learning capabilities of neural network models to learn user preferences from a second or third image, thereby providing pre-learning functions based on user preferences.

[0031] In one embodiment, the first image is output from the post-processing end of the processing algorithm flow module of the electronic device; or, the first image is output from the back end of the processing algorithm flow module of the electronic device.

[0032] By using the above method, the image processing model can be set after the processing algorithm flow module of the ISP without affecting the processing algorithm of the ISP.

[0033] Secondly, embodiments of this application also provide an image processing model training method applied to a server, comprising: obtaining a first image and a third image; the third image being obtained by updating the second image based on user operation information on the second image; the second image being obtained by the image processing model processing the first image; training the image processing model based on the first image and the third image to obtain an optimized image processing model; and sending information about the optimized image processing model to an electronic device.

[0034] Thirdly, embodiments of this application also provide an image processing method, including: obtaining a first image; processing the first image using an updated image processing model to obtain a fourth image; wherein the updated image processing model is an optimized image processing model provided in any embodiment of this application.

[0035] Fourthly, embodiments of this application provide an electronic device, which includes: a processor and a memory;

[0036] The memory is used to store a program for an electronic device to perform the method provided in any embodiment of the present application, and to store data involved in implementing the method provided in any embodiment of the present application;

[0037] The processor is configured to execute programs stored in memory.

[0038] Optionally, there may be one or more processors and one or more memories.

[0039] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0040] The processing device in the fourth aspect above can be a chip. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0041] In the specific implementation process, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. This application does not limit the type of memory or the way the memory and processor are set.

[0042] Fifthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the method described in the first aspect.

[0043] Sixthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform any of the possible implementations of the first aspect.

[0044] In a seventh aspect, embodiments of this application also provide a processor, including: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive signals through the input circuit and transmit signals through the output circuit, causing the processor to execute the method in any of the embodiments of the first aspect described above.

[0045] In specific implementation, the processor can be a chip, the input circuit can be input pins, the output circuit can be output pins, and the processing circuit can be transistors, gate circuits, flip-flops, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to and transmitted by a transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as both the input circuit and the output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0046] The technical effects achieved by the second, third, fourth, fifth, sixth, and seventh aspects mentioned above are similar to the technical effects achieved by the corresponding technical means in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0047] Figure 1This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of a software architecture according to an embodiment of this application;

[0049] Figure 3 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0050] Figures 4A to 4C This is a schematic diagram illustrating other possible application scenarios of the embodiments of this application;

[0051] Figure 5 This is a flowchart illustrating the image processing model training method according to an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of a possible structure of the image processing model provided in the embodiments of this application;

[0053] Figure 7 A schematic diagram of the user interface for an embodiment of this application;

[0054] Figure 8 This is a schematic diagram of the image storage structure according to an embodiment of this application;

[0055] Figure 9 This is a flowchart illustrating a training method for an image processing model, as exemplified in this application.

[0056] Figure 10 This is a flowchart illustrating the training method of an image processing model according to an example of this application.

[0057] Figure 11 This is a schematic diagram of the structure of an image processing model training device according to an embodiment of this application;

[0058] Figure 12 This is a schematic diagram illustrating the conversion between the first image and the second image in an embodiment of this application. Detailed Implementation

[0059] In the following description, specific details such as particular system architectures and technologies are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0060] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0061] It should be understood that "one or more" as mentioned in this application refers to one, two, or more, and "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0062] Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0063] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0064] The display method provided in this application can be applied to electronic devices. These electronic devices can be mobile phones, tablets, wearable devices, digital cameras, in-vehicle devices, augmented reality (AR) devices, virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), laptops, etc., and this application does not limit the specific application to these devices.

[0065] First, the possible structures of the electronic devices in the embodiments of this application will be introduced.

[0066] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0067] See Figure 1 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It should be noted that, Figure 10 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 10 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 10 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 10 The components shown are sub-components of certain components. For example, Figure 10 The proximity sensor 180G shown is optional. Figure 10 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0069] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices.

[0070] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0071] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0072] Figure 1 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0073] Electronic device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] The display screen 194 can be used to display images or videos. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0075] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0076] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0077] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard red-green-blue (RGB) formats, luminance, and chrominance (YUV). In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0078] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0079] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0080] An NPU (Neural Processing Unit) is a processor that borrows from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, to rapidly process input information and continuously learn. NPUs can enable intelligent cognitive functions in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0081] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0082] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, such as in a shooting scenario, the electronic device 100 can utilize the distance sensor 180F to measure distance for fast focusing.

[0083] Button 190 includes a power button and volume buttons. Motor 191 can generate vibration when the electronic device receives information. Motor 191 can be used to display notification information.

[0084] exist Figure 1 Based on the electronic device shown, the user can control the camera 193 to capture images or videos via button 190 or display screen 194.

[0085] Figure 1 The electronic device shown can be configured with a certain software architecture. Figure 2 This is a schematic diagram of an architecture (including a software system and some hardware) applied in an embodiment of this application. Figure 2 As shown, the architecture of an electronic device is divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the application architecture can be divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer.

[0086] like Figure 2 As shown, the application layer includes the camera and gallery. This is understandable. Figure 2 The application layer in the document may only include a portion of the applications in the electronic device's application layer. In fact, the application layer of an electronic device can also include other applications, which this application does not limit. For example, the application layer may also include applications such as messaging, alarm clock, weather, stopwatch, compass, timer, flashlight, calendar, and payment programs.

[0087] like Figure 2 As shown, the application framework layer, also known as the application programming framework layer, can include a camera access interface. The camera access interface includes camera management and camera devices. The hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer includes multiple camera devices. The camera algorithm library includes post-processing algorithm modules and decision-making modules.

[0088] It should be understood that the decision-making module can also be placed in other layers. As one possible implementation, the decision-making module can be placed in the application layer or the application framework layer.

[0089] The driver layer is used to drive hardware resources. The driver layer can include multiple driver modules. For example... Figure 2 As shown, the driver layer includes camera device drivers, digital signal processor drivers, and graphics processor drivers, etc.

[0090] The hardware layer includes sensors, image signal processors, digital signal processors, and graphics processors. The sensors include various types of sensors, time-of-flight (TOF) cameras, and multispectral sensors. The image signal processor includes a front-end, a back-end, and a post-processing unit. The steps performed by the front-end, back-end, and post-processing unit can be implemented through software units or modules within the image signal processor. Alternatively, in different design architectures, the image signal processor may also include a front-end and a back-end.

[0091] For example, a user can tap the camera application to activate the camera on their electronic device and take a picture. When the user taps the camera application icon to take a picture, the shooting command is sent to the camera hardware abstraction layer (HIP) through the camera access interface. The HIP responds to the shooting command by calling the camera device driver and the camera algorithm library. The camera algorithm library can be configured with the image processing model provided in this embodiment. The decision module in the camera algorithm library determines the shooting mode (e.g., a first mode, a second mode, or a third mode) based on the zoom level and ambient light, and sends the configured parameters (including the sensor output method, parameter configurations of each ISP module, and parameter configurations of the post-processing algorithm module) to the HIP. The HIP sends the parameters configured by the decision module to the camera device driver. The camera device driver sends the configuration parameters sent by the HIP to the hardware layer. For example, the decision module in the camera algorithm library can send the sensor output method to the sensor and the parameter configurations of each ISP module to the image signal processor. The sensor outputs an image based on the sensor output method. The image signal processor performs corresponding processing based on the parameter configurations of each ISP module. The camera algorithm library also sends digital signals to the digital signal processor (DSP) driver in the driver layer, enabling the DSP driver to invoke the DSP in the hardware layer for digital signal processing. The DSP can then return the processed digital signals to the camera algorithm library via its driver. The camera algorithm library also sends digital signals to the image signal processor (ESP) driver in the driver layer, enabling the ESP driver to invoke the graphics processor (GPU) in the hardware layer for digital signal processing. The GPU can then return the processed image data to the camera algorithm library via its driver. The camera algorithm library then uses a pre-configured image processing model to further process the image data.

[0092] Additionally, the image output from the image signal processor can be sent to the camera device driver. The camera device driver can then send the image output from the image signal processor to the camera hardware abstraction layer. The camera hardware abstraction layer can then send the image to the post-processing algorithm module for further processing, or it can send the image to the camera access interface. The camera access interface can then send the image returned by the camera hardware abstraction layer to the camera at the application layer.

[0093] The image processing method provided in this application can be applied to scenarios where electronic devices capture images or videos. The following provides examples of scenarios where this application embodiment can be applied.

[0094] Figure 3 This is a schematic diagram illustrating an application scenario according to an embodiment of this application. Figure 3 In the scenario depicted, a user might use various electronic devices to take photos, resulting in images with different effects. These differences might include varying lighting conditions, different image processing models within the same electronic device, and the resulting image effects. Furthermore, different image processing models might employ different image processing algorithms to process the captured images. Figure 3 As shown, in an indoor shooting scenario, the user used electronic device model 1 and electronic device model 2 to photograph the human body, respectively. Figure 3 The image captured by the electronic device of Model 1 shown in (a) is color-graded using a first target color tone. Figure 3 In (a) of the image, color correction can be represented using shading. For example... Figure 3 The image captured by the Model 2 electronic device shown in (b) was not color-graded using the first target tone. Users may have concerns about this. Figure 3 The images captured by the electronic device of model 2 shown in (b) are more satisfactory.

[0095] Furthermore, the same user's preference for images taken with different electronic devices may not be entirely constant. If the subject and shooting context differ, the user's preference for images taken with different electronic devices will also differ. In other words, a user's satisfaction with images taken with the same electronic device is not always fixed. For example, Figure 3 (c) and Figure 3 As shown in (d), under strong ambient light (e.g., near a window or outdoors), the user uses electronic devices of models 1 and 2 respectively to capture images of the human body. Figure 3 The image captured by the electronic device of Model 1 shown in (c) is color-graded using the second target color tone. Figure 3In (c) of the diagram, color correction is also indicated using shading. For example... Figure 3 The image taken by the Model 2 electronic device shown in (d) did not undergo color correction using the second target tone. In this scenario, the user may be dissatisfied with the images taken by both Model 1 and Model 2 electronic devices.

[0096] Besides adding target tones to images, image processing models in electronic devices can also process captured images in other ways. These include adjusting contrast, color temperature, saturation, brightness, darkness, or sharpness.

[0097] Figure 4A This is a schematic diagram illustrating another possible scenario for an embodiment of this application. Figure 4A In the scenario shown, the electronic device can be configured with an artificial intelligence stylization module (AI stylization), which processes images acquired by the electronic device's camera. The AI ​​stylization module is equivalent to... Figure 2 The image processing model in the software architecture is shown. Before the AI ​​stylization module, the electronic device can also preprocess the image using the ISP's processing algorithm pipeline module. ISP, or Image Signal Processor, is generally used to process the raw image data output by the image sensor (such as CMOS or CCD). When processing the raw image data, the ISP may use the 3A algorithm to obtain image signal processing parameters, and then use the image signal processing parameters to process the raw image to generate a preview data stream.

[0098] The aforementioned 3A algorithms may include an automatic exposure (AE) algorithm, an automatic white balance (AWB) algorithm, and an automatic focus (AF) algorithm. The automatic exposure algorithm is used to determine exposure parameters. The automatic white balance algorithm is used to determine white balance parameters. The automatic focus algorithm is used to determine focus parameters. Correspondingly, image signal processing parameters may include at least one of the exposure parameters, white balance parameters, and focus parameters.

[0099] Figure 4AThe scenario shown illustrates a possible processing algorithm pipeline for an ISP. During the processing operations performed by the ISP, multiple image processing algorithms can be invoked. In this embodiment, at least one image processing algorithm is used to process the digital signal of the raw image captured by the sensor of the electronic device. During image capture, the lens projects light signals within its field of view (equivalent to the acquired data in other embodiments of this application) onto the photosensitive area of ​​the image sensor. The image sensor then performs photoelectric conversion on the light signals and inputs Bayer format raw images into the ISP frontend (FE) at certain time intervals. Multiple raw image frames input to the ISP frontend constitute a raw image data stream. The ISP's FE can be the first processing node after the image enters the processing algorithm pipeline module. The FE can be primarily responsible for the initial processing of the image signal. During the FE's image data processing stage, the FE can receive raw image data from the input device and perform preliminary processing and conversion on the raw image data in the raw domain to obtain the frontend data stream. The FE can then pass the processed frontend data stream to subsequent nodes of the ISP module. The image processing operations performed by the FE of the ISP module can include image cropping (to remove unwanted parts or adjust image size), format conversion, etc. A statistics submodule can be configured in the front-end data stream output by the ISP module to perform statistical analysis on the front-end data stream, obtain statistical data, and adjust (or update, or correct) the image signal processing parameters of the 3A algorithm based on the statistical data.

[0100] The front-end data stream output by the FE (Feature Frame) of the ISP module can be stored in the memory of the electronic device. The back-end (BE) of the ISP module can obtain at least one second image frame from the front-end data stream in the memory of the electronic device, and the PE (Back-End Frame) can process the data output by the FE. The memory of the aforementioned electronic device may include double data rate synchronous dynamic random access memory (DDRSDRAM). Compared with the FE of the ISP module, the BE of the ISP module can be responsible for performing more complex image processing operations, such as noise reduction, color correction, automatic exposure, and automatic white balance, making the image frames clearer, more natural, and more in line with human visual perception. The data output by the BE can be called the back-end data stream.

[0101] The back-end data stream output by the BE of the ISP module is stored in the DDR synchronous dynamic random access memory of the electronic device. The post-processing end (PE) of the ISP module retrieves the back-end data stream from the DDR synchronous dynamic random access memory and performs further processing on at least one third image frame in the YUV domain to obtain the PE post-processing end data stream.

[0102] exist Figure 4A In the illustrated ISP module structure, the electronic device can configure a tiny pipe at the back end of the processing algorithm flow module. The ISP module receives a tiny stream (tiny stream) from the BE node output through the tiny pipe, which can be used for face detection. The tiny stream can be a back-end data stream output by the BE node, which can be used for face detection, portrait segmentation, and optical flow information calculation, etc. The tiny stream has a lower resolution, which can be lower than the original image data obtained by the image sensor. The original image data can also include multiple video frames. For example, the resolution of the original image data can be 4096×3072, while the resolution of the tiny stream may be 720×540.

[0103] exist Figure 4A In the scenario shown, the aforementioned AI stylization module is set after the PE node. The AI ​​stylization module can process at least one fourth image frame in the post-processing data stream output by the PE node. In possible implementation scenarios, the AI ​​stylization module can perform at least one of the following functions.

[0104] Function 1: Image Enhancement

[0105] The AI ​​stylization module can also be used for image enhancement, making images more vivid and distinct by adjusting parameters such as contrast, saturation, and brightness.

[0106] Function 2: Image Restoration

[0107] The AI ​​stylization module can identify and repair imperfections in images, such as noise and scratches, thereby improving the overall image quality.

[0108] Function 3: Add filters

[0109] The image is processed using the default filter style, giving the image a filter effect.

[0110] Function 4: Tone mapping

[0111] Tone mapping is a step in the image reproduction process. It maps the lighting of the original scene to the luminous intensity of the display device, and is primarily used to convert high dynamic range (HDR) images to low dynamic range (LDR) images. During tone mapping, the AI ​​stylization module may utilize... Figure 4B The HDR network shown is shown.

[0112] Figure 4B The HDRnet shown may include a downsampling module 41, a coefficient network 42, a lookup table 43, and a convolutional network 44. The image frames output by the ISP module are processed in two branches. In branch 1, the full-resolution input data (full-res input) of the image frames output by the ISP is downsampled by the downsampling module 41 to obtain low-resolution input data. The low-resolution input data is then input to the coefficient network 42, which processes the low-resolution input data to obtain a bilateral grid for coefficient transformation. The bilateral grid includes bilateral slice data. The bilateral grid can be a three-dimensional array (xyz array) containing a two-dimensional spatial domain and a one-dimensional value domain. Specifically, its first two dimensions (x, y) correspond to the two-dimensional spatial domain of the image, and the third dimension z corresponds to the image intensity or color value. In branch 2, the full-resolution input data is color-corrected by lookup table 43, and the color-corrected data is then convolved by convolutional network 44 to obtain a guide map. The guide map contains crucial information about the original image, which can guide the execution of the image processing algorithm. By utilizing the guide map, the image processing algorithm can more accurately identify features in the image, thereby optimizing the processing results. The bilateral slice data is then applied to the guide map to obtain the output image. The aforementioned bilateral slice data refers to the data generated by the slicing operation. In a bilateral grid, slicing refers to the process of extracting a piecewise smooth two-dimensional image from the bilateral grid. Specifically, slicing can be an operation based on the third dimension of the bilateral grid, extracting image segments with specific intensity or color values ​​by selecting different thresholds or ranges.

[0113] In the embodiments of this application, downsampling can refer to reducing the sampling rate of a signal or image by reducing the number of data points. In some cases, downsampling can typically refer to reducing the resolution or size of an image, that is, reducing the number of pixels in the image. Through downsampling, the image size becomes smaller.

[0114] During tone mapping, the AI ​​stylization module may utilize... Figure 4C The three-dimensional lookup table (3D LUT) shown is shown.

[0115] Figure 4C The 3D LUT shown may include a downsampling module 45 and a correlation coefficient network 46. The image frames output by the ISP module are processed through two branches. In branch 1, the full-resolution data of the image frames output by the ISP module is downsampled by the downsampling module 41 to obtain low-resolution data. The low-resolution data is input into the correlation coefficient network 46 to obtain a 3D lookup table result. The 3D lookup table result is then applied to the full-resolution data in branch 2 to obtain the output image.

[0116] exist Figures 4A to 4C In the scenario shown, the ISP module can obtain images captured by the camera of an electronic device and process them. During the image processing process, the ISP may call image processing models such as HDRnet or 3D LUT to implement the image processing process.

[0117] exist Figures 3 to 4C In the illustrated scenario, the image processing model may be deployed locally on the electronic device or in the cloud. When the image processing model is deployed locally on the electronic device, the ISP module can call the local image processing model to process images captured by the electronic device's camera. In this embodiment, the cloud can also be referred to as the server. When the image processing model is deployed in the cloud, the electronic device can call the cloud-deployed image processing model to process the post-processing data stream output by the ISP module.

[0118] To provide users with a better photography experience, current electronic devices, by default, call image processing models to process images output by the ISP module (including post-processing data streams). However, some image processing models called by the ISP module do not produce images that meet users' requirements for image quality. Therefore, this application provides an image processing method that can satisfy users' personalized preferences for image processing. This method includes, for example,... Figure 5 The steps are shown.

[0119] Step S51: The electronic device acquires the first image.

[0120] In this embodiment of the application, "the electronic device obtains a first image" can refer to the electronic device obtaining a captured first image. The first image can refer to... Figure 4A The image output by the processing algorithm module shown. That is, the first image can be... Figure 4A The output of the node preceding the AI ​​stylization module shown.

[0121] In other possible implementations, the first image may be output from the image sensor of the electronic device, or it may be output from the ISP module of the electronic device. The first image may include a digital signal of the image obtained from an analog signal of the image.

[0122] In one possible implementation, the first image acquired by the electronic device may be captured by a camera pre-configured within the electronic device. For example, if the electronic device is a mobile phone, the first image could be captured by the phone's camera. The first image captured by the camera of the electronic device could include an image acquired by the electronic device's image sensor, or an image processed by the electronic device's ISP module after processing the image acquired by the image sensor.

[0123] In other possible implementations, the first image can also be obtained by an electronic device operating the camera of another device, or by an electronic device operating an externally connected camera. For example, the electronic device can be a desktop computer, which can send a shooting command to an externally connected camera based on user instructions to obtain the first image.

[0124] In another possible implementation, the electronic device is pre-configured with an image processing model. The first image is an arbitrary image that has not yet been processed by the image processing model.

[0125] In another possible implementation, the electronic device acquiring the first image may also refer to any of the following situations.

[0126] Scenario 1: The electronic device obtains a first image that has been pre-stored locally in the photo album.

[0127] In one possible implementation, the first image might be an image pre-stored in the photo album of an electronic device. The source of the pre-stored images in the photo album could include images generated by a camera, etc.

[0128] Scenario 2: The electronic device acquires a first image pre-stored in an external device connected to the electronic device.

[0129] External devices that connect to electronic devices may include external storage devices, external computing devices, etc.

[0130] Scenario 3: The electronic device receives the first image sent over the network.

[0131] The aforementioned network transmission may include wired network transmission, near-field communication network transmission, or wireless network transmission. The first image transmitted via the network may be sent by other terminal devices through the network or by the server through the network.

[0132] Scenario 4: The electronic device generates the first image according to the user's instructions.

[0133] User instructions can include instructions for editing images or instructions for drawing images.

[0134] The aforementioned instructions for editing images can refer to editing based on an already generated image. The aforementioned instructions for drawing images can refer to drawing image content and creating an image on a blank canvas of an electronic device's display screen.

[0135] Case 5: The first image can be any image that needs to be processed by the image processing model.

[0136] For example, the first image can be processed by the image processing model N times, and then the image needs to be processed by the image processing model again, where N is an integer greater than or equal to 1.

[0137] In another possible implementation, the first image obtained by the electronic device can be any image, and may also include images that have already been processed by an image processing model.

[0138] In possible implementations, the electronic device acquiring the first image can also refer to the electronic device acquiring at least one image frame from a video captured by its camera, where the at least one image frame in the video is the first image. The electronic device can acquire at least one image frame from a preview stream generated in real-time by the camera through the ISP module, where the at least one image frame in the preview stream is the first image. The electronic device can also acquire an image upon receiving a shooting command from the user, i.e., the first image is a photograph taken.

[0139] When an electronic device's camera captures an image, the camera's sensor captures light within the field of view (FOV) in real time, generating an image preview stream. In a possible implementation, at least one image frame in the image preview stream may be pre-processed by a processing algorithm module, and then the electronic device obtains the at least one image frame processed by the processing algorithm module. The at least one image frame processed by the processing algorithm module constitutes the first image.

[0140] Step S52: The electronic device stores the image data of the first image.

[0141] In this embodiment of the application, the electronic device can directly store the first image, for example, the electronic device can store the first image in a photo album. Alternatively, the electronic device can process the first image to obtain and store the image data of the first image.

[0142] In other possible implementations, the electronic device may also store the first image at a designated external location, thus implementing step S52.

[0143] In a possible implementation, the electronic device can store image data in the original format of the first image. The original format of the first image can be an image format captured by the electronic device. The image format captured by the electronic device can include any of the following: graphics interchange format (GIF), portable network graphics (PNG), raw format, bitmap format, and joint photographic experts group (JPEG) format, etc.

[0144] In other possible implementations, the electronic device may perform format conversion on the first image to obtain and store the image data of the first image.

[0145] In one possible implementation, step S52 may further include: the electronic device downsampling the first image to obtain a downsampled image of the first image; and the electronic device storing the downsampled image of the first image in the photo storage record of the second image. If the first image is an image captured by the electronic device, the electronic device can downsample the first image after capturing it to obtain a downsampled image, and then store the downsampled image of the first image in the album, in the same storage location as the second image. If the first image is an image captured by the electronic device, after obtaining the first image, the electronic device can input the first image into an image processing model to obtain a second image output by the image processing model. The second image is used as the image captured in response to the user's shooting operation, presented to the user, and stored in the album.

[0146] In one possible implementation, the electronic device can store a downsampled image of the first image in the same location as the second image. Furthermore, the electronic device can store the downsampled image of the first image in the same location as the second image through hidden storage. In another possible implementation, the electronic device can perform a series of transformations on the pixels of the second image to embed the downsampled image of the first image into the second image, such that the second image with the embedded downsampled image of the first image is virtually indistinguishable from the original second image in appearance.

[0147] In possible implementations, the image format of the downsampled image of the first image can be the same as that of the first image. The image format of the second image can be the same as or different from that of the first image. Similarly, the image format of the second image can be the same as or different from that of the downsampled image of the first image. For example, the first image, the downsampled image of the first image, and the second image can all be in JPEG format.

[0148] In one possible implementation, the electronic device may store a downsampled image of the first image in the same location as the second image by at least one of the following methods.

[0149] Method 1: Airspace concealment.

[0150] Electronic devices can directly modify the pixels of the original image, embedding the information that needs to be hidden into the pixel values.

[0151] For example, when hiding in the spatial domain, the least significant bit (LSB) substitution method can be used to process the pixels of the second image. LSB substitution refers to embedding the binary code of the information to be hidden into the least significant bit of the pixels of the original image. Since the human eye is not sensitive to small changes in an image, it is almost imperceptible to the eye.

[0152] Method 2: Frequency domain hiding.

[0153] Electronic devices can transform images to embed the information to be hidden into the frequency domain of the image, taking advantage of the different sensitivities of the human eye to images to achieve information hiding.

[0154] When hiding information in the frequency domain, electronic devices can use the discrete cosine transform (DCT) to process the pixels of the second image. The DCT transform converts the image to the frequency domain, embeds the information to be hidden into the frequency domain coefficients, and finally restores the image through the inverse transform.

[0155] In one possible implementation, the photo storage record for the second image includes the second image and its metadata. For the image, the metadata can be a set of tags embedded in the image file that describe image attributes, source, shooting data, etc. The metadata can be used for image recognition, classification, retrieval, and management. In another possible implementation, the downsampled image of the first image and the second image can correspond to the same metadata.

[0156] For example, the storage format of the first image and its corresponding metadata is as follows: Figure 8As shown. When the second image is generated, the electronic device can simultaneously generate metadata for the second image, associate the second image with the metadata of the second image, and then merge and store the downsampled image of the first image with the second image.

[0157] Step S53: The electronic device inputs the first image into the image processing model to obtain the second image output by the image processing model.

[0158] In a possible implementation, the electronic device can input the first image output by the ISP module into the image processing model after the ISP module, and the image processing model may include the AI ​​stylization module in the foregoing embodiments.

[0159] In one possible implementation, the electronic device can also input the digital signal of the image obtained by the sensor into the image processing model, whereby the digital signal obtained by the sensor is the first image.

[0160] In possible implementations, the image processing model can be implemented in at least one of the following ways.

[0161] Method 1: The image processing model can be generated based on mathematical formulas. For example, the image processing model can perform a series of mathematical operations on the input first image to obtain the second image. In Method 1, the image processing model can be an image processing method and parameters used to analyze and process images.

[0162] Method 2: The image processing model can include modules with data processing capabilities. For example, the image processing model includes encoding modules, decoding modules, watermark removal modules, etc. In Method 2, the image processing module can be a functional module that is a set of image processing operation instructions and parameters for analyzing and processing image data.

[0163] Method 3: The image processing model can include a neural network model, which can be trained through supervised learning. The image processing model processes the input data to obtain output data, and improves its internal parameters based on the differences between the output data and reference data, completing at least one training iteration. Through multiple training iterations, the output data obtained by the image processing model after processing the input data can become infinitely close to the reference data. The aforementioned reference data can be the output data that the image processing model is expected to obtain based on the input data.

[0164] In some examples of this application, images marked as favorites by the user or edited by the user can be used as reference data. In Method 3, the image processing model can be a mathematical model for analyzing and processing image data, a neural network model obtained from a set of parameters and algorithms, a combination of neural network models, or a branch of a neural network model.

[0165] Method four: The image processing model can be a self-learning neural network model. A self-learning neural network model refers to a neural network model that can learn in an unsupervised manner. During the learning and optimization process, a self-learning neural network model can learn without manually labeled data.

[0166] In possible implementations, the neural network model can include a deep learning model and / or a machine learning model. The unoptimized image processing model may have been obtained through supervised learning or self-learning before optimization.

[0167] In one possible implementation, the electronic device can be pre-configured with multiple different neural network models to be applied to multiple different shooting scenarios. For example, the electronic device can be a mobile phone, which has built-in multiple photo shooting scenarios, including portrait scenes, night scene scenes, landscape scenes, seaside scenes, and indoor scenes, etc. Each scenario may be configured with a corresponding image processing model, and the image processing models for different scenarios may be completely different, partially the same, or completely identical. When the electronic device inputs the first image into the image processing model, it can first obtain the current shooting scene, and then input the first image into the image processing model corresponding to the current shooting scene according to the current shooting scene.

[0168] In another possible implementation, the electronic device can internally configure a unique image processing model, which can have multiple branches. Each scene can correspond to one branch or a combination of two or more branches. When the first image is input, the current shooting scene can be obtained first. The current shooting scene and the first image are then input into the image processing model, and the first image is processed using at least one branch within the image processing model.

[0169] In one possible implementation, the image processing model may include multiple processing modules, which can process the input image data in parallel or sequentially. For example, the image processing model may include processing module 1, processing module 2, and processing module 3. The input data of the image processing model is processed sequentially by processing module 1, processing module 2, and processing module 3 to obtain the output data of the image processing model, for example... Figure 6 As shown in (a) above. For example, an image processing model might include processing module 4, processing module 5, and processing module 6, such as... Figure 6 As shown in (b), the input data is input to processing module 4 and processing module 5 respectively. The data output by processing module 4 and processing module 5 are combined and then input to processing module 6 to obtain the output data of the image processing model.

[0170] In another possible implementation, the image processing model may include multiple sub-models, each of which is used to implement a corresponding image processing function. For example, the image processing model may include a first sub-model, a second sub-model, and a third sub-model. The first sub-model may be used to process the background tone, the second sub-model may be used to process the subject tone, and the third sub-model may be used to soften the lighting effect.

[0171] Step S54: The electronic device obtains operation information of the user operating the second image; the operation information is used to indicate the user's preference for the second image.

[0172] Among the possible implementations, Figure 5 The method shown can be triggered based on the user's operation information on the second image. That is, whenever the user uses the camera function of the electronic device to open the camera application and take a picture, the electronic device can execute steps S51 to S53 in response to the user's use of the camera function. After receiving the user's operation information on the second image, the electronic device can continue to execute steps S54 and S55.

[0173] Alternatively, in another possible implementation, the electronic device can perform actions at set time intervals. Figure 5 The method is as shown. That is, whenever a user uses the camera function of an electronic device to take a picture, the electronic device, in response to the user's use of the camera function, can execute steps S51 to S53. After receiving the user's operation information on the second image, the electronic device can store the operation information of the second image, or the electronic device can update the second image based on the operation information of the second image to obtain a third image and store the third image. When a set time interval is reached, the electronic device can continue to execute steps S54 and S55. In this case, the electronic device can obtain the first image and the third image each time it trains the image processing model, and train the image processing model based on the first image and the third image. Alternatively, the electronic device can record the operation information each time it obtains the user's operation information on the second image, and then when the set time interval is reached, the electronic device can obtain the first image, the operation information, and the corresponding second image, and then the electronic device can train the image processing model based on the first image, the operation information, and the second image. Thus, for a single device, data closure and sub-learning of the image processing model can be achieved. In other words, during the training of the image processing model, no intervention from anyone other than the user of the electronic device is required to optimize the image processing model on the electronic device, thus avoiding the leakage of image data used for further training of the image processing model.

[0174] In this embodiment of the application, the operation information of the second image may include: information for marking the second image, and / or, image editing information for changing the second image.

[0175] In possible implementations, the information used to mark the second image may include information about adding it to favorites via a click-to-favorite control. For example... Figure 7 As shown in (a), the photo display interface includes a photo display area 73 and a control area 72. The control area 72 can be positioned above and / or below the photo display area 73 (in the visual direction). The control area 72 can include multiple controls for manipulating images, such as a favorite control 74, a delete control 75, and an edit control 76. The user can click the favorite control 74, and the electronic device, in response to the user's click, adds a favorite mark to the second image. After the user favorites at least one second image, the user can also view the second image with the favorite mark in the "My Favorites" sub-album. The user can also click the delete control 75, and the electronic device, in response to the user's click, moves the second image to the "Deleted" sub-album, where images in the "Deleted" sub-album will no longer be displayed. Therefore, the information for marking the second image can also include information about deleting the image by clicking the delete control.

[0176] In possible implementations, the information used to manipulate the second image can be information from the photo album of an electronic device or information from a third-party application.

[0177] In other possible implementations, the information used to mark the second image can also be information from controls other than the favorite and delete functions that mark the second image.

[0178] In possible implementations, the editing information used to change the second image may include information about how the user edits the second image. For example, in Figure 7 Based on the image display interface shown in (a), the user can click the editing control 76. In response to the click of the control 76, the electronic device can display... Figure 7 The editing interface shown in (b) is referenced. Figure 7 As shown in (b), the image editing interface includes multiple lighting and shadow effect adjustment controls. Users can adjust the lighting and shadow effects of the second image using the corresponding lighting and shadow effect adjustment controls.

[0179] exist Figure 7In (b) of the diagram, the lighting and shadow effect adjustment controls may include an exposure adjustment control 77, a sharpness adjustment control 78, and a highlight adjustment control 79, etc. The selected control can be highlighted by making its outline bold. The user can click on a specific lighting and shadow effect adjustment control. In response to the click operation, the electronic device displays an adjustment toolbar 710 below the control. When the user slides the adjustment toolbar 710 left or right, the electronic device responds to the user's sliding operation of the adjustment toolbar 710 and can adjust the corresponding lighting and shadow effects such as exposure effect, sharpness effect, and highlight effect.

[0180] In another possible implementation, the operation information for the second image may further include labeling information for generating training samples based on the second image. This labeling information for generating training samples may differ from the information described above for labeling the second image. The labeling information for generating training samples may be specifically used to generate training samples for the image processing model. For example, in an interface displaying an image, a control for labeling the second image can be provided. The user can operate this control, and the electronic device, responding to the user's operation, can perform simple labeling on the second image, making it a training sample. The information for labeling the second image can be used to indicate the user's preference for or against the second image. For example, the information for labeling the second image could indicate that the user considers the image brightness too low. Alternatively, the user can operate on the second image to cancel the image processing model's processing effect on the second image; correspondingly, the operation information for the second image could indicate that the image processed by the image processing model is unsatisfactory.

[0181] In one possible implementation, after obtaining the user's operation information, the second image can be updated immediately to obtain the third image. At the same time, the metadata corresponding to the original second image is associated with the third image. The first image and the third image are stored together. The pixel values ​​of the third image are changed according to the first image, so that the first image is hidden.

[0182] In this embodiment of the application, since the first image is downsampled first to obtain a downsampled image of the first image when storing the first image, the memory space occupied by the downsampled image of the first image is significantly smaller than that of the first image, thus saving storage space of the electronic device.

[0183] Step S55: The electronic device trains the image processing model based on the operation information, the second image, and the first image to obtain an optimized image processing model; the optimized image processing model is used to implement some functions of the image processing model, or to cancel at least some functions of the image processing model.

[0184] In one possible implementation, the electronic device can send a first image, a second image, and operation information to a target server, where an image processing model is trained. For example, the electronic device trains the image processing model based on the operation information and the first image to obtain an optimized image processing model, which may include: the electronic device sending the operation information, the second image corresponding to the operation information, and a downsampled image of the first image to the target server; the electronic device receiving model update information returned by the target server; the model update information is used to determine the optimized image processing module, and the model update information is obtained based on the downsampled image of the second image and the first image.

[0185] When the image processing model is updated via the server, after the target server receives the first image, the second image, and the operation information, it can update the second image according to the operation information to obtain the third image. Then, based on the first image (or a downsampled image of the first image) and the third image (or a downsampled image of the third image), the image processing model is trained to obtain an optimized image processing model.

[0186] In another possible implementation, the image processing model can be trained locally on the electronic device. In this embodiment, the information used to label the second image can also be referred to as the labeling information of the second image. For example, when the operation information includes labeling information, in this case, the third image is the second image. Accordingly, the electronic device trains the image processing model based on the operation information and the first image to obtain an optimized image processing model. This may include: the electronic device downsampling the second image corresponding to the labeling information to obtain a downsampled image of the second image; the electronic device training the image processing model based on the downsampled image of the first image and the downsampled image of the second image to obtain model update information; and the electronic device obtaining the optimized image processing model based on the model update information.

[0187] The aforementioned model update information may include information about model modules that need to be omitted. A model module can be a module within an image processing model. For example, if an image processing model includes model module 1, model module 2, and model module 3, the model update information may include the identifier of model module 2. Accordingly, the model update information indicates that model module 2 in the image processing model can be deleted or omitted.

[0188] In possible implementations, the image processing model may be a standalone model. The image processing model may internally include multiple model branches. When the image processing model is a standalone model, the electronic device can obtain a second image based on the currently selected shooting scene using the corresponding image processing model. During image processing model training, downsampled images of the second and first images can be input into the model, allowing it to calculate a loss function based on the second and first images. The parameters of the image processing model are then adjusted according to the loss function to obtain an optimized image processing model. In this embodiment, the shooting scene can also be called a shooting mode. For example, a shooting mode or shooting scene may include: night scene, landscape, portrait, still life, or close-up, etc. The electronic device can be configured with image processing models corresponding to multiple different shooting modes, i.e., an image processing model for night scene shooting, an image processing model for landscape shooting, an image processing model for portrait shooting, an image processing model for still life shooting, or an image processing model for close-up shooting. When the user selects the portrait shooting mode, the electronic device can input downsampled images of the second and first images into the image processing model corresponding to the portrait.

[0189] When the image processing model is a standalone model with branches, at least one branch of the image processing model is used to process the first image in the currently selected shooting scene of the electronic device to obtain the second image. During training, the electronic device can input a downsampled image of the first image and a third image into the image processing model, allowing the model to learn the differences between the downsampled image of the first image and the third image (or a downsampled image of the third image) and optimize accordingly. The third image is the second image updated with operation information; that is, if the operation information is for editing the second image, the third image may be different from the second image, while if the operation information is for labeling the second image, the third image may be the second image.

[0190] In another possible implementation, the image processing model can learn from the first image, the second image, and operational information without downsampling either the first or second image. That is, in step S55, the electronic device can first update the second image based on the operational information to obtain the third image. Then, the electronic device can input the first and third images into the image processing model, train the image processing model based on the first and third images, and obtain an optimized image processing model. In yet another possible implementation, the electronic device can downsample the third image to obtain a downsampled image of the third image. Then, the downsampled image of the first image and the downsampled image of the third image can be used to further train the image processing model to obtain an optimized image processing model.

[0191] When an image processing model includes at least one functional module, the electronic device can determine, based on the differences between the first and third images, at least one target functional module that needs to be retained in the image processing model, and remove at least one functional module. For example, if a user edits the second image, removing the contrast processing effect of the image processing model on the second image, and the electronic device obtains the third image based on the operation information and the second image, then the electronic device can determine, based on the first image (or a downsampled image of the first image) and the third image (or a downsampled image of the third image), that the target functional module used for contrast processing in the image processing model can be deleted, and the remaining functional module in the image processing model is the at least one functional module that needs to be retained.

[0192] When the image processing model includes multiple different model branches, and each shooting scene corresponds to at least one model branch, the electronic device can input the first image and the third image into the model branch corresponding to the shooting scene, and train the model branch corresponding to the shooting scene based on the first image and the third image, thereby optimizing the image processing model.

[0193] In the case where an image processing model includes multiple different model branches, and each model branch can be removed, the model branch can also be regarded as a functional module.

[0194] In one possible implementation, electronic devices can train an image processing model in a supervised manner to obtain an optimized image processing model.

[0195] In one possible implementation, when the electronic device executes step S55, it can label the third image as the target image and the downsampled image of the first image as the input image. Then, the electronic device can simultaneously input both the input image and the target image into the image processing model and process the image using the image processing model. The target image can be the image output by the expected image processing model based on the input image.

[0196] In another possible implementation, when the electronic device performs step S55, the electronic device can mark the third image as the target image and mark the downsampled image of the first image as the input image. Then, the electronic device can input the input image into the image processing model to obtain the processed image. Then, the electronic device can calculate the loss value based on the processed image and the third image, input the loss value into the image processing model, and train the image processing model.

[0197] Because users have different subjective preferences, their preference for the processing effect of a uniformly configured image processing model on electronic devices also varies. Figure 3 Taking the image shown as an example, user A might... Figure 3 User B is more satisfied with the image shown in (a) and may prefer it. Figure 3 The image shown in (b) is more satisfactory. The method provided in this application embodiment can be applied to electronic devices to further train the image processing model of the electronic device based on user operation information of the image. This allows the image processing method of the image processing model deployed on the electronic device to better suit the user's preferences, thereby improving the user's experience of taking photos with the electronic device.

[0198] In one possible implementation, the electronic device can hide and store a downsampled image of the first image, which may include: the electronic device can convert the downsampled image of the first image into the format of the description data of the second image to obtain the converted first data, the converted data including the pixel values ​​of all pixels in the downsampled image of the first image; then, the electronic device can add the converted first data to the metadata of the second image.

[0199] Because servers have more powerful computing and storage capabilities, in one embodiment of this application, image processing models can be trained using a server, and the server can perform actions such as... Figure 10 The steps are shown.

[0200] Step S1001: The server receives the third image from the electronic device.

[0201] In one possible implementation, the third image is obtained by the electronic device marking and / or editing the second image based on user operation information. After obtaining the third image, the electronic device can send it to the server.

[0202] In other possible implementations, the server can also receive the second image and operation information sent by the electronic device, update the second image according to the operation information, and obtain the third image.

[0203] The second image mentioned above is obtained by processing the first image using the image processing model that needs to be trained.

[0204] Step S1002: The server parses the downsampled image of the first image based on the third image.

[0205] The downsampled image of the first image can be stored in an array of the third image and the metadata of the third image (that is, the data obtained by the electronic device updating the metadata of the second image when the electronic device changes the second image to the third image).

[0206] In other possible implementations, the electronic device can send a downsampled image of the third image and a downsampled image of the first image to the server. If the second image is not changed to the third image by the electronic device according to operation information, the electronic device can send both the downsampled images of the second and first images to the server. When the electronic device performs the downsampling operation on the third image and sends both the downsampled images of the first and third images to the server, the downsampled image of the third image can be stored merged with the downsampled image of the first image. For example, on the electronic device side, the downsampled images of the third and first images can be embedded separately within the third image, or they can be hidden within the third image. Alternatively, the downsampled images of the third and first images can be displayed merged and independent of the third image; the merged downsampled image includes both the downsampled images of the third and first images.

[0207] When the server parses the downsampled image of the first image, it can separate the downsampled image of the first image from the third image.

[0208] Step S1003: The server downsamples the third image to obtain a downsampled image of the third image.

[0209] In one possible implementation, the size of the downsampled image of the third image can be the same as the size of the downsampled image of the second image, which makes it easier for the image processing model to learn the difference information between images from images of the same size, and optimize the parameters inside the model based on the difference information.

[0210] In other possible implementations, the server can also upsample the downsampled image of the first image to obtain an upsampled image of the first image with the same size as the second (or third) image. In this embodiment, upsampling can be the inverse process of downsampling, which refers to increasing the resolution of the data, i.e., increasing the number of data points. Through upsampling, the server can enlarge the image. When the server enlarges the downsampled image of the first image, it can enlarge the downsampled image of the first image to the size of the first image, obtaining a fifth image. Then, the server can train an image processing model based on the fifth image and the third image.

[0211] Step S1004: The server uses the downsampled image of the third image and the downsampled image of the first image to train the image processing model and obtain the optimized image processing model.

[0212] When training an image processing model, the server can label the downsampled image of the third image as the target image and the downsampled image of the first image as the input image. Then, the target image and the input image are simultaneously input into the image processing model for training.

[0213] Because of the server's strong computing power, the downsampled image of the first image can be input into the image processing model on the server side. Based on the processing result of the downsampled image of the first image and the third image, the loss value is calculated. Then the server can input the loss value into the image processing model, so that the image processing model can be trained and optimized.

[0214] Step S1005: The server sends the optimized image processing model to the electronic device.

[0215] In one example of this application, such as Figure 9 As shown, the image processing model training method includes the following steps.

[0216] Step S91: The electronic device receives the user's command to take a picture.

[0217] In this example, the user can send a photo-taking command to the camera application by tapping the camera control or pressing the shutter button on the camera interface. In possible implementations, the photo-taking command may include instructions to take a photo and / or instructions to record video.

[0218] In other possible implementations, the aforementioned camera interface can be generated by a dedicated camera application configured in the electronic device, or it can be generated by a non-dedicated application. For example, a third-party application on the electronic device may have a real-time chat function. In the real-time chat input box, the camera of the electronic device can be activated, and a photo-taking command can be sent to the camera of the electronic device.

[0219] Step S92: Based on the photographing command, the electronic device obtains a downsampled image of the first image and a second image.

[0220] The second image can be obtained by processing the first image using the electronic device's AI stylization module. From the user's perspective, the second image is the result of taking a photo presented to the user by the electronic device based on the photo-taking command.

[0221] Reference Figure 12 As shown, after the electronic device obtains the first image, it can simultaneously input the first image into the downsampling module and the AI ​​stylization module to obtain a downsampled image of the first image output by the downsampling module and a second image output by the AI ​​stylization module.

[0222] Step S93: The electronic device stores the downsampled image of the first image and the second image into the photo album of the electronic device.

[0223] In this system, a downsampled version of the first image can be hidden and stored in the photo album. Furthermore, the first and second images can be stored together, with the second image displayed and the first image hidden within the merged file. Thus, the electronic device can locate and obtain both the second image and the downsampled version of the first image using the metadata of the second image.

[0224] Step S94: The electronic device receives operation information from the user regarding the operation of the second image.

[0225] For example, the operational information may only affect the image presented to the user, without changing the downsampled image of the first image that has already been stored.

[0226] Step S95: The electronic device changes the second image to the third image according to the operation information.

[0227] Step S96: The electronic device stores the third image on the server side.

[0228] The electronic device stores the third image on the server side, which can refer to the electronic device sending the third image to the server or uploading the third image to the cloud. Since the downsampled image of the first image and the third image are stored together, when the electronic device sends the third image to the server, it also sends the downsampled image of the first image.

[0229] Alternatively, electronic devices can store third-party images locally. In this case, the electronic device does not need to send the third-party image to the server.

[0230] In one possible implementation, when step S96 is executed, the AI ​​stylization module of the electronic device may have already undergone at least one optimization, and the AI ​​stylization modules of different electronic devices may be different. If the server stores the latest updated AI stylization modules for each electronic device, the server can generate model numbers for different AI stylization modules. When an electronic device uploads a third image to the server, it can simultaneously upload the electronic device's identifier and / or the current AI stylization module number, enabling the server to obtain the correct version of the AI ​​stylization module for training.

[0231] In another possible implementation, the electronic device can record the differences between the optimized AI stylization module and the original AI stylization module each time the AI ​​stylization module is trained. If the AI ​​stylization module of the electronic device has been optimized at least once when step S96 is executed, the electronic device can send the difference information between the current AI stylization module and the original AI stylization module to the server, so that the server can train and optimize the AI ​​stylization module based on the latest AI stylization module.

[0232] In another possible implementation, if the AI ​​stylization module of the electronic device has been optimized at least once when step S96 is executed, the electronic device can send the latest AI stylization module when sending the third image to the server, so that the server can train and optimize based on the latest AI stylization module.

[0233] Step S97: The server parses the downsampled image of the first image in the third image.

[0234] The aforementioned server can also be referred to as the server-side.

[0235] Step S98: The server downsamples the third image to obtain a downsampled image of the third image.

[0236] The server can also downsample the third image. The resulting downsampled image of the third image is close in size to the downsampled image of the first image. The image processing model can more easily learn the difference between the downsampled images of the third and first images and train based on the difference between the downsampled images of the third and first images.

[0237] Step S99: The server marks the downsampled image of the first image as the input image and the downsampled image of the third image as the target image.

[0238] Step S910: The server inputs the input image and the target image into the AI ​​stylization module, retrains the AI ​​stylization module, and obtains the optimized AI stylization module.

[0239] The retrained AI stylization module is the latest version of the AI ​​stylization module for the electronic device. If steps S97 to S910 are executed on the server side, the server side stores the AI ​​stylization module configured on the electronic device. If the AI ​​stylization module of the electronic device has been optimized at least once, the server side can also store the optimized version of the AI ​​stylization module.

[0240] Step S911: The server sends the optimized AI stylization module to the electronic device.

[0241] Steps S97 to S910 can also be performed locally on the electronic device. In this case, step S911 does not need to be performed.

[0242] Step S912: The electronic device replaces the original AI stylization module with the optimized AI stylization module.

[0243] After replacing the AI ​​stylization module in electronic devices, the optimized AI stylization module can be used to process images output by the ISP module.

[0244] In one possible example, the first image could be an image frame in a preview data stream of the camera interface of an electronic device.

[0245] exist Figure 9 In the illustrated embodiments, the electronic device can utilize the server-side computing and storage capabilities to optimize the AI ​​stylization module, reducing local resource consumption and facilitating faster training and optimization of the AI ​​stylization module. When an electronic device is uniformly configured with an image processing model, different users may have varying levels of satisfaction with the model's processing effects. Some users may prefer the processed image, while others may prefer the unprocessed image. The method provided in this application can optimize the uniformly configured image processing model, retaining or deleting at least some of its functions based on user interaction with the second image, thus addressing the issue that a uniformly configured image processing model cannot meet the personalized aesthetic needs of different users.

[0246] This application also provides an image processing method that can be applied to electronic devices. The image processing method of this application embodiment may include: obtaining a first image captured by an electronic device; processing the first image using an optimized image processing model; and obtaining a fourth image. The optimized image processing model may be obtained by training an image processing model using the image processing model training method provided in any embodiment of this application.

[0247] This application also provides another image processing method that can be applied to electronic devices. In this image processing method, the electronic device uses an optimized image processing model to process at least one image frame in the preview stream obtained by the camera of the electronic device to obtain at least one fourth image, and displays the fourth image in the preview area of ​​the camera interface.

[0248] This application also provides a photographing method that can be applied to electronic devices. The photographing method provided in this application may include: obtaining a photographing command; responding to the photographing command, obtaining a first image; processing the first image using an optimized image processing model to obtain a fourth image; and outputting the fourth image as a photograph. Furthermore, during the photographing process, the optimized image processing model processes the first image to obtain the fourth image. When further training the optimized image processing model, the fourth image can become a new second image. If the user edits and / or annotates the fourth image, the optimized image processing model can be further trained based on the user's operation information on the new second image.

[0249] The image processing model training apparatus provided in this application embodiment can be used to perform, for example... Figure 5 And the steps of the method provided in any related embodiments. An image processing model training apparatus provided in this application includes, as... Figure 11 The first image acquisition module, operation information acquisition module, and optimization module are shown.

[0250] The first image acquisition module is used to acquire the first image.

[0251] An operation information acquisition module is used to acquire operation information of the user operating the second image; the second image is obtained by the image processing model of the electronic device processing the first image; the operation information includes information indicating the user's preference for the second image.

[0252] An optimization module is used to train the image processing model based on the operation information, the second image, and the first image to obtain an optimized image processing model.

[0253] This application also provides an image processing model training device applied on a server side, which can be used to perform, for example... Figure 10 The methods and steps provided in related embodiments are also described. The image processing apparatus provided in this application includes: an image acquisition module, an image processing model training module, and a transmission module.

[0254] An image acquisition module is used to acquire a first image and a third image; the third image is obtained by updating the second image based on the user's operation information on the second image; the second image is obtained by processing the first image by the image processing model.

[0255] The image processing model training module is used to train the image processing model based on the first image and the third image to obtain an optimized image processing model.

[0256] The sending module is used to send information about the optimized image processing model to electronic devices.

[0257] This application also provides an image processing apparatus for processing a first image captured by a camera using an optimized image processing model, wherein the optimized image processing model is obtained by the image processing model training method provided in any embodiment of this application.

[0258] This application also provides an imaging device capable of capturing images using an image processing model optimized according to any image processing model training method provided in this application. In one embodiment, the imaging device includes an image acquisition module and an image processing module.

[0259] An image acquisition module is used to acquire a first image using the ISP of an electronic device.

[0260] The imaging module is used to process the first image using an optimized image processing model to obtain the fourth image.

[0261] The imaging device provided in this application embodiment can execute the image processing method provided in any embodiment of this application, or can execute the image processing method and image processing model training method provided in any embodiment of this application.

[0262] This application also provides an electronic device having the image processing apparatus provided in any embodiment of this application.

[0263] This application also provides an electronic device having the imaging device provided in any embodiment of this application.

[0264] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0265] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0266] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0267] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0268] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip system can be a single chip or a chip module composed of multiple chips.

[0269] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0270] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. An image processing model training method, applied to electronic devices, characterized in that, include: Obtain the first image; Obtain user interaction information with the second image; The second image is obtained by processing the first image using the image processing model of the electronic device; the operation information includes information indicating the user's preference for the second image; Based on the operation information, the second image, and the first image, the image processing model is trained to obtain an optimized image processing model; the optimized image processing model is used to implement some of the functions of the image processing model, or to cancel at least some of the image processing functions of the image processing model.

2. The method according to claim 1, characterized in that, The step of training the image processing model based on the operation information, the second image, and the first image to obtain an optimized image processing model includes: Send the operation information, the second image corresponding to the operation information, and the first image to the target server; The server receives model update information returned by the target server; the model update information is used to determine the optimized image processing module, and the model update information is obtained by the server based on the second image, the operation information, and the downsampled image of the first image.

3. The method according to claim 2, characterized in that, Sending the first image to the target server includes: A downsampled image of the first image is sent to the target server. The downsampled image of the first image is obtained by downsampling the first image according to a preset downsampling ratio.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The first image is downsampled to obtain a downsampled image of the first image; The downsampled image of the first image is stored in the photo storage record of the second image.

5. The method according to any one of claims 1-4, characterized in that, The step of training the image processing model based on the operation information, the second image, and the first image to obtain an optimized image processing model includes: Based on the operation information, the second image is updated to obtain the third image; Based on the first image and the third image, train the image processing model to obtain model update information; The optimized image processing model is obtained based on the model update information.

6. The method according to claim 4 or 5, characterized in that, The photo storage record of the second image includes the second image and its metadata.

7. The method according to claim 5 or 6, characterized in that, Based on the first image and the third image, the image processing model is trained to obtain model update information, including: The downsampled images of the first image and the third image are input into the image processing model, and the image processing model obtains the model update information based on the downsampled images of the first image and the third image.

8. The method according to any one of claims 1-7, characterized in that, The operation information for the second image includes: information for marking the second image, and / or image editing information for changing the second image.

9. The method according to any one of claims 1-8, characterized in that, The image processing model is a neural network model pre-configured on the terminal device.

10. The method according to any one of claims 1-9, characterized in that, The first image is output from the post-processing end of the processing algorithm module of the electronic device; or, The first image is the backend output of the processing algorithm flow module of the electronic device.

11. An image processing model training method, applied on a server side, characterized in that, include: Obtain the first and third images; The third image is obtained by updating the second image based on the user's operation information on the second image; The second image is obtained by processing the first image using the image processing model; Based on the first image and the third image, the image processing model is trained to obtain an optimized image processing model; Send the optimized image processing model information to the electronic device.

12. An image processing method, characterized in that, include: Obtain the first image; The first image is processed using an updated image processing model to obtain a fourth image; the updated image processing model is the optimized image processing model as described in any one of claims 1-11.

13. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory is used to store a program for the electronic device to perform the method as described in any one of claims 1-12, and to store data related to implementing the method as described in any one of claims 1-12; The processor is configured to execute programs stored in the memory.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-12.

Images