Image processing method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure relates to an image processing method and device, electronic equipment, storage medium and program product. The method comprises: obtaining an image to be processed and a scene parameter associated with the image; processing the image based on a preset image processing model and the scene parameter to obtain a processed image; wherein the image processing model is obtained by training a preset neural network model based on first sample images in multiple different scenes. Through the method, the quality of the processed image can be improved, and the flexibility and intelligence are high.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the continuous development of technology, cameras are being used more and more widely in daily life, and people's requirements for image quality are gradually increasing. As a result, image processing technology used to improve image quality has become increasingly important. However, current image processing technology mainly relies on fixed algorithm processes, applying the same processing procedures and parameters to images in different scenes. This often fails to achieve precise optimization, resulting in limited improvement in image quality and poor flexibility. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, electronic device, storage medium, and program product.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising:

[0005] Obtain the image to be processed and the scene parameters associated with the image;

[0006] Based on a preset image processing model and the scene parameters, the image is processed to obtain a processed image; wherein, the image processing model is obtained by training a preset neural network model based on first sample images under multiple different scenes.

[0007] In some embodiments, the scene parameters include a first parameter associated with the shooting mode of the image and a second parameter of the shooting environment;

[0008] The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including:

[0009] Based on the shooting mode of the image, a first model is determined; wherein, the first model is obtained by training the image processing model using a second sample image associated with the shooting mode of the image.

[0010] Based on the first model and the second parameters, the image is processed to obtain a processed image.

[0011] In some embodiments, the image shooting modes include multiple types, the first model includes multiple types, and each first model is associated with a shooting mode;

[0012] The step of processing the image based on the first model and the second parameters to obtain the processed image includes:

[0013] Based on each first model and the second parameter, the image is processed to obtain the image after processing each first model;

[0014] The images processed by each of the first models are fused to obtain the fused image.

[0015] In some embodiments, the method further includes:

[0016] Determine the imaging device used to capture the image;

[0017] The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including:

[0018] Based on the image capturing device, a second model is determined; wherein the second model is obtained by training the image processing model using a third sample image associated with the image capturing device.

[0019] Based on the second model and the scene parameters, the image is processed to obtain a processed image.

[0020] In some embodiments, the scene parameters further include a third parameter indicating the style of the processed image;

[0021] The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including:

[0022] Based on a preset image processing model and the third parameter, the image is processed to obtain an image that conforms to the image style indicated by the third parameter.

[0023] In some embodiments, the image is a RAW image; the processing of the image based on a preset image processing model and the scene parameters to obtain a processed image includes:

[0024] Based on a preset image processing model and the scene parameters, the RAW image is subjected to image signal processing to obtain the processed image.

[0025] In some embodiments, the first sample images under various different scenarios include at least one of the following:

[0026] First sample images under various shooting modes;

[0027] First sample images captured by various different shooting devices;

[0028] First sample images under various shooting environments;

[0029] First sample images of various different image styles.

[0030] According to a second aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0031] The acquisition module is configured to acquire the image to be processed and the scene parameters associated with the image;

[0032] The processing module is configured to process the image based on a preset image processing model and the scene parameters to obtain a processed image; wherein the image processing model is obtained by training a preset neural network model based on first sample images under multiple different scenes.

[0033] In some embodiments, the scene parameters include a first parameter associated with the shooting mode of the image and a second parameter of the shooting environment; the processing module is further configured to determine a first model based on the shooting mode of the image; wherein the first model is trained on the image processing model using a second sample image associated with the shooting mode of the image; and the image is processed based on the first model and the second parameter to obtain a processed image.

[0034] In some embodiments, the image capture mode includes multiple modes, the first model includes multiple modes, and each first model is associated with a capture mode; the processing module is further configured to process the image based on each first model and the second parameter to obtain an image processed by each first model; and to fuse the images processed by each first model to obtain a fused image.

[0035] In some embodiments, the apparatus further includes:

[0036] The determination module is configured to determine the capturing device for capturing the image;

[0037] The processing module is further configured to determine a second model based on the image capturing device; wherein the second model is trained on the image processing model using a third sample image associated with the image capturing device; and the image is processed based on the second model and the scene parameters to obtain a processed image.

[0038] In some embodiments, the scene parameters further include a third parameter indicating the style of the processed image; the processing module is further configured to process the image based on a preset image processing model and the third parameter to obtain an image that conforms to the image style indicated by the third parameter.

[0039] In some embodiments, the image is a RAW image; the processing module is further configured to perform image signal processing on the RAW image based on a preset image processing model and the scene parameters to obtain the processed image.

[0040] In some embodiments, the first sample images under various different scenarios include at least one of the following:

[0041] First sample images under various shooting modes;

[0042] First sample images captured by various different shooting devices;

[0043] First sample images under various shooting environments;

[0044] First sample images of various different image styles.

[0045] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0046] processor;

[0047] Memory used to store computer programs or instructions;

[0048] The processor executes the computer program or instructions to implement the steps of the image processing method described in the first aspect above.

[0049] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions which, when executed by a processor, implement the steps of the image processing method described in the first aspect above.

[0050] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the image processing method described in the first aspect above.

[0051] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0052] In this embodiment of the disclosure, the electronic device acquires the image to be processed and the scene parameters of the associated image, and processes the image based on a preset image processing model and the scene parameters to obtain the processed image. On the one hand, when processing the image to be processed, by combining the scene parameters of the associated image, the model can more accurately understand the image content and its contextual information, thereby generating an image that conforms to the scene parameters, which can achieve precise image processing and improve the visual effect of the processed image. On the other hand, since the image processing model is trained based on sample images in various different scenes, the model can adjust the processing strategy according to the scene of the image when facing the image to be processed, thereby improving the quality of the processed image, and has high flexibility and intelligence.

[0053] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0055] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0056] Figure 2 This is an example diagram illustrating the development process of an image processing method according to an exemplary embodiment.

[0057] Figure 3 This is an architectural diagram illustrating an image processing method according to an exemplary embodiment.

[0058] Figure 4 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0059] Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0060] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0061] Figure 1This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 As shown, the method mainly includes the following steps:

[0062] S11. Obtain the image to be processed and the scene parameters associated with the image;

[0063] S12. Based on a preset image processing model and the scene parameters, the image is processed to obtain a processed image; wherein, the image processing model is obtained by training a preset neural network model based on first sample images under multiple different scenes.

[0064] In this embodiment, the image processing method can be applied to electronic devices with image processing capabilities, such as user equipment (UE), mobile devices, user terminals, mobile phones, tablets, personal digital assistants (PDAs), handheld devices, computing devices, and in-vehicle devices. It can also be applied to wearable devices, cameras, camcorders, and other electronic devices with photo-taking or video-taking capabilities. It should be noted that the image processing method in this embodiment can be applied to terminal devices or cloud devices such as servers; however, this embodiment does not limit its application in this regard. The following description focuses on electronic devices as the execution subject.

[0065] In step S11, the electronic device acquires an image to be processed. This image can be a RAW image, an RGB image, or a YUV image; the format of the image to be processed is not limited in this embodiment. In some embodiments, the electronic device may be equipped with an image acquisition component, which acquires the image to be processed. This image acquisition component may be, for example, a front-facing camera or a rear-facing camera. In other embodiments, the electronic device may also acquire images from other devices.

[0066] The method of this disclosure embodiment can be applied to the shooting scenario of an electronic device. For example, the electronic device can acquire an image to be processed through an image acquisition component in response to the detection of a shooting command. The shooting command can be a touch command detected by the electronic device, a voice command detected by the electronic device, or the execution of an application within the electronic device that triggers the start of the image acquisition component. The electronic device can detect the shooting command based on the execution of the application.

[0067] The method of this disclosure can also be applied to image post-processing scenarios, such as images that have been captured and stored in an electronic device, or images received by the electronic device from other devices.

[0068] In this embodiment of the disclosure, the electronic device also acquires scene parameters of the associated image. In some embodiments, after acquiring the image to be processed, the electronic device can use a preset algorithm to identify the image in order to obtain the scene parameters of the associated image. In other embodiments, the electronic device may also be equipped with sensors, such as temperature sensors, humidity sensors, ambient light sensors, etc., and the electronic device can use the sensors to synchronously acquire the scene parameters of the associated image when capturing the image to be processed.

[0069] In other embodiments, the electronic device can also acquire scene parameters of associated images input by the user. For example, the user can set scene parameters of associated images when taking a picture of the image to be processed, such as shutter speed, aperture size, and other shooting parameters. The user can also annotate the image after taking the picture, such as the shooting location and weather conditions. The electronic device can acquire scene parameters of associated images based on the user's annotations. The user can also input information indicating the image style of the processed image. In other embodiments, the electronic device can also utilize cloud computing and big data technologies, combined with the user's historical data, geographical location information, time information, etc., to infer scene parameters of associated images.

[0070] In some embodiments, the scene parameters of the associated image may be parameters of the associated image shooting environment, such as parameters of the associated image shooting light, such as light intensity, light direction, color temperature, white balance parameters, dynamic range, etc.; weather conditions of the shooting environment, such as cloudy, sunny, rainy, etc.; the time of image shooting, such as daytime, nighttime, etc.; and the number of frames of the image, wherein the number of frames of the image is associated with the brightness of the image shooting environment, such as the number of frames of the image in a dark environment can be 8 frames or 16 frames.

[0071] In other embodiments, the scene parameters of the associated image can be parameters of the associated image shooting mode, such as aperture (e.g., a larger aperture corresponds to portrait mode, which can produce a shallow depth of field effect, blurring the background and highlighting the subject); shutter speed (e.g., a larger shutter speed corresponds to fast motion photography mode to freeze the moment of motion); focal length (e.g., the focal length corresponding to landscape mode is smaller than that corresponding to portrait mode); focus mode (e.g., a still shooting mode corresponds to single focus, and a dynamic shooting mode corresponds to continuous focus); and the number of frames of the image (e.g., when the shooting mode is high dynamic range mode, the number of frames of the image can be 7 frames of long exposure and 1 frame of short exposure).

[0072] In other embodiments, the scene parameters of the associated image may be user-indicated parameters of the style of the processed image, such as the sharpness, clarity, contrast, saturation, and graininess of the processed image, or the hue and filter effects of the processed image.

[0073] In other embodiments, the scene parameters of the associated image may also be parameters of the associated image composition, parameters of the associated image subject, parameters of the associated image shooting technology and / or shooting device, parameters of the associated image content, etc.

[0074] In step S12, the electronic device processes the image based on a preset image processing model and scene parameters to obtain the processed image. Image processing can include image signal processing, such as Lens Shading Correction (LSC) to correct uneven brightness at image edges caused by lens characteristics and improve overall image uniformity; Black Level Correction (BLC) to correct black level deviations in the image sensor, ensuring accurate representation of black areas in low-light conditions; Auto White Balance (AWB) to automatically adjust white balance based on ambient light, improving color reproduction; Color Correction Matrix (CCM) to perform color correction through matrix operations, compensating for color casts introduced by the sensor and lens; Gamma Correction to adjust grayscale levels for brightness and contrast; High Dynamic Range (HDR) processing; Tone Mapping; Noise Reduction; and Style Transfer.

[0075] In this embodiment of the disclosure, the image processing model is obtained by training a preset neural network model based on first sample images from various different scenarios. The preset neural network model can be a generative intelligence-generated content (AIGC) model, a convolutional neural network (CNN) model, or a generative adversarial network (GAN) model, etc., and this embodiment of the disclosure does not impose any limitations on it.

[0076] In this embodiment of the disclosure, the electronic device acquires a first sample image, wherein the first sample images under various different scenarios include at least one of the following:

[0077] First sample images under various shooting modes;

[0078] First sample images captured by various different shooting devices;

[0079] First sample images under various shooting environments;

[0080] First sample images of various different image styles.

[0081] In this embodiment of the disclosure, the electronic device can acquire a first training sample set, which includes first sample images under various shooting modes, scene parameters associated with each first sample image, and a target sample image corresponding to each first sample image. The shooting modes may include portrait mode, night scene mode, high dynamic range mode, etc. The first sample image may be a historical image or an image after processing the target sample image, such as an image after adding noise to the target sample image. This embodiment of the disclosure does not limit this.

[0082] In this embodiment of the present disclosure, the electronic device processes each first sample image based on a preset neural network model and scene parameters associated with the first sample image to obtain a processed first sample image. Based on the difference between the processed first sample image and the target sample image corresponding to the first sample image, the parameters of the neural network model are adjusted to obtain an image processing model. The processing procedure can be as described above. The electronic device can determine the difference between the processed first sample image and the target sample image corresponding to the first sample image using methods such as Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM), and use this difference as a loss value to adjust the parameters of the neural network model. For example, an optimization algorithm can be used to update the parameters of the neural network model. The optimization algorithm can be gradient descent, Adaptive Moment Estimation (Adam), etc.

[0083] It should be noted that the electronic device can adjust the parameters of the neural network model multiple times until the convergence condition of the neural network model is met. The convergence condition can be that the difference between each processed first sample image and the target sample image corresponding to the first sample image is less than a preset difference threshold. The convergence condition can also be that the number of times the parameters of the neural network model are adjusted reaches a preset number threshold. The preset difference threshold and the preset number threshold are both set values, and this disclosure embodiment does not limit them.

[0084] In this embodiment of the disclosure, the first training sample set acquired by the electronic device may further include first sample images from multiple different shooting devices, scene parameters associated with each first sample image, and target sample images corresponding to each sample image. The shooting devices may be shooting devices of different models or brands. The electronic device performs the above-mentioned training process based on the first sample images from multiple different shooting devices to obtain an image processing model.

[0085] In this embodiment of the disclosure, the first training sample set acquired by the electronic device may further include first sample images under various shooting environments, scene parameters associated with each first sample image, and target sample images corresponding to each sample image. The shooting environment may be a strong light environment, a weak light environment, etc. The electronic device performs the above-mentioned training process based on the first sample images under various shooting environments to obtain an image processing model.

[0086] In this embodiment of the disclosure, the first training sample set acquired by the electronic device may further include sample images, scene parameters associated with each sample image, and first sample images of different image styles corresponding to each sample image. It should be noted that the first sample images here are processed images, and the processed images can be images of different image styles, such as the brightness style of the image, the tonal style of the image, etc. The electronic device performs the above training process based on first sample images of multiple different image styles to obtain an image processing model.

[0087] In this embodiment of the disclosure, during or after the electronic device trains a preset neural network model based on first sample images from various different scenarios to obtain an image processing model, a preset model compression technique can be used to compress the image processing model to reduce its size. This model compression technique can include weight pruning, low-rank decomposition, knowledge distillation, quantization sensing, etc. When processing images using the image processing model, the electronic device can employ a preset hardware acceleration method, such as a heterogeneous computing method using a Neural Processing Unit (NPU) and a Graphics Processing Unit (GPU).

[0088] In this embodiment of the disclosure, the electronic device acquires the image to be processed and the scene parameters of the associated image, and processes the image based on a preset image processing model and the scene parameters to obtain the processed image. On the one hand, when processing the image to be processed, by combining the scene parameters of the associated image, the model can more accurately understand the image content and its contextual information, thereby generating an image that conforms to the scene parameters, which can achieve precise image processing and improve the visual effect of the processed image. On the other hand, since the image processing model is trained based on sample images in various different scenes, the model can adjust the processing strategy according to the scene of the image when facing the image to be processed, thereby improving the quality of the processed image, and has high flexibility and intelligence.

[0089] In some embodiments, the scene parameters include a first parameter associated with the shooting mode of the image and a second parameter of the shooting environment;

[0090] The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including:

[0091] Based on the shooting mode of the image, a first model is determined; wherein, the first model is obtained by training the image processing model using a second sample image associated with the shooting mode of the image.

[0092] Based on the first model and the second parameters, the image is processed to obtain a processed image.

[0093] In this embodiment, the scene parameters include a first parameter of the shooting mode associated with the image and a second parameter of the shooting environment. The first parameter can be: aperture (e.g., a larger aperture for portrait mode, which produces a shallow depth of field, blurring the background and highlighting the subject); shutter speed (e.g., a larger shutter speed for fast motion photography mode to freeze motion); focal length (e.g., a shorter focal length for landscape mode than portrait mode); focus mode (e.g., single-shot focus for static mode, continuous focus for dynamic mode); and the number of frames in the image (e.g., 7 frames for long exposure and 1 frame for short exposure when the shooting mode is high dynamic range). The second parameter can be parameters related to the light used in the image capture (e.g., light intensity, light direction, color temperature, white balance parameters, dynamic range); weather conditions of the shooting environment (e.g., cloudy, sunny, rainy); the time of image capture (e.g., daytime, nighttime); and the number of frames in the image, which is related to the brightness of the shooting environment (e.g., 8 or 16 frames in a dark environment).

[0094] In this embodiment, the electronic device determines a first model based on the image's shooting mode. The first model is trained on an image processing model using second sample images associated with the shooting mode of the associated images. After obtaining the image processing model based on the above training method, the image processing model is trained using a second training sample set to obtain the first model. The second training sample set includes second sample images, scene parameters associated with each second sample image, and a target sample image corresponding to each second sample image. The second sample images are associated with the shooting mode of the image to be processed. For example, the shooting mode of the second sample image can be the same as the shooting mode of the image to be processed. The second sample image can be a portion of the first sample images or an image different from the first sample images. This embodiment does not limit the scope of the invention. In this embodiment, the process of training the image processing model to obtain the first model can be the same as the process of training the neural network model to obtain the image processing model described above.

[0095] In this embodiment of the disclosure, the electronic device processes the image based on the first model and the second parameters to obtain the processed image. The image processing process can be as described above.

[0096] In this embodiment, a first model is first determined based on the image's shooting mode, and then the image is processed based on the first model and a second parameter to obtain the processed image. On the one hand, the first model is further trained on the basis of the image processing model, eliminating the need to rebuild the first model, resulting in lower training costs and higher scalability. On the other hand, processing the image based on the first model and the second parameter corresponding to the image's shooting mode, while also considering the image's shooting mode and shooting environment, enables targeted processing of the image based on the shooting mode and shooting environment, thereby improving the quality of the processed image.

[0097] In some embodiments, the image shooting mode includes one, such as portrait mode or night scene mode, and a first model is determined based on the image shooting mode. Based on the first model and the second parameters of the image, the image is processed to obtain a processed image.

[0098] In other embodiments, the image capture modes include multiple modes, and the first model includes multiple modes, with each first model associated with a capture mode.

[0099] The step of processing the image based on the first model and the second parameters to obtain the processed image includes:

[0100] Based on each first model and the second parameter, the image is processed to obtain the image after processing each first model;

[0101] The images processed by each of the first models are fused to obtain the fused image.

[0102] In this embodiment of the disclosure, the image shooting mode includes multiple types, and the first model includes multiple types, with each first model associated with a shooting mode; for example, the image shooting mode may be a night scene portrait mode. In this case, the image shooting mode includes a night scene mode and a portrait mode. The night scene mode corresponds to one first model, the portrait mode corresponds to one first model, and the image corresponds to two first models. The first model corresponding to the night scene mode is trained on the image processing model using a second sample image associated with the night scene mode, and the first model corresponding to the portrait mode is trained on the image processing model using a second sample image associated with the portrait mode.

[0103] In this embodiment of the disclosure, the electronic device processes the image based on each first model and the second parameter to obtain the image processed by each first model. The image processing process can be as described above.

[0104] In this embodiment of the disclosure, after the electronic device obtains the image processed by each first model, it fuses the images processed by each first model to obtain a fused image. In some embodiments, a pixel averaging fusion method can be used, such as for each pixel in the fused image, obtaining the pixel at the same position as the pixel in each image processed by the first model, and taking the average of the pixel values ​​of the pixels at the same position as the pixel in the fused image as the pixel value of the corresponding pixel; in other embodiments, pixel weighted averaging, pyramid fusion, feature-level fusion, Poisson fusion, and other methods can also be used.

[0105] In this embodiment of the disclosure, when there are multiple image shooting modes, the electronic device determines a first model for each shooting mode, and processes the image based on each first model and second parameters to obtain an image processed by each first model. Then, the images processed by each first model are fused to obtain a fused image. On the one hand, training each first model individually is relatively low-cost because each model is optimized for only one shooting mode. In contrast, training a model that includes multiple shooting modes simultaneously may require more data and computing resources, and the optimization process is more complex. On the other hand, by fusing images processed by multiple first models, the advantages of each model can be utilized to obtain a more comprehensive and high-quality fused image, demonstrating higher intelligence.

[0106] In some embodiments, the method further includes:

[0107] Determine the imaging device used to capture the image;

[0108] The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including:

[0109] Based on the image capturing device, a second model is determined; wherein the second model is obtained by training the image processing model using a third sample image associated with the image capturing device.

[0110] Based on the second model and the scene parameters, the image is processed to obtain a processed image.

[0111] In this embodiment of the disclosure, the electronic device further determines the capturing device for the captured image, such as determining the model and / or brand of the capturing device. After determining the capturing device for the captured image, the electronic device determines a second model based on the capturing device. The second model is trained on the image processing model using third sample images associated with the capturing device. After obtaining the image processing model based on the above training method, the image processing model is trained using a third training sample set to obtain the second model. The third training sample set includes third sample images, scene parameters associated with each third sample image, and a target sample image corresponding to each third sample image. The third sample images are associated with the capturing device of the image to be processed. For example, the capturing device of the third sample image can be the capturing device of the image to be processed. The third sample image can be a portion of the first sample images or an image different from the first sample images. This embodiment of the disclosure does not limit this. In this embodiment of the disclosure, the process of training the image processing model to obtain the second model can be the same as the process of training the neural network model to obtain the image processing model described above.

[0112] In this embodiment of the disclosure, the electronic device processes an image based on a second model and scene parameters to obtain a processed image. The image processing process can be as described above.

[0113] In this embodiment, a second model is determined based on the image capturing device. Then, based on the second model and scene parameters, the image is processed to obtain a processed image. On the one hand, the second model is further trained on the basis of the image processing model, eliminating the need to rebuild the second model, resulting in lower training costs and higher scalability. On the other hand, since images captured by different devices may differ in color, resolution, noise, etc., using a second model trained for a specific device can better adapt to these differences, thereby achieving more personalized image processing and improving the quality of the processed image.

[0114] In some embodiments, the scene parameters further include a third parameter indicating the style of the processed image;

[0115] The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including:

[0116] Based on a preset image processing model and the third parameter, the image is processed to obtain an image that conforms to the image style indicated by the third parameter.

[0117] In this embodiment of the disclosure, the scene parameters also include a third parameter indicating the style of the processed image. This third parameter can be the sharpness, clarity, contrast, saturation, graininess, etc., of the processed image, or it can be the hue, filter effects, etc., of the processed image. For example, the third parameter could be "old photo style," "highlighting the person with a blurred background," etc. Users can upload the third parameter indicating the style of the processed image when taking a picture of the image to be processed or sending the image to an electronic device for processing. The electronic device may include a touch area, and users can use the touch area to indicate the third parameter indicating the style of the processed image. Users can also indicate the third parameter indicating the style of the processed image through voice, gestures, etc. This embodiment of the disclosure does not limit this.

[0118] In this embodiment of the disclosure, during the process of training the neural network model using the first sample image to obtain the image processing model, the description of the third parameter can be continuously changed based on the prompt learning method, and different sample training sets can be set accordingly. For example, when the third parameter is "old photo style", the target sample image corresponding to the first sample image in the aforementioned first training dataset is old photo style.

[0119] In this embodiment of the disclosure, the electronic device processes an image based on a preset image processing model and a third parameter to obtain an image that conforms to the image style indicated by the third parameter. The image processing process can be as described above.

[0120] In this embodiment of the disclosure, the scene parameters also include a third parameter indicating the style of the processed image. The electronic device processes the image based on a preset image processing model and the third parameter to obtain an image that conforms to the image style indicated by the third parameter. This enables customized processing of the image style, meets the personalized needs of different users for image style, and has a high degree of intelligence.

[0121] In some embodiments, the image is a RAW image; the processing of the image based on a preset image processing model and the scene parameters to obtain a processed image includes:

[0122] Based on a preset image processing model and the scene parameters, the RAW image is subjected to image signal processing to obtain the processed image.

[0123] In this embodiment of the disclosure, the image to be processed acquired by the electronic device is a RAW image. The RAW image is the raw data captured by the image sensor, which records the raw data and information of the image sensor converting light signals into digital signals.

[0124] In this embodiment of the disclosure, the electronic device performs image signal processing on the RAW image based on a preset image processing model and scene parameters to obtain the processed image. Here, image signal processing refers to all image processing processes in the traditional ISP processing process, such as the processed image being a displayable RGB image.

[0125] In this embodiment of the disclosure, the electronic device performs image signal processing on RAW images based on an image processing model and scene parameters. Since the image processing model is trained on first sample images of various different scenes, and the image processing model also processes the image in combination with the scene parameters of the image, it can improve image quality, increase processing flexibility, and has higher intelligence compared to traditional ISPs.

[0126] Figure 2 This is an example diagram illustrating the development process of an image processing method according to an exemplary embodiment. It should be noted that, in... Figure 2 In this paper, image signal processing methods are used as an example to illustrate the development process of image processing methods. Figure 2 The process from top to bottom, such as Figure 2As shown, L21 is the initial image processing method, which is the traditional ISP processing. The RAW image L211 is the image to be processed. The RAW image L211 is processed sequentially as follows: lens shading correction L212, noise reduction L213, mosaic removal L214, automatic white balance L215, high dynamic range tone mapping L216, color correction L217, and gamma correction L218, resulting in the processed RGB image L219. From L21 to L22, in image processing method L22, some image processing modules are replaced with neural network models. The RAW image L221 is the image to be processed. The following processes are sequentially applied to the RAW image L221: lens shading correction (L222), denoising (L223), de-mosaicing (L224), automatic white balance (L225), high dynamic range tone mapping (L226), color correction (L227), and gamma correction (L228), resulting in the processed RGB image L229. Notably, denoising (L223), de-mosaicing (L224), and high dynamic range tone mapping (L226) all utilize neural network models. From L22 to L23, all image processing steps utilize neural network models. The RAW image L231 is the image to be processed. The RAW image L231 is input into the neural network model L232 to obtain the processed RGB image L233. From L23 to the image processing method L24 of this embodiment, the RAW image L241 is the image to be processed. The RAW image L241 and the scene parameters associated with the RAW image L241 are input into the large model-image signal processing system L242 to obtain the processed RGB image L243. It should be noted that the number of training samples of the neural network L232 model and the number of scene types associated with the training samples are less than those of the large model-image signal processing system L242. The training samples of the large model-image signal processing system L242 may include sample images of various scenes.

[0127] Figure 3 This is an architectural diagram illustrating an image processing method according to an exemplary embodiment. It should be noted that, in... Figure 3 In this example, image signal processing is used as an example to illustrate the image processing method. RAW image L33 is the image to be processed, large model L37 is the image processing model, RGB image L39 is the processed image, user preference information L31 and original image information L32 are both scene parameters associated with RAW image L33. User preference information L31 is a third parameter indicating the style of the processed image, and original image information L32 includes a first parameter of the shooting mode associated with RAW image L33 and a second parameter of the shooting environment. Figure 3As shown, a RAW image L33 is acquired and encoded using an image encoding module L35, converting the image from pixel space to a high-dimensional feature space for processing by the large model L37. User preference information L31 and original image information L32 are acquired and encoded using an information encoding module L34, converting semantic information into a high-dimensional vector representation that the large model L37 can understand. The encoded RAW image L33, along with the user preference information L31 and original image information L32, are then processed. The input module L32 is then injected into the injection module L36. The injection module L36 then inputs the encoded RAW image L33, along with user preference information L31 and original image information L32, into the large model L37. The large model L37, combining the encoded user preference information L31 and original image information L32, processes the RAW image L33 and outputs the processing result to the image decoding module L38. The image decoding module L38 decodes the output of the large model, converting the image from the feature domain to a device-recognizable RGB domain image, resulting in the RGB image L39. It should be noted that this embodiment can also process RGB images to further improve their quality. This embodiment can also process RAW images to improve their quality, and then use traditional ISP technology to convert the processed RAW image into an RGB image. Both the image encoding module L35 and the image decoding module L38 can be open-source models. For example, the image encoding module L35 can be a variational autoencoder (VAE-Encoder), and the image decoding module L38 can be a variational autodecoder (VAE-Decoder). The information encoding module L34 can also be an open-source model, such as a contrastive language-image pre-training model. The injection module L36 can use algorithms such as ControlNet, Low-Rank Adaptation (LoRA), Cross-Attention, and Spatially Adaptive Normalization.

[0128] Figure 4 This is a block diagram of an image processing apparatus 400 according to an exemplary embodiment. (See diagram below.) Figure 4 As shown, the device mainly includes:

[0129] The acquisition module 401 is configured to acquire the image to be processed and the scene parameters associated with the image;

[0130] The processing module 402 is configured to process the image based on a preset image processing model and the scene parameters to obtain a processed image; wherein the image processing model is obtained by training a preset neural network model based on first sample images under multiple different scenes.

[0131] In some embodiments, the scene parameters include a first parameter associated with the shooting mode of the image and a second parameter of the shooting environment; the processing module 402 is further configured to determine a first model based on the shooting mode of the image; wherein the first model is trained on the image processing model using a second sample image associated with the shooting mode of the image; and the image is processed based on the first model and the second parameter to obtain a processed image.

[0132] In some embodiments, the image shooting mode includes multiple modes, the first model includes multiple modes, and each first model is associated with a shooting mode; the processing module 402 is further configured to process the image based on each first model and the second parameter to obtain an image processed by each first model; and to fuse the images processed by each first model to obtain a fused image.

[0133] In some embodiments, the apparatus further includes:

[0134] The determination module is configured to determine the capturing device for capturing the image;

[0135] The processing module 402 is further configured to determine a second model based on the image capturing device; wherein the second model is trained on the image processing model using a third sample image associated with the image capturing device; and the image is processed based on the second model and the scene parameters to obtain a processed image.

[0136] In some embodiments, the scene parameters further include a third parameter indicating the style of the processed image; the processing module 402 is further configured to process the image based on a preset image processing model and the third parameter to obtain an image that conforms to the image style indicated by the third parameter.

[0137] In some embodiments, the image is a RAW image; the processing module 402 is further configured to perform image signal processing on the RAW image based on a preset image processing model and the scene parameters to obtain the processed image.

[0138] In some embodiments, the first sample images under various different scenarios include at least one of the following:

[0139] First sample images under various shooting modes;

[0140] First sample images captured by various different shooting devices;

[0141] First sample images under various shooting environments;

[0142] First sample images of various different image styles.

[0143] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0144] Figure 5 This is a structural block diagram of an electronic device 500 according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, computer, game console, tablet device, medical device, personal digital assistant, or other terminal device, and may also be a cloud device such as a server. This disclosure does not limit the scope of the embodiments.

[0145] Reference Figure 5 The electronic device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0146] Processing component 502 typically controls the overall operation of electronic device 500, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0147] Memory 504 is configured to store various types of data to support operation on electronic device 500. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 500, contact data, phonebook data, messages, pictures, and videos. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0148] Power supply component 506 provides power to various components of electronic device 500. Power supply component 506 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 500.

[0149] Multimedia component 508 includes a screen that provides an output interface between electronic device 500 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When electronic device 500 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0150] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when electronic device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0151] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0152] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of electronic device 500. For example, sensor assembly 514 may detect the on / off state of electronic device 500, the relative positioning of components such as the display and keypad of electronic device 500, changes in position of electronic device 500 or one of its components, the presence or absence of user contact with electronic device 500, orientation or acceleration / deceleration of electronic device 500, and temperature changes of electronic device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0153] Communication component 516 is configured to facilitate wired or wireless communication between electronic device 500 and other devices. Electronic device 500 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0154] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0155] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including executable instructions or a computer program, which can be executed by a processor 520 of an electronic device 500 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0156] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the image processing methods described above in the embodiments of this disclosure. For example, the method includes:

[0157] Obtain the image to be processed and the scene parameters associated with the image;

[0158] Based on a preset image processing model and the scene parameters, the image is processed to obtain a processed image; wherein, the image processing model is obtained by training a preset neural network model based on first sample images under multiple different scenes.

[0159] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image processing methods described above in this disclosure.

[0160] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0161] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image to be processed and the scene parameters associated with the image; Based on a preset image processing model and the scene parameters, the image is processed to obtain a processed image; wherein, the image processing model is obtained by training a preset neural network model based on first sample images under multiple different scenes.

2. The method according to claim 1, characterized in that, The scene parameters include a first parameter associated with the shooting mode of the image and a second parameter related to the shooting environment; The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including: Based on the shooting mode of the image, a first model is determined; wherein, the first model is obtained by training the image processing model using a second sample image associated with the shooting mode of the image. Based on the first model and the second parameters, the image is processed to obtain a processed image.

3. The method according to claim 2, characterized in that, The image capture modes include multiple types, and the first model includes multiple types, with each first model associated with a capture mode. The step of processing the image based on the first model and the second parameters to obtain the processed image includes: Based on each first model and the second parameter, the image is processed to obtain the image after processing each first model; The images processed by each of the first models are fused to obtain the fused image.

4. The method according to claim 1, characterized in that, The method further includes: Determine the imaging device used to capture the image; The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including: Based on the image capturing device, a second model is determined; wherein the second model is obtained by training the image processing model using a third sample image associated with the image capturing device. Based on the second model and the scene parameters, the image is processed to obtain a processed image.

5. The method according to any one of claims 1 to 4, characterized in that, The scene parameters also include a third parameter that indicates the style of the processed image; The image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including: Based on a preset image processing model and the third parameter, the image is processed to obtain an image that conforms to the image style indicated by the third parameter.

6. The method according to any one of claims 1 to 4, characterized in that, The image is a RAW image; the image is processed based on a preset image processing model and the scene parameters to obtain a processed image, including: Based on a preset image processing model and the scene parameters, the RAW image is subjected to image signal processing to obtain the processed image.

7. The method according to any one of claims 1 to 4, characterized in that, The first sample images under various different scenarios include at least one of the following: First sample images under various shooting modes; First sample images captured by various different shooting devices; First sample images under various shooting environments; First sample images of various different image styles.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is configured to acquire the image to be processed and the scene parameters associated with the image; The processing module is configured to process the image based on a preset image processing model and the scene parameters to obtain a processed image; wherein the image processing model is obtained by training a preset neural network model based on first sample images under multiple different scenes.

9. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the image processing method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the image processing method according to any one of claims 1 to 7.

Images