Image processing method and device and electronic equipment

Abstract

The invention discloses an image processing method and device and electronic equipment, and belongs to the technical field of image processing.The method comprises the steps that according to a first color image, a scene label of the first color image and an original image, a dynamic color lookup table matched with a shooting scene of the first color image is determined; wherein the scene label is used for representing a semantic scene category of the first color image; and performing color conversion processing on the first color image based on the dynamic color lookup table to generate a second color image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image processing method and device and electronic equipment. BACKGROUND

[0002] Color conversion refers to a process of converting colors in an image, a video or a graph from one color space or color model to another, and a core purpose thereof is to adapt to different devices, application scenarios or visual requirements and ensure that colors are accurately and consistently presented between different systems.

[0003] In related technologies, static color look-up tables (LUTs) based on fixed interpolation algorithms are mainly relied on for color conversion processing. However, this color conversion processing manner is prone to cause color discontinuity and unnatural transition when illumination conditions such as color temperature and illumination change, and may also cause picture flicker or color abnormality in extreme scene switching such as day-night transition, resulting in inaccurate color restoration. SUMMARY

[0004] Embodiments of the present application provide an image processing method, device and electronic equipment, which can improve color restoration accuracy of color images in a color conversion link.

[0005] In a first aspect, an image processing method is provided, and the method comprises:

[0006] determining a dynamic color look-up table matched with a shooting scene of the first color image according to the first color image, a scene label of the first color image and an original image; wherein the scene label is used to represent a semantic scene category of the first color image;

[0007] performing color conversion processing on the first color image based on the dynamic color look-up table to generate a second color image.

[0008] Optionally, in some embodiments of the first aspect, before the dynamic color look-up table matched with the shooting scene of the first color image is determined according to the first color image, the scene label of the first color image and the original image, the method further comprises:

[0009] inputting the first color image into a scene detection network, performing semantic scene recognition processing on the first color image through the scene detection network, and outputting the scene label of the first color image.

[0010] Optionally, in some embodiments of the first aspect, before determining a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes:

[0011] Based on the first dataset, the network parameters of the first backbone network are adjusted to obtain the converged first backbone network, and the converged first backbone network is determined as the scene detection network.

[0012] The scene detection network is used to perform semantic scene recognition on the first color image to obtain scene labels for the first color image;

[0013] The first backbone network is an image classification network, and the first dataset includes multiple sets of first training data. Each set of first training data includes a sample color image and a sample scene label corresponding to the sample color image.

[0014] Optionally, in some embodiments of the first aspect, before determining a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes:

[0015] When the image sensor has the function of outputting raw images, the raw image of the first color image acquired by the image sensor is obtained;

[0016] When the image sensor does not have the function of outputting the original image, the first color image is input into the backward network, and the image reconstruction processing of the first color image is performed by the backward network to output the original image of the first color image.

[0017] Optionally, in some embodiments of the first aspect, before determining a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes:

[0018] Based on the second dataset, the network parameters of the second backbone network are adjusted to obtain the converged second backbone network, which is then identified as the backward network.

[0019] The feedforward network is used to reconstruct the first color image to obtain the original image of the first color image;

[0020] The second backbone network is a convolutional neural network, and the second dataset includes multiple sets of second training data. Each set of second training data includes a sample color image and a sample original image of the sample color image.

[0021] Optionally, in some embodiments of the first aspect, determining a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image includes:

[0022] The first color image, its scene label, and the original image are input into a dynamic color lookup table generation network. The network then performs multimodal feature fusion processing on the first color image, its scene label, and the original image using the dynamic color lookup table, and outputs a dynamic color lookup table that matches the shooting scene of the first color image.

[0023] Optionally, in some embodiments of the first aspect, before determining a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes:

[0024] Based on the third dataset, the network parameters of the third backbone network are adjusted to obtain the converged third backbone network. The converged third backbone network is then identified as the dynamic color lookup table generation network.

[0025] The dynamic color lookup table generation network is used to generate a dynamic color lookup table that matches the shooting scene of the first color image.

[0026] The third backbone network is a convolutional neural network. The third dataset includes multiple sets of third training data. Each set of third training data includes a sample color image, a sample scene label corresponding to the sample color image, a sample original image of the sample color image, and a sample color lookup table that matches the shooting scene of the sample color image. The sample color lookup table is used to perform color conversion on the sample color image to obtain a reference color image.

[0027] Optionally, in some embodiments of the first aspect, before adjusting the network parameters of the third backbone network according to the third dataset to obtain the converged third backbone network, and determining the converged third backbone network as the dynamic color lookup table generation network, the method further includes:

[0028] Based on the fourth dataset, the network parameters of the fourth backbone network are adjusted to obtain the converged fourth backbone network, which is then determined as the feedforward network. The fourth dataset includes multiple sets of fourth training data, each set of fourth training data including a sample color image, a sample scene label corresponding to the sample color image, and a reference color image corresponding to the sample color image.

[0029] Based on the network structure and parameters of the feedforward network, a third backbone network is constructed.

[0030] Optionally, in some embodiments of the first aspect, the first color image includes at least one of the following: a high dynamic range image and a standard dynamic range image.

[0031] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0032] The processing module is configured to determine a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image; wherein the scene label is used to characterize the semantic scene category of the first color image; and based on the dynamic color lookup table, perform color conversion processing on the first color image to generate a second color image.

[0033] Optionally, in some embodiments of the second aspect, the processing module is further configured to acquire the original image of the first color image acquired by the image sensor when the image sensor has the original image output function; and input the first color image into the backward network when the image sensor does not have the original image output function, perform image reconstruction processing on the first color image through the backward network, and output the original image of the first color image.

[0034] Optionally, in some embodiments of the second aspect, the processing module is further configured to adjust the network parameters of the second backbone network according to the second dataset to obtain a converged second backbone network, and determine the converged second backbone network as a feedforward network; wherein the feedforward network is used to reconstruct the first color image to obtain the original image of the first color image; the second backbone network is a convolutional neural network, and the second dataset includes multiple sets of second training data, each set of second training data including a sample color image and a sample original image of the sample color image.

[0035] Optionally, in some embodiments of the second aspect, the processing module is further configured to adjust the network parameters of the third backbone network according to the third dataset to obtain a converged third backbone network, and determine the converged third backbone network as a dynamic color lookup table generation network; wherein, the dynamic color lookup table generation network is used to generate a dynamic color lookup table matching the shooting scene of the first color image; the third backbone network is a convolutional neural network, the third dataset includes multiple sets of third training data, each set of third training data includes a sample color image, a sample scene label corresponding to the sample color image, a sample original image of the sample color image, and a sample color lookup table matching the shooting scene of the sample color image, the sample color lookup table is used to perform color conversion on the sample color image to obtain a reference color image.

[0036] Optionally, in some embodiments of the second aspect, the processing module is further configured to adjust the network parameters of the fourth backbone network according to the fourth dataset to obtain a converged fourth backbone network, and determine the converged fourth backbone network as a feedforward network; wherein, the fourth dataset includes multiple sets of fourth training data, each set of fourth training data includes a sample color image, a sample scene label corresponding to the sample color image, and a reference color image corresponding to the sample color image; and a third backbone network is constructed based on the network structure and network parameters of the feedforward network.

[0037] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the image processing method as described in the first aspect.

[0038] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in the first aspect.

[0039] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the image processing method as described in the first aspect.

[0040] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the image processing method as described in the first aspect.

[0041] In this embodiment of the application, a dynamic color lookup table matching the shooting scene of the first color image is determined based on the first color image, the scene label of the first color image, and the original image; wherein, the scene label is used to characterize the semantic scene category of the first color image; based on the dynamic color lookup table, the first color image is subjected to color conversion processing to generate a second color image.

[0042] As can be seen, in this embodiment, the first color image provides rich spatial texture features, the scene label of the first color image provides semantic information about the shooting scene of the first color image, and the original image of the first color image provides the original light intensity data when the first color image was taken. Because the texture features, semantic scene, and lighting conditions of the image are comprehensively considered, a dynamic color lookup table matching the shooting scene of the first color image can be generated. This dynamic color lookup table differs from traditional static color lookup tables; it can automatically adjust according to different shooting scenes, lighting conditions, and image features, more accurately simulating the color distribution and lighting effects of the actual scene, and has stronger adaptability and flexibility. Using the generated dynamic color lookup table to perform color conversion processing on the first color image, the color-converted second color image can more accurately restore the colors of the original scene, reduce color deviation, and improve the accuracy of color restoration in the color conversion process. Attached Figure Description

[0043] Figure 1 This is a flowchart of an image processing method provided in some embodiments of this application;

[0044] Figure 2 This is a network structure diagram of the first backbone network provided in some embodiments of this application;

[0045] Figure 3 This is a network structure diagram of the second backbone network provided in some embodiments of this application;

[0046] Figure 4 This is a network structure diagram of a third backbone network provided in some embodiments of this application;

[0047] Figure 5 This is a network structure diagram of the fourth backbone network provided in some embodiments of this application;

[0048] Figure 6 These are example diagrams illustrating image processing methods provided in some embodiments of this application;

[0049] Figure 7 This is a structural block diagram of an image processing apparatus provided in some embodiments of this application;

[0050] Figure 8 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application;

[0051] Figure 9 This is a schematic diagram of the hardware structure of an electronic device that implements some embodiments of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0053] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0054] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0055] Dynamic range refers to the ratio of the brightness of the brightest and darkest parts of an image.

[0056] High Dynamic Range (HDR) is a technology used to represent a wider dynamic range of exposure than ordinary digital imaging techniques. It can simultaneously capture details in both extremely bright and extremely dark areas of a scene, making images closer to the real world as seen by the human eye. For example, in photography, HDR images are created by taking multiple images at different exposures and then combining them into a single image with rich detail, clearly rendering both bright skies and dark shadows. In film and television production, HDR technology enhances video quality, allowing viewers to see more realistic and vivid images, thus enhancing the viewing experience. In the display device field, HDR-enabled TVs and monitors can present a wider color gamut and higher contrast, making images more vibrant and three-dimensional.

[0057] Standard Dynamic Range (SDR) is a traditional technical standard in the video and image industry used to describe the brightness range and color performance capabilities of content. SDR typically ranges from 0.1 nits to 100 nits in brightness, with some standards extending to 300 nits. Compared to HDR, SDR has lower upper and lower limits in brightness, making it unable to render details in extremely bright areas such as sunlight and flames, or extremely dark areas such as shadows and starry skies.

[0058] Auto White Balance (AWB) is a technique used in image processing to correct the colors of an image, ensuring it appears as a natural white under different lighting conditions. Different light sources, such as sunlight, incandescent lamps, and fluorescent lamps, have different color temperatures, which affect the color of objects in photos or videos. The purpose of white balance is to eliminate the influence of the light source's color temperature on image colors, allowing white objects to appear true white under different lighting conditions, thereby ensuring the accuracy of other colors.

[0059] A Color Correction Matrix (CCM) is a mathematical tool used for precise color correction in images. It is a 3×3 matrix that adjusts the hue, saturation, and brightness of colors by linearly transforming the RGB color values ​​of each pixel in the image, achieving a more accurate and natural color representation.

[0060] Deep learning is a branch of machine learning that is based on the architecture of Artificial Neural Networks (ANNs). It learns representations of data by constructing multi-layered, non-linearly transformed network structures. Compared to traditional machine learning algorithms, deep learning can automatically learn complex features and patterns from large amounts of data without requiring manual feature extraction.

[0061] A color lookup table (LUT) is a table used for color conversion that contains the relationship between input color values ​​and their corresponding output color values. In image processing and video editing, LUTs can quickly and efficiently adjust and convert the colors of images or videos in batches to achieve specific color styles and effects.

[0062] The structure and working principle of a color lookup table: A color lookup table typically has three components: red (R), green (G), and blue (B). The input components are called Rin, Gin, and Bin, and the output components are called Rout, Gout, and Bout. There is a one-to-one correspondence between the input and output. When performing color conversion on an image, for the RGB value of each pixel in the image, the corresponding input value in the LUT is looked up to find the corresponding output value. Then, the color of that pixel is replaced with the output value, thus completing the color conversion of the entire image. For example, by applying different LUTs, an ordinary color image can be converted into a retro, cinematic, or high-contrast style image.

[0063] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios. It should be noted that the image processing method provided in this application is applicable to electronic devices. In practical applications, such electronic devices include, but are not limited to, mobile terminals such as smartphones, tablets, or personal digital assistants. This application does not limit the scope of these devices.

[0064] The image processing method provided in this application can be applied to color conversion scenarios of color images. For example, one specific application scenario is mobile phone camera photography optimization, such as color conversion of HDR images captured by a mobile phone camera to optimize the image output effect of the mobile phone camera; another specific application scenario is social media image processing, such as color conversion of SDR images to optimize the image effect of images shared on social media.

[0065] Figure 1 This is a flowchart of an image processing method provided in some embodiments of this application, such as... Figure 1 As shown, the method includes at least the following steps: step 101 and step 102.

[0066] In step 101, a dynamic color lookup table matching the shooting scene of the first color image is determined based on the first color image, the scene label of the first color image, and the original image; wherein, the scene label is used to characterize the semantic scene category of the first color image.

[0067] In some embodiments of this application, the first color image is a color image to be processed, which contains rich color information and serves as the basic data source for the subsequent generation of the color lookup table. For example, in a post-processing scenario for photography, the first color image may be an initial photograph taken by the photographer, whose color performance may be insufficient due to factors such as shooting equipment and environment, requiring optimization through a dynamic color lookup table.

[0068] In some embodiments of this application, the first color image may include at least one of the following: a high dynamic range image and a standard dynamic range image.

[0069] On the one hand, color images may come from different devices and shooting environments. For example, professional cameras and high-end mobile phones can shoot HDR images, which can record richer brightness and color information; while images taken by ordinary devices or in the early days are mostly SDR images. On the other hand, different application scenarios have different requirements for the dynamic range of images. For example, in film and television production and high-end photography, HDR images can better present scene details under complex lighting, while in some real-time monitoring and ordinary social sharing scenarios, SDR images can already meet basic needs.

[0070] In some embodiments of this application, both high dynamic range (HDR) images and standard dynamic range (SMR) images are supported, enabling effective processing regardless of the source of the input image. On the one hand, no additional conversion steps are required, simplifying the processing flow and improving the versatility of the image processing method; on the other hand, it can be flexibly applied to various scenarios to meet the needs of different users.

[0071] In some embodiments of this application, the format of the first color image is typically RGB or BGR, and the pixel bits of the first color image are typically 8 bits, 10 bits, or 12 bits.

[0072] In some embodiments of this application, the first color image serves as the base image for color conversion, providing rich spatial texture features. These texture features contain information such as the shape, structure, and details of objects in the image, which are crucial for maintaining the visual coherence and detail representation of the image during color conversion. For example, in landscape images, the outlines of mountains and the texture of tree branches and leaves can help the color conversion algorithm better understand the image content, thereby enabling more accurate color adjustments.

[0073] In some embodiments of this application, scene tags are used to characterize the semantic scene category of the first color image. By performing semantic analysis on the first color image, it is categorized into specific scene types, such as indoor, outdoor, beach, forest, backlight, night scene, portrait, greenery, food, etc. Different scenes typically have different color styles and lighting conditions. Scene tags provide high-level semantic guidance for color conversion, helping to generate dynamic color lookup tables that better match the characteristics of the scene. For example, landscape scenes usually require vibrant colors and rich gradations, while portrait scenes emphasize the natural reproduction of skin tones.

[0074] In some embodiments of this application, since scene tags can characterize information such as illuminance and color temperature of the first color image, scene tags are used as one of the reference data for the subsequent generation of dynamic color lookup tables. On the one hand, the color lookup table is dynamically adjusted through semantic tags to solve the fixed deviation problem of traditional color lookup tables. On the other hand, the color lookup table module and the color correction matrix module are globally coordinated and optimized to build an end-to-end color conversion framework, reducing error accumulation.

[0075] In some embodiments of this application, the original image is the raw data acquired by the image sensor, and the format of the original image is usually RAW format. For ease of understanding, "RAW image" may be used to describe the original image below.

[0076] In some embodiments of this application, the original image provides the original light intensity data at the time the first color image was captured. This original light intensity data reflects the actual light intensity distribution of the shooting scene, which is crucial for color reproduction. This is because color conversion must consider not only attributes such as hue and saturation, but also the influence of lighting conditions on color, to ensure that the converted color matches the lighting effect of the actual scene. Obtaining the original image of the first color image provides a high-fidelity color reference for the generation of a dynamic color lookup table.

[0077] In some embodiments of this application, since the original image retains the original light intensity data of the image sensor, the original image is used as one of the reference data for the subsequent generation of the dynamic color lookup table. On the one hand, utilizing the original light intensity information of the original image can improve the accuracy of color reproduction; on the other hand, it realizes the global collaborative optimization of the color lookup table module and the white balance module, constructs an end-to-end color conversion framework, and reduces error accumulation.

[0078] In some embodiments of this application, based on three input information—a first color image, a scene label for the first color image, and the original image—a dynamic color lookup table can be generated by comprehensively analyzing the texture features, semantic scene, and lighting conditions of the image to match the shooting scene of the first color image. On one hand, the dynamic color lookup table differs from the traditional static color lookup table; it can be adjusted in real time according to different scenes and image features, avoiding the fixed deviations of the traditional static color lookup table and possessing greater adaptability and flexibility. On the other hand, it enables color conversion to automatically adjust according to the environment. Under different shooting scenes, lighting conditions, and device parameters, the dynamic color lookup table can adapt to changes in real time, providing a more suitable color conversion scheme, thereby improving the color performance of the color image in different environments.

[0079] In some embodiments of this application, by comprehensively considering the texture features, semantic scenes and lighting conditions of the image, the dynamic color lookup table can more accurately simulate the color distribution and lighting effects of the actual scene. This enables the second color image after color conversion to more accurately reproduce the colors of the original scene, reduce color deviation and improve the accuracy of color reproduction.

[0080] For example, taking the color conversion of an HDR image captured by a mobile phone camera as an example, when a user uses their mobile phone camera to shoot a city night scene, an HDR image is captured during the shooting process. The scene label of the HDR image, such as "night scene lights," is automatically identified. Combined with the corresponding RAW image, a dynamic color lookup table is generated. This dynamic color lookup table can enhance the color saturation and brightness of lights in night scene shooting, while reducing noise in dark areas, making the colors of the HDR image more vivid and clear, thereby achieving accurate color conversion of high dynamic range images. In this example, the HDR image is the first color image, the RAW image is the original image, and "night scene lights" is the scene label.

[0081] For example, consider color conversion of SDR images on social media. Before sharing an SDR image on a social media platform, a user might want to perform color conversion on the image to attract more attention. For instance, when sharing photos of food taken during a trip, the social media application has this image processing method built-in. When a user uploads an SDR image, the application obtains the SDR image and automatically identifies the scene tag, such as "food," while simultaneously obtaining the corresponding RAW image. Based on this information, it generates a dynamic color lookup table. This dynamic color lookup table can be used to highlight the colors of the food photo, thereby achieving accurate color conversion of a standard dynamic range image. In this example, the SDR image is the first color image, the RAW image is the original image, and "food" is the scene tag.

[0082] In step 102, the first color image is subjected to color conversion processing based on a dynamic color lookup table to generate a second color image.

[0083] In some embodiments of this application, a generated dynamic color lookup table is used to perform color conversion processing on a first color image to generate a second color image. During the conversion process, the corresponding conversion rule in the dynamic color lookup table is looked up based on the color value of each pixel in the first color image, mapping the original color to the target color space, thereby achieving color adjustment and optimization.

[0084] For example, taking the color conversion of an HDR image captured by a mobile phone camera as an example, when a user uses a mobile phone camera to shoot a city night scene, the captured HDR image is obtained during the shooting process. The scene label of the HDR image, such as night scene lights, is automatically identified. Combined with the illumination data of the corresponding RAW image, a dynamic color lookup table is generated. Then, based on the dynamic color lookup table, the HDR image is color converted to generate a night scene image with more vivid colors, richer layers, and better visual aesthetics, thus improving the user's photography experience.

[0085] For example, consider color conversion of SDR images on social media. Before sharing an SDR image on a social media platform, a user might want to perform color conversion on the image to attract more attention. For instance, when sharing food photos from a trip, the social media application can incorporate this image processing method. When a user uploads an SDR image, the application obtains the SDR image and automatically identifies the scene tag, such as "food," while simultaneously obtaining the corresponding RAW image. Based on this information, a dynamic color lookup table is generated. Then, based on this dynamic color lookup table, the SDR image is color converted to generate a more attractive food image, improving the sharing effect on the social media platform.

[0086] In some embodiments of this application, the first color image provides rich spatial texture features, the scene label of the first color image provides semantic information about the shooting scene of the first color image, and the original image of the first color image provides the original light intensity data when the first color image was captured. Because the texture features, semantic scene, and lighting conditions of the image are comprehensively considered, a dynamic color lookup table matching the shooting scene of the first color image can be generated. This dynamic color lookup table differs from traditional static color lookup tables; it can automatically adjust according to different shooting scenes, lighting conditions, and image features, more accurately simulating the color distribution and lighting effects of the actual scene, and has stronger adaptability and flexibility. Using the generated dynamic color lookup table to perform color conversion processing on the first color image, the color-converted second color image can more accurately restore the colors of the original scene, reduce color deviation, and improve the accuracy of color restoration in the color conversion process.

[0087] In some embodiments provided in this application, applicable to scenarios where scene labels of a first color image are obtained, the provided image processing method, in Figure 1 Based on the embodiment shown, before step 101 above, the following step may also be included: step 201.

[0088] In step 201, the first color image is input into the scene detection network, and the scene detection network performs semantic scene recognition processing on the first color image to output the scene label of the first color image.

[0089] In some embodiments of this application, the scene detection network is a deep learning-based computer vision model that automatically identifies and classifies the scene type of an image, such as indoor, outdoor, urban, natural, nighttime, or daytime, by analyzing visual features in an image or video, such as color, texture, object layout, and spatial relationships. Its core is to transform pixel-level visual information into high-level scene labels through semantic understanding, thereby providing contextual information for subsequent image processing, analysis, or decision-making.

[0090] For example, when the first color image is an HDR image of a city night scene captured by a mobile phone camera, the HDR image of the city night scene is input into a scene detection network, and the scene detection network performs semantic scene recognition processing on the HDR image of the city night scene, and outputs the scene label "night scene lights".

[0091] For example, when the first color image is a food SDR image from social media, the food SDR image is input into a scene detection network, and the scene detection network performs semantic scene recognition processing on the food SDR image to output the scene label "food".

[0092] In some embodiments of this application, a scene detection network is used to identify scene labels in a first color image. On the one hand, this eliminates the need for manual intervention and enables rapid identification of scenes in the image, reducing labor costs. On the other hand, the scene detection network contains a precise mapping relationship from image visual features to semantic scene labels, enabling it to output highly accurate scene classification results.

[0093] In some embodiments provided in this application, applicable to scenarios where scene labels of a first color image are obtained, the provided image processing method, in Figure 1 Based on the embodiment shown, before step 101 above, the following step may also be included: step 301.

[0094] In step 301, the network parameters of the first backbone network are adjusted according to the first dataset to obtain the converged first backbone network, and the converged first backbone network is determined as the scene detection network.

[0095] In some embodiments of this application, a scene detection network is used to perform semantic scene recognition on a first color image, and the recognition result is a scene label for the first color image.

[0096] In some embodiments of this application, the first backbone network is essentially an image classification network, serving as the basic architecture for constructing a scene detection network.

[0097] In some embodiments of this application, a first backbone network can be constructed based on image classification networks such as ResNeSt50 or MobileViT to balance the accuracy of the model with the efficiency of training.

[0098] In some embodiments of this application, the first dataset includes multiple sets of first training data, each set of first training data consisting of a sample color image and a sample scene label corresponding to the sample color image, providing data support for adjusting the parameters of the first backbone network.

[0099] In some embodiments of this application, the first dataset can be expanded and updated according to actual needs, adding new sample color images and corresponding sample scene labels. The scene detection network trained based on this dataset can continuously learn and adapt to new scene types, which makes the entire image processing method more scalable and adaptable. It can continuously improve scene recognition capabilities and overall processing performance as the data becomes richer and the scene changes, meeting the needs of different application scenarios.

[0100] In some embodiments of this application, the first dataset may include sample color images of more than 100 scene categories and corresponding scene labels, with at least 1,000 images for each scene category. In practical applications, the first dataset may be taken from datasets such as COCO-Stuff and Places365 Extended Set.

[0101] In some embodiments of this application, the first training data in the first dataset covers sample color images from various different scenarios. This diverse data helps the first backbone network learn a wider range of scene features, rather than being limited to features from a specific scene or a small number of samples. Through this training process, the resulting scene detection network can better adapt to and recognize various complex and changing scene images in real-world applications, possessing strong generalization ability and reducing recognition errors caused by scene changes.

[0102] In some embodiments of this application, the cross-entropy loss function can be used as the loss function during the training process of the scene detection network.

[0103] In some embodiments of this application, the cross-entropy loss function clearly reflects the difference between the predicted result and the true label, and provides stable gradient information during training. Therefore, it helps the optimization algorithm find the global minimum or local minimum of the loss function more quickly. Compared to some other loss functions, using the cross-entropy loss function allows the scene detection network to achieve better performance in fewer training iterations, significantly accelerating the network's convergence speed and reducing training time and computational resource consumption.

[0104] In some embodiments of this application, during the training process of the scene detection network, the gradient of each parameter in the network can be calculated using the backpropagation method, and the network parameters can be updated using the stochastic gradient descent method until the network loss function converges. The network parameters are then saved, and the training process of the network ends, ensuring the stability of the gradient and facilitating deep training of the network.

[0105] For example, such as Figure 2 As shown, the first backbone network 20 includes a first network layer 21, a second network layer 22, ..., a twelfth network layer 212 connected from left to right.

[0106] The first network layer 21 may include a convolutional layer 1 and a ReLU layer. The convolutional kernel size of the convolutional layer 1 is 1×1×3, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0107] The second network layer 22 may include a convolutional layer 2 and a ReLU layer. The convolutional kernel size of the convolutional layer 2 is 1×1×10, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0108] The third network layer 23 may include a convolutional layer 3 and a ReLU layer. The convolutional kernel size of the convolutional layer 3 is 1×1×20, the number of convolutional kernels is 8, and the size of the output feature map is 256*256*8.

[0109] The fourth network layer 24 may include a convolutional layer 4 and a ReLU layer. The convolutional kernel size of the convolutional layer 4 is 1×1×20, the number of convolutional kernels is 8, and the size of the output feature map is 256*256*8.

[0110] The fifth network layer 25 may include a convolutional layer 5 and a ReLU layer. The convolutional kernel size of the convolutional layer 5 is 1×1×20, the number of convolutional kernels is 8, and the size of the output feature map is 256*256*8.

[0111] The sixth network layer 26 may include a convolutional layer 6 and a ReLU layer. The convolutional kernel size of the convolutional layer 6 is 1×1×20, the number of convolutional kernels is 8, and the size of the output feature map is 256*256*8.

[0112] The seventh network layer 27 may include a convolutional layer 7 and a ReLU layer. The convolutional kernel size of the convolutional layer 7 is 1×1×20, the number of convolutional kernels is 8, and the size of the output feature map is 256*256*8.

[0113] The eighth network layer 28 may include a convolutional layer 8 and a ReLU layer. The convolutional kernel size of the convolutional layer 8 is 1×1×20, the number of convolutional kernels is 8, and the size of the output feature map is 256*256*8.

[0114] The ninth network layer 29 may include a convolutional layer 9 and a ReLU layer. The convolutional kernel size of the convolutional layer 9 is 1×1×8, the number of convolutional kernels is 3, and the size of the output feature map is 10*10*3.

[0115] The tenth network layer 210 may include a convolutional layer 10 and a ReLU layer. The convolutional kernel size of the convolutional layer 10 is 1×1×8, the number of convolutional kernels is 3, and the size of the output feature map is 10*10*3.

[0116] The ReLU layers in the first network layer 21 to the tenth network layer 210 all use the function of formula (1):

[0117]

[0118] The eleventh network layer 211 may include a convolutional layer 11, the convolutional kernel size of the convolutional layer 11 is 1×1×1, the number of convolutional kernels is 1, and the size of the output feature map is 10*10*1.

[0119] The twelfth network layer 212 may include a fully connected layer, and the output result is a 1*1 scene label value.

[0120] Next, taking a set of first training data as an example, the first training data includes a sample color image with a size of 256*256*3 and its corresponding sample scene label. The sample color image is input into the first backbone network 20 and processed sequentially through the first network layer 21 to the twelfth network layer 212 to finally obtain the predicted scene label.

[0121] Finally, the loss value is calculated based on the cross-entropy loss function, the sample scene label, and the predicted scene label, and the network parameters of the first backbone network 20 are adjusted based on the loss value.

[0122] For example, when it is necessary to train a scene detection network for recognizing scene labels for HDR images, the sample color images are sample HDR images.

[0123] For example, when it is necessary to train a scene detection network for recognizing scene labels of SDR images, the sample color images are sample SDR images.

[0124] In some embodiments of this application, the parameters of the first backbone network are adjusted and trained using a first dataset containing sample color images and their corresponding sample scene labels, enabling the network to learn rich image scene features and patterns. The scene detection network obtained after sufficient training and convergence can more accurately determine the scene to which the image belongs and output precise scene labels when performing semantic scene recognition on the first color image, effectively improving the accuracy and reliability of scene recognition. In the image processing method, a high-quality scene detection network is provided for subsequent steps. Accurate scene labels can provide important prior information for subsequent image processing tasks, enabling subsequent processing to perform more targeted operations based on the characteristics of different scenes, thereby optimizing the performance of the entire image processing method and improving processing effect and efficiency.

[0125] In some embodiments provided in this application, applicable to scenarios where the original image of a first color image is acquired, the provided image processing method, in... Figure 1 Based on the embodiment shown, before step 101 above, the following steps may also be included: step 401 and step 402.

[0126] In step 401, if the image sensor has the function of outputting raw images, the raw image of the first color image acquired by the image sensor is obtained.

[0127] In some embodiments of this application, the image sensor has a raw image output function, which means that the image sensor can output an image format that is closest to the raw light signal data directly captured by its photosensitive element without or with only minimal internal processing.

[0128] In some embodiments of this application, when the image sensor has a raw image output function, the raw image of the first color image acquired by the image sensor is directly obtained. In this case, no additional conversion or processing of the image is required, and the most raw image data can be directly obtained from the image sensor.

[0129] In step 402, when the image sensor does not have the function of outputting the original image, the first color image is input into the backward network, and the image reconstruction processing of the first color image is performed by the backward network to output the original image of the first color image.

[0130] In some embodiments of this application, when the image sensor does not have the function of outputting the original image, a first color image is input into the backward network. The backward network performs image reconstruction processing on the input first color image and finally outputs the original image of the first color image. In this case, a pre-trained network model simulates the original image that the image sensor should have output.

[0131] In some embodiments of this application, the feedforward network is an image reconstruction network with powerful nonlinear modeling capabilities. It can establish a mapping relationship from input to output by learning patterns and features from large amounts of data. In image reconstruction tasks, the feedforward network can learn the mapping from degraded or damaged images to the original clear images.

[0132] For example, when the first color image is an HDR image of a city night scene captured by a mobile phone camera, the HDR image of the city night scene is input into a backpropagation network, and the image reconstruction processing of the HDR image of the city night scene is performed by the backpropagation network to output the RAW image corresponding to the HDR image of the city night scene.

[0133] For example, when the first color image is a food SDR image from social media, the food SDR image is input into a back feeder network, and the back feeder network performs image reconstruction processing on the food SDR image to output the corresponding RAW image.

[0134] In some embodiments of this application, considering that image sensors vary widely in type and function in practical applications, and not all image sensors have raw image output capabilities, the settings of steps 401 and 402 enable the image processing method to be compatible with image sensors of various functions. Whether the electronic device is a high-end professional camera with raw image output capabilities or a common consumer-grade camera without such capabilities, the raw image of the first color image can be acquired through appropriate methods, thereby expanding the applicability of the image processing method. Furthermore, depending on the functionality of the image sensor, the raw image can be acquired flexibly, meeting the diverse image data needs of different users and application scenarios.

[0135] In some embodiments provided in this application, applicable to scenarios where the original image of a first color image is acquired, the provided image processing method, in... Figure 1 Based on the embodiment shown, before step 101 above, the following step may also be included: step 501.

[0136] In step 501, the network parameters of the second backbone network are adjusted according to the second dataset to obtain the converged second backbone network, and the converged second backbone network is determined as the backward network.

[0137] In some embodiments of this application, the feedforward network is used to reconstruct the first color image to obtain the original image of the first color image; wherein, the original image of the first color image is the image reconstruction result.

[0138] In some embodiments of this application, the second backbone network is a convolutional neural network, which serves as the basic architecture for constructing the feedforward network.

[0139] In some embodiments of this application, a second backbone network can be constructed based on a depthwise separable convolutional network to balance the accuracy of the model with the efficiency of training.

[0140] In some embodiments of this application, the second dataset includes multiple sets of second training data, each set of second training data including a sample color image and a sample original image of the sample color image, providing data support for adjusting the parameters of the second backbone network.

[0141] In some embodiments of this application, the second dataset can be expanded and updated according to actual needs, adding new sample color images and corresponding original sample images. The feedforward network trained based on this dataset can continuously learn and adapt to new scene types, which makes the entire image processing method more scalable and adaptable. It can continuously improve scene recognition capabilities and overall processing performance as the data becomes richer and the scene changes, meeting the needs of different application scenarios.

[0142] In some embodiments of this application, the second dataset may include more than 1,000 pairs of sample color images and sample original images. In practical applications, the second dataset may be taken from datasets such as the MIT-Adobe FiveK RAW extension set or SIDDataset.

[0143] In some embodiments of this application, the second training data in the second dataset covers sample color images and their original sample images under various different scenarios. This diverse data helps the second backbone network learn more accurate and richer image features under different scenarios and conditions, thereby improving the network's adaptability to various input images. This enables the subsequent reconstruction of the first color image to more accurately recover the details and textures of its original image, improving the image's clarity and quality.

[0144] In some embodiments of this application, the second backbone network is pre-adjusted before step 101 and determined as the backward network, which can provide a stable and reliable foundation for subsequent image processing steps. This avoids real-time adjustments to the network when processing the first color image, reduces processing time and computational resource consumption, and improves the efficiency of the entire image processing method.

[0145] In some embodiments of this application, the mean squared error loss function can be used as the loss function in the backward network training process.

[0146] In some embodiments of this application, since the mean squared error loss function is calculated based on the square of the error, it tends to make the network output smoother. In image processing, a smooth output can reduce noise and artifacts in the image, making the reconstructed image look more natural and realistic, thereby helping to improve the overall quality of image reconstruction and making the reconstructed image closer to the original image.

[0147] In some embodiments of this application, during the training process of the feedforward network, the gradient of each parameter in the network can be calculated using the backpropagation method, and the network parameters can be updated using the stochastic gradient descent method until the loss function of the network converges. The network parameters are then saved, and the training process of the network ends, thus ensuring the stability and convergence of the training process.

[0148] For example, such as Figure 3 As shown, the second backbone network 30 includes a first network layer 31, a second network layer 32, ..., a thirteenth network layer 313 connected from left to right.

[0149] The first network layer 31 may include a convolutional layer 1, wherein the convolutional kernel size of the convolutional layer 1 is 1×1×3, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3.

[0150] The second network layer 32 may include a convolutional layer 2 and a ReLU layer. The convolutional kernel size of the convolutional layer 2 is 1×1×3, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0151] The third network layer 33 may include a convolutional layer 3 and a ReLU layer. The convolutional kernel size of the convolutional layer 3 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0152] The fourth network layer 34 may include a convolutional layer 4 and a ReLU layer. The convolutional kernel size of the convolutional layer 4 is 1×1×10, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0153] The fifth network layer 35 may include a convolutional layer 5 and a ReLU layer. The convolutional kernel size of the convolutional layer 5 is 1×1×20, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0154] The sixth network layer 36 may include a convolutional layer 6 and a ReLU layer. The convolutional kernel size of the convolutional layer 6 is 1×1×20, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0155] The seventh network layer 37 may include a convolutional layer 7 and a ReLU layer. The convolutional kernel size of the convolutional layer 7 is 1×1×20, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0156] The eighth network layer 38 may include a convolutional layer 8 and a ReLU layer. The convolutional kernel size of the convolutional layer 8 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0157] The ninth network layer 39 may include a convolutional layer 9 and a ReLU layer. The convolutional kernel size of the convolutional layer 9 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0158] The tenth network layer 310 may include a convolutional layer 10 and a ReLU layer. The convolutional kernel size of the convolutional layer 10 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0159] The eleventh network layer 311 may include a convolutional layer 11 and a ReLU layer. The convolutional kernel size of the convolutional layer 11 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0160] The ReLU layers in the second network layer 32 to the tenth network layer 311 all use the function in formula (1) above;

[0161] The twelfth network layer 312 may include a convolutional layer 12, the convolutional kernel size of the convolutional layer 12 is 1×1×10, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3.

[0162] The thirteenth network layer 313 may include a convolutional layer 13. The convolutional kernel size of the convolutional layer 13 is 1×1×3, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3, which is the reconstructed original image.

[0163] Next, taking a second set of training data as an example, the second training data includes a sample color image with a size of 256*256*3 and its corresponding original sample image. The sample color image is input into the second backbone network 30 and processed sequentially through the first network layer 31 to the thirteenth network layer 313 to finally obtain the predicted original image.

[0164] Finally, the loss value is calculated based on the mean squared error loss function, the original sample image, and the original predicted image, and the network parameters of the second backbone network 30 are adjusted based on the loss value.

[0165] For example, when it is necessary to train a backward network for reconstructing HDR images from original images, the sample color images are sample HDR images, and the sample original images are the RAW images corresponding to the sample HDR images.

[0166] For example, when it is necessary to train a backward network for recognizing the original image of the SDR image, the sample color image is the sample SDR image, and the sample original image is the RAW image corresponding to the sample SDR image.

[0167] In some embodiments of this application, since the feedforward network is trained and optimized using a second dataset rather than solely relying on the first color image itself, the image processing method is unaffected by the specific conditions and limitations of the first color image when processing it, thus reducing the requirements for the quality of the input image. Even if the first color image has a certain degree of degradation or damage, the feedforward network can still reconstruct it using the general features it has learned, improving the robustness of the image processing method.

[0168] In some embodiments provided in this application, applicable to scenarios of generating dynamic color lookup tables, step 102 above may include the following steps: step 1021.

[0169] In step 1021, the first color image, the scene label of the first color image, and the original image are input into the dynamic color lookup table generation network. The dynamic color lookup table is used to perform multimodal feature fusion processing on the first color image, the scene label of the first color image, and the original image, and outputs a dynamic color lookup table that matches the shooting scene of the first color image.

[0170] In some embodiments of this application, the dynamic color lookup table generation network is typically a specially designed neural network model. Its function is to receive the input first color image, scene labels, and original image, and to fuse this multimodal information. The network may contain multiple layers of neurons, capable of extracting, transforming, and integrating information from different modalities. Since the input information includes both image and text data, the dynamic color lookup table generation network needs to employ appropriate methods to fuse these different modalities. For example, the scene labels can be converted into vector form and concatenated or weighted with feature vectors extracted from the image, enabling the dynamic color lookup table generation network to comprehensively consider both color and scene information, generating a more accurate dynamic color lookup table.

[0171] For example, when the first color image is an HDR image of a city night scene captured by a mobile phone camera, the HDR image of the city night scene, its original image, and the scene label "food" are input into a dynamic color lookup table generation network to generate a dynamic color lookup table corresponding to the HDR image of the city night scene.

[0172] For example, when the first color image is a food SDR image from social media, the food SDR image, its original image, and the scene label "food" are input into a dynamic color lookup table generation network to generate a dynamic color lookup table corresponding to the food SDR image.

[0173] In some embodiments of this application, after multimodal feature fusion processing by the dynamic color lookup table generation network, the network outputs a dynamic color lookup table specifically customized for the shooting scene of the first color image. This color lookup table contains a mapping relationship from the input image color space to the output image color space, enabling precise adjustment of the colors in the first color image to achieve optimal color effects in different scenes. For example, in night scene scenes, the dynamic color lookup table can enhance the details in the dark areas of the image while maintaining the color gradation in the bright areas, creating a more realistic night scene atmosphere. Compared with traditional static color lookup tables, dynamic color lookup tables can adaptively adjust according to different shooting scenes and image content, improving the flexibility and accuracy of color conversion and meeting the diverse color needs of different users.

[0174] In some embodiments provided in this application, applicable to scenarios involving the generation of dynamic color lookup tables, the provided image processing method, in Figure 1 Based on the illustrated embodiment, prior to step 102 above, the following steps may also be included:

[0175] Step 701.

[0176] In step 701, the network parameters of the third backbone network are adjusted according to the third dataset to obtain the converged third backbone network, and the converged third backbone network is determined as the dynamic color lookup table generation network.

[0177] In some embodiments of this application, a dynamic color lookup table generation network is used to generate a dynamic color lookup table that matches the shooting scene of the first color image.

[0178] In some embodiments of this application, the third backbone network is a convolutional neural network, serving as the basic architecture for constructing a dynamic color lookup table generation network.

[0179] In some embodiments of this application, the third dataset includes multiple sets of third training data. Each set of third training data includes a sample color image, a sample scene label corresponding to the sample color image, a sample original image of the sample color image, and a sample color lookup table that matches the shooting scene of the sample color image. The sample color lookup table is used to perform color conversion on the sample color image to obtain a reference color image.

[0180] In some embodiments of this application, sample color images provide actual image content information for the dynamic color lookup table generation network. The dynamic color lookup table generation network needs to learn various features such as color, texture, and shape from these images in order to subsequently generate a dynamic color lookup table related to the image content.

[0181] In some embodiments of this application, the third dataset contains various types of sample color images, allowing the network to learn more comprehensive and richer image features, such as the shape, texture, and color distribution of different objects. This helps the network better understand the image content and generate a more accurate dynamic color lookup table.

[0182] In some embodiments of this application, sample scene labels are used to explicitly identify the shooting scene type of the sample color image. These labels exist in text form and can be manually annotated or automatically generated based on image recognition algorithms. The sample scene labels provide the dynamic color lookup table generation network with prior knowledge about the image scene. Different shooting scenes have different color characteristics and aesthetic requirements. For example, landscape scenes typically require vibrant colors and rich gradations, while portrait scenes emphasize the natural reproduction of skin tones. Scene labels help the dynamic color lookup table generation network understand the scene in which the image is located, thereby generating a dynamic color lookup table that better matches the characteristics of that scene.

[0183] In some embodiments of this application, the introduction of sample scene labels enables the network to learn and optimize specifically for different scenes. The network can learn corresponding color adjustment strategies based on the characteristics of different scenes. This scene adaptation optimization allows the generated color lookup table to better meet the color requirements of different scenes.

[0184] In some embodiments of this application, the original sample image provides the original light intensity data when the sample color image was captured. The original light intensity data helps the dynamic color lookup table generation network understand the actual light intensity distribution of the shooting scene, thereby generating a dynamic color lookup table that better matches the characteristics of that light intensity.

[0185] In some embodiments of this application, the reference color image is an image with standard colors under ideal conditions or after professional processing. It may be an image captured by a professional photographer using high-end equipment under optimal shooting conditions, or it may be an image that has undergone careful color correction in post-production. The reference color image serves as a reference standard for network training. By comparing the color differences between the sample color image and the reference color image, the dynamic color lookup table generation network learns how to adjust the colors of the sample color image to more closely resemble the color representation of the reference color image. This helps the network master the rules and techniques of color adjustment, improving the accuracy of generating the dynamic color lookup table.

[0186] In some embodiments of this application, a sample color lookup table is matched with the shooting scene of the sample color image to perform color conversion on the sample color image to obtain a reference color image. The sample color lookup table defines the mapping relationship from the color space of the sample color image to the color space of the reference color image, and it contains input color values ​​and corresponding output color values.

[0187] In some embodiments of this application, the sample color lookup table is used as the expected output of the network. The dynamic color lookup table generation network learns the method of generating dynamic color lookup tables by learning the complex mapping relationship between sample color images, sample scene labels, sample original images, and sample color lookup tables. When processing new images subsequently, the dynamic color lookup table generation network can generate similar and compliant dynamic color lookup tables based on the input image and scene information.

[0188] In some embodiments of this application, reference color images and sample color lookup tables provide clear standards and references for training the dynamic color lookup table generation network. The dynamic color lookup table generation network can continuously adjust its parameters to improve generation quality by comparing its output with the sample color lookup table. Simultaneously, the reference color image also provides the network with an ideal color target, enabling the generated color lookup table to be optimized towards a color closer to the original image, ensuring the stability and reliability of the generated color lookup table quality. Furthermore, by learning the color mapping relationships in a large amount of sample data, the network can more accurately grasp the magnitude and direction of color adjustments, avoiding color distortion or over-adjustment problems.

[0189] In some embodiments of this application, the sample color images in the third training set can cover HDR and SDR images, including more than 100 scene types such as backlighting, night scenes, and indoor scenes. The sample scene labels can be annotated by a professional annotation team or automatically generated through a pre-trained scene detection network.

[0190] In some embodiments of this application, the third backbone network is pre-adjusted and determined as a dynamic color lookup table generation network before step 102, which can provide a stable and reliable foundation for subsequent image processing steps. This avoids real-time adjustments to the network when processing the first color image, reduces processing time and computational resource consumption, and improves the efficiency of the entire image processing method.

[0191] In some embodiments of this application, the mean squared error loss function can also be used as the loss function in the training process of the dynamic color lookup table generation network.

[0192] In some embodiments of this application, during the training process of the dynamic color lookup table generation network, the gradient of each parameter in the network can be calculated using the backpropagation method, and the network parameters can be updated using the stochastic gradient descent method until the network loss function converges. Then, the network parameters are saved, and the training process of the network ends, thus ensuring the stability and convergence of the training process.

[0193] For example, such as Figure 4As shown, the third backbone network 40 includes a first network layer 41, a second network layer 42, ..., a twenty-third network layer 423 connected from left to right.

[0194] The first network layer 41 may include a convolutional layer 1, wherein the kernel size of the convolutional layer 1 is 1×1×3, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3.

[0195] The second network layer 42 may include a convolutional layer 2 and a ReLU layer. The convolutional kernel size of the convolutional layer 2 is 1×1×7, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0196] The third network layer 43 may include a convolutional layer 3 and a ReLU layer. The convolutional kernel size of the convolutional layer 3 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0197] The fourth network layer 44 may include a convolutional layer 4 and a ReLU layer. The convolutional kernel size of the convolutional layer 4 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0198] The fifth network layer 45 may include a convolutional layer 5 and a ReLU layer. The convolutional kernel size of the convolutional layer 5 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0199] The sixth network layer 46 may include a convolutional layer 6 and a ReLU layer. The convolutional kernel size of the convolutional layer 6 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0200] The seventh network layer 47 may include a convolutional layer 7 and a ReLU layer. The convolutional kernel size of the convolutional layer 7 is 1×1×10, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0201] The eighth network layer 48 may include a convolutional layer 8 and a ReLU layer. The convolutional kernel size of the convolutional layer 8 is 1×1×20, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0202] The ninth network layer 49 may include a convolutional layer 9 and a ReLU layer. The convolutional kernel size of the convolutional layer 9 is 1×1×20, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0203] The tenth network layer 410 may include a convolutional layer 10 and a ReLU layer. The convolutional kernel size of the convolutional layer 10 is 1×1×20, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0204] The eleventh network layer 411 may include a convolutional layer 11 and a ReLU layer. The convolutional kernel size of the convolutional layer 11 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0205] The twelfth network layer 412 may include a convolutional layer 12, the convolutional kernel size of the convolutional layer 12 is 1×1×10, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3;

[0206] The thirteenth network layer 413 may include a convolutional layer 13, the kernel size of which is 1×1×3, the stride is 1, the number of kernels is 3, and the size of the output feature map is 256*256*3.

[0207] The fourteenth network layer 414 may include a convolutional layer 14 and a ReLU layer. The convolutional kernel size of the convolutional layer 14 is 1×1×6, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0208] The fifteenth network layer 415 may include a convolutional layer 15 and a ReLU layer. The convolutional kernel size of the convolutional layer 15 is 1×1×10, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0209] The sixteenth network layer 416 may include a convolutional layer 16 and a ReLU layer. The convolutional kernel size of the convolutional layer 16 is 4×4×20, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 64*64*20.

[0210] The seventeenth network layer 417 may include a convolutional layer 17 and a ReLU layer. The convolutional kernel size of the convolutional layer 17 is 1×1×20, the stride is 1, the number of convolutional kernels is 40, and the size of the output feature map is 64*64*40.

[0211] The eighteenth network layer 418 may include a convolutional layer 18 and a ReLU layer. The convolutional kernel size of the convolutional layer 18 is 1×1×40, the stride is 1, the number of convolutional kernels is 40, and the size of the output feature map is 64*64*40.

[0212] The nineteenth network layer 419 may include a convolutional layer 19 and a ReLU layer. The convolutional kernel size of the convolutional layer 19 is 1×1×40, the stride is 1, the number of convolutional kernels is 50, and the size of the output feature map is 64*64*50.

[0213] The twentieth network layer 420 may include a convolutional layer 20 and a ReLU layer. The convolutional kernel size of the convolutional layer 20 is 1×1×50, the stride is 1, the number of convolutional kernels is 60, and the size of the output feature map is 64*64*60.

[0214] The twenty-first network layer 421 may include a convolutional layer 21 and a ReLU layer. The convolutional kernel size of the convolutional layer 21 is 1×1×60, the stride is 1, the number of convolutional kernels is 65, and the size of the output feature map is 64*64*65.

[0215] The twenty-second network layer 422 may include a convolutional layer 22 and a ReLU layer. The convolutional kernel size of the convolutional layer 22 is 1×1×65, the stride is 1, the number of convolutional kernels is 65, and the size of the output feature map is 64*64*65. After upsampling, the feature map is obtained as a feature map with a size of 65*65*65, which is then fed into the twenty-third network layer 423.

[0216] The 23rd network layer 423 may include a convolutional layer 23 and a ReLU layer. The convolutional kernel size of the convolutional layer 23 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0217] The ReLU layers in the second network layer 42 to the twenty-third network layer 423 mentioned above can all use the function in formula (1);

[0218] The feature map output from the 23rd network layer 423 is downsampled to obtain data of size 17*17*17, which is the final LUT value; if other precision LUTs are needed, they can be downsampled to 33*33*33.

[0219] Next, taking a third set of training data as an example, the third training data includes a sample color image with a size of 256*256*3 and its corresponding original sample image and sample scene label. The sample color image, the original sample image and the sample scene label are respectively input into the third backbone network 40. After the sample color image is processed by the first network layer 41, the processing result is concatenated with the original sample image and the sample scene label, and then processed by the second network layer 42 to the twenty-third network layer 423 in sequence, and finally obtained by downsampling to obtain the prediction LUT.

[0220] Finally, the loss value is calculated based on the mean squared error loss function, the sample color lookup table, and the prediction LUT, and the network parameters of the third backbone network 40 are adjusted according to the loss value.

[0221] For example, when training a network to generate a dynamic color lookup table for HDR images, both the sample color image and the reference color image are HDR images.

[0222] For example, when it is necessary to train a network for generating a dynamic color lookup table for SDR images, both the sample color image and the reference color image are SDR images.

[0223] In some embodiments of this application, the third dataset contains multiple sets of training data of different types, covering various shooting scenarios and image features. By training on these samples, the dynamic color lookup table generation network can learn a more comprehensive and accurate mapping relationship between image-scene-color lookup table. This enables more precise adjustments based on image content and scene characteristics when subsequently generating a dynamic color lookup table that matches the shooting scene of the first color image, thereby improving the accuracy of color lookup table generation.

[0224] In some embodiments provided in this application, applicable to the scenario of constructing a third backbone network, the provided image processing method may further include the following steps before step 701: step 801 and step 802.

[0225] In step 801, the network parameters of the fourth backbone network are adjusted according to the fourth dataset to obtain the converged fourth backbone network, and the converged fourth backbone network is determined as the feedforward network; wherein, the fourth dataset includes multiple sets of fourth training data, each set of fourth training data includes a sample color image, a sample scene label corresponding to the sample color image, and a reference color image corresponding to the sample color image.

[0226] In some embodiments of this application, a feedforward network can be pre-trained to perform network pre-training processing for the dynamic color lookup table generation network, thereby reducing the network training difficulty of the dynamic color lookup table generation network and accelerating the network convergence speed.

[0227] In some embodiments of this application, a fourth backbone network can be constructed based on a depthwise separable convolutional network to balance the accuracy of the model with the efficiency of training.

[0228] In some embodiments of this application, the fourth dataset may include more than 1,000 pairs of sample color images and their corresponding sample scene labels and reference color images. The sample scene labels may be annotated by a professional annotation team or automatically generated using a pre-trained scene detection network. The reference color image is an ideal color image obtained by color conversion of the sample color images based on a sample color lookup table. The sample color lookup table may be manually created by a senior colorist or generated using a traditional color lookup table generation algorithm and then manually fine-tuned. The ideal color image refers to a color image with a relatively good color conversion effect.

[0229] In some embodiments of this application, the mean squared error loss function can also be used as the loss function in the feedforward network training process.

[0230] In some embodiments of this application, during the training process of the feedforward network, the gradient of each parameter in the network can be calculated using the backpropagation method, and the network parameters can be updated using the stochastic gradient descent method until the loss function of the network converges, the network parameters are saved, and the training process of the network ends.

[0231] For example, such as Figure 5 As shown, the fourth backbone network 50 includes, from left to right: the first network layer 51, the second network layer 52, ..., the thirteenth network layer 513;

[0232] The first network layer 51 may include a convolutional layer 1, wherein the convolutional kernel size of the convolutional layer 1 is 1×1×3, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3.

[0233] The second network layer 52 may include a convolutional layer 2 and a ReLU layer. The convolutional kernel size of the convolutional layer 2 is 1×1×4, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0234] The third network layer 53 may include a convolutional layer 3 and a ReLU layer. The convolutional kernel size of the convolutional layer 3 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0235] The fourth network layer 54 may include a convolutional layer 4 and a ReLU layer. The convolutional layer 4 has a kernel size of 1×1×10, a stride of 1, 10 kernels, and an output feature map size of 256*256*10.

[0236] The fifth network layer 55 may include a convolutional layer 5 and a ReLU layer. The convolutional kernel size of the convolutional layer 5 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0237] The sixth network layer 56 may include a convolutional layer 6 and a ReLU layer. The convolutional kernel size of the convolutional layer 6 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0238] The seventh network layer 57 may include a convolutional layer 7 and a ReLU layer. The convolutional kernel size of the convolutional layer 7 is 1×1×10, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0239] The eighth network layer 58 may include a convolutional layer 8 and a ReLU layer. The convolutional kernel size of the convolutional layer 8 is 1×1×20, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0240] The ninth network layer 59 may include a convolutional layer 9 and a ReLU layer. The convolutional kernel size of the convolutional layer 9 is 1×1×20, the stride is 1, the number of convolutional kernels is 20, and the size of the output feature map is 256*256*20.

[0241] The tenth network layer 510 may include a convolutional layer 10 and a ReLU layer. The convolutional kernel size of the convolutional layer 10 is 1×1×20, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0242] The eleventh network layer 511 may include a convolutional layer 11 and a ReLU layer. The convolutional kernel size of the convolutional layer 11 is 1×1×10, the stride is 1, the number of convolutional kernels is 10, and the size of the output feature map is 256*256*10.

[0243] The ReLU layers in the second network layer 52 to the tenth network layer 511 can all use the functions in formula (1) above;

[0244] The twelfth network layer 512 includes a convolutional layer 12. The convolutional kernel size of the convolutional layer 12 is 1×1×10, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3.

[0245] The thirteenth network layer 513 includes a convolutional layer 13. The convolutional kernel size of the convolutional layer 13 is 1×1×3, the stride is 1, the number of convolutional kernels is 3, and the size of the output feature map is 256*256*3, which is the predicted color image.

[0246] Next, taking a set of fourth training data as an example, the fourth training data includes a sample color image with a size of 256*256*3 and its corresponding reference color image and sample scene label. The sample color image and sample scene label are respectively input into the fourth backbone network 50. After the sample color image is processed by the first network layer 51, the processing result is concatenated with the sample scene label and then processed by the second network layer 52 to the thirteenth network layer 513 in sequence to finally obtain the predicted color image.

[0247] Finally, the loss value is calculated based on the mean squared error loss function, the reference color image, and the predicted color image, and the network parameters of the fourth backbone network 50 are adjusted based on the loss value.

[0248] In step 802, a third backbone network is constructed based on the network structure and network parameters of the feedforward network.

[0249] For example, Figure 4 The network structure and parameters of the first network layer 51 and the third network layers 53 to thirteenth network layers 513 of the third backbone network 40 shown are, in contrast to... Figure 5The first network layer 51 and the third network layers 53 to the thirteenth network layers 513 of the feedforward network shown are the same; compared to Figure 5 The second network layer 52 of the feedforward network shown is... Figure 4 The second network layer 42 of the third backbone network 40 shown includes a convolutional layer 2 and a ReLU layer. The convolutional layer 2 has a kernel size of 1×1×7, a stride of 1, and 10 kernels, and the output feature map size is 256*256*10. Compared to... Figure 5 The feedforward network shown, Figure 4 The fourteenth network layer 414 to the twenty-third network layer 423 of the third backbone network 40 shown are newly added network layers.

[0250] In some embodiments of this application, a feedforward network is trained to learn prior knowledge of the mapping relationship from the original color image to the ideal color image from sample color images, sample scene labels, and reference color images. This trained feedforward network is then used as part of the third backbone network of the dynamic color lookup table generation network, reducing the training difficulty and accelerating network convergence. Based on the feedforward network, one layer is fine-tuned while keeping the network parameters of other layers unchanged. Furthermore, based on the above modifications, additional network layers are added to obtain the third backbone network. That is, some layers of the third backbone network inherit the network parameters of the feedforward network, thereby improving training efficiency.

[0251] In some embodiments provided in this application, the scene detection network and the dynamic color lookup table generation network can be deployed in the ISP link, such as... Figure 6 As shown, the ISP link 60 includes: an image sensor 61, an AWB module 62, a CCM module 63, a LUT module 64, and an image output module 65; the scene detection network and the dynamic color lookup table network are located in the LUT module 64.

[0252] Image sensor 61 acquires a raw image, which is then processed by AWB module 62 and CCM module 63 to obtain a first color image. The first color image is processed by the scene detection network in LUT module 64 to obtain a scene label. The aforementioned first color image, raw image, and scene label are then processed by the LUT generation network in LUT module 64 to obtain a dynamic color lookup table matching the scene captured by the first color image. Image output module 65 performs color conversion processing on the first color image based on the aforementioned dynamic color lookup table to obtain a second color image.

[0253] In some embodiments of this application, manual adjustment of color lookup tables for different scenes is unnecessary. The dynamic color lookup table generation network automatically outputs better color lookup table parameters based on scene detection results, thereby improving image quality and correcting image colors. Furthermore, the collaborative processing strategy of the AWB module, CCM module, and LUT module, through staged correction, dynamic parameter matching, and stylistic optimization, achieves accurate color restoration and stylistic processing from raw image data to the final display output.

[0254] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0255] Figure 7 This is a structural block diagram of an image processing apparatus provided in some embodiments of this application, such as... Figure 7 As shown, the image processing device 900 may include a processing module 901.

[0256] The processing module 901 is used to determine a dynamic color lookup table that matches the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image; wherein, the scene label is used to characterize the semantic scene category of the first color image; and based on the dynamic color lookup table, the first color image is subjected to color conversion processing to generate a second color image.

[0257] In some embodiments of this application, the first color image provides rich spatial texture features, the scene label of the first color image provides semantic information about the shooting scene of the first color image, and the original image of the first color image provides the original light intensity data when the first color image was captured. Because the texture features, semantic scene, and lighting conditions of the image are comprehensively considered, a dynamic color lookup table matching the shooting scene of the first color image can be generated. This dynamic color lookup table differs from traditional static color lookup tables; it can automatically adjust according to different shooting scenes, lighting conditions, and image features, more accurately simulating the color distribution and lighting effects of the actual scene, and has stronger adaptability and flexibility. Using the generated dynamic color lookup table to perform color conversion processing on the first color image, the color-converted second color image can more accurately restore the colors of the original scene, reduce color deviation, and improve the accuracy of color restoration in the color conversion process.

[0258] Optionally, in some embodiments of this application, the processing module 901 is further configured to input the first color image into a scene detection network, perform semantic scene recognition processing on the first color image through the scene detection network, and output scene labels for the first color image.

[0259] Optionally, in some embodiments of this application, the processing module 901 is further configured to adjust the network parameters of the first backbone network according to the first dataset to obtain a converged first backbone network, and determine the converged first backbone network as a scene detection network; wherein, the scene detection network is used to perform semantic scene recognition on the first color image to obtain the scene label of the first color image; the first backbone network is an image classification network, the first dataset includes multiple sets of first training data, each set of first training data includes a sample color image and a sample scene label corresponding to the sample color image.

[0260] Optionally, in some embodiments of this application, the processing module 901 is further configured to acquire the original image of the first color image acquired by the image sensor when the image sensor has the original image output function; and to input the first color image into the backhaul network when the image sensor does not have the original image output function, and to perform image reconstruction processing on the first color image through the backhaul network to output the original image of the first color image.

[0261] Optionally, in some embodiments of this application, the processing module 901 is further configured to adjust the network parameters of the second backbone network according to the second dataset to obtain a converged second backbone network, and determine the converged second backbone network as a feedforward network; wherein, the feedforward network is used to reconstruct the first color image to obtain the original image of the first color image; the second backbone network is a convolutional neural network, and the second dataset includes multiple sets of second training data, each set of second training data including a sample color image and a sample original image of the sample color image.

[0262] Optionally, in some embodiments of this application, the processing module 901 is further configured to input the first color image, the scene label of the first color image, and the original image into a dynamic color lookup table generation network, perform multimodal feature fusion processing on the first color image, the scene label of the first color image, and the original image through the dynamic color lookup table, and output a dynamic color lookup table that matches the shooting scene of the first color image.

[0263] Optionally, in some embodiments of this application, the processing module 901 is further configured to adjust the network parameters of the third backbone network according to the third dataset to obtain a converged third backbone network, and determine the converged third backbone network as a dynamic color lookup table generation network; wherein, the dynamic color lookup table generation network is used to generate a dynamic color lookup table that matches the shooting scene of the first color image; the third backbone network is a convolutional neural network, the third dataset includes multiple sets of third training data, each set of third training data includes a sample color image, a sample scene label corresponding to the sample color image, a sample original image of the sample color image, and a sample color lookup table that matches the shooting scene of the sample color image, the sample color lookup table is used to perform color conversion on the sample color image to obtain a reference color image.

[0264] Optionally, in some embodiments of this application, the processing module 901 is further configured to adjust the network parameters of the fourth backbone network according to the fourth dataset to obtain a converged fourth backbone network, and determine the converged fourth backbone network as a feedforward network; wherein, the fourth dataset includes multiple sets of fourth training data, each set of fourth training data includes a sample color image, a sample scene label corresponding to the sample color image, and a reference color image corresponding to the sample color image; and a third backbone network is constructed based on the network structure and network parameters of the feedforward network.

[0265] Optionally, in some embodiments of this application, the first color image includes at least one of the following: a high dynamic range image and a standard dynamic range image.

[0266] The image processing device in this application embodiment can be an electronic device, or a component of an electronic device such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0267] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0268] The imaging device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the illustrated method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0269] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0270] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0271] Figure 9 This is a schematic diagram of the hardware structure of an electronic device that implements some embodiments of this application.

[0272] The electronic device 1100 includes, but is not limited to, components such as: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

[0273] Those skilled in the art will understand that the electronic device 1100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0274] In some embodiments of this application, the processor 1110 is configured to determine a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image; wherein the scene label is used to characterize the semantic scene category of the first color image; and based on the dynamic color lookup table, perform color conversion processing on the first color image to generate a second color image.

[0275] In some embodiments of this application, the first color image provides rich spatial texture features, the scene label of the first color image provides semantic information about the shooting scene of the first color image, and the original image of the first color image provides the original light intensity data when the first color image was captured. Because the texture features, semantic scene, and lighting conditions of the image are comprehensively considered, a dynamic color lookup table matching the shooting scene of the first color image can be generated. This dynamic color lookup table differs from traditional static color lookup tables; it can automatically adjust according to different shooting scenes, lighting conditions, and image features, more accurately simulating the color distribution and lighting effects of the actual scene, and has stronger adaptability and flexibility. Using the generated dynamic color lookup table to perform color conversion processing on the first color image, the color-converted second color image can more accurately restore the colors of the original scene, reduce color deviation, and improve the accuracy of color restoration in the color conversion process.

[0276] Optionally, in some embodiments of this application, the processor 1110 is specifically configured to input a first color image into a scene detection network, perform semantic scene recognition processing on the first color image through the scene detection network, and output scene labels for the first color image.

[0277] Optionally, in some embodiments of this application, the processor 1110 is further configured to adjust the network parameters of the first backbone network according to the first dataset to obtain a converged first backbone network, and determine the converged first backbone network as a scene detection network; wherein, the scene detection network is used to perform semantic scene recognition on the first color image to obtain the scene label of the first color image; the first backbone network is an image classification network, the first dataset includes multiple sets of first training data, each set of first training data includes a sample color image and a sample scene label corresponding to the sample color image.

[0278] Optionally, in some embodiments of this application, the processor 1110 is further configured to acquire the original image of the first color image acquired by the image sensor when the image sensor has the original image output function; and to input the first color image into a backward network when the image sensor does not have the original image output function, perform image reconstruction processing on the first color image through the backward network, and output the original image of the first color image.

[0279] Optionally, in some embodiments of this application, the processor 1110 is further configured to adjust the network parameters of the second backbone network according to the second dataset to obtain a converged second backbone network, and to determine the converged second backbone network as a backward network; the backward network is used to perform image reconstruction on the first color image to obtain the original image of the first color image; the second backbone network is a convolutional neural network, and the second dataset includes multiple sets of second training data, each set of second training data including a sample color image and a sample original image of the sample color image.

[0280] Optionally, in some embodiments of this application, the processor 1110 is specifically configured to input a first color image, a scene label of the first color image, and an original image into a dynamic color lookup table generation network, perform multimodal feature fusion processing on the first color image, the scene label of the first color image, and the original image through the dynamic color lookup table, and output a dynamic color lookup table that matches the shooting scene of the first color image.

[0281] Optionally, in some embodiments of this application, the processor 1110 is further configured to adjust the network parameters of the third backbone network according to the third dataset to obtain a converged third backbone network, and to determine the converged third backbone network as a dynamic color lookup table generation network; wherein, the dynamic color lookup table generation network is used to generate a dynamic color lookup table that matches the shooting scene of the first color image; the third backbone network is a convolutional neural network, the third dataset includes multiple sets of third training data, each set of third training data includes a sample color image, a sample scene label corresponding to the sample color image, a sample original image of the sample color image, and a sample color lookup table that matches the shooting scene of the sample color image, the sample color lookup table is used to perform color conversion on the sample color image to obtain a reference color image.

[0282] Optionally, in some embodiments of this application, the processor 1110 is further configured to adjust the network parameters of the fourth backbone network according to the fourth dataset to obtain a converged fourth backbone network, and determine the converged fourth backbone network as a feedforward network; wherein, the fourth dataset includes multiple sets of fourth training data, each set of fourth training data includes a sample color image, a sample scene label corresponding to the sample color image, and a reference color image corresponding to the sample color image; and a third backbone network is constructed based on the network structure and network parameters of the feedforward network.

[0283] Optionally, in some embodiments of this application, the first color image includes at least one of the following: a high dynamic range image and a standard dynamic range image.

[0284] It should be understood that, in this embodiment, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042. The GPU 11041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1106 may include a display panel 11061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0285] The memory 1109 can be used to store software programs and various data. The memory 1109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0286] Processor 1110 may include one or more processing units; in some embodiments, processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the modem processor may also not be integrated into processor 1110.

[0287] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0288] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0289] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here. It should be understood that the chip mentioned in this application embodiment can also be called a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0290] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0291] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0292] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium such as ROM / RAM, magnetic disk, or optical disk, and includes several instructions to cause a terminal such as a mobile phone, computer, server, or network device to execute the methods described in the various embodiments of this application.

[0293] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Based on the first color image, the scene label of the first color image, and the original image, a dynamic color lookup table matching the shooting scene of the first color image is determined; wherein, the scene label is used to characterize the semantic scene category of the first color image; Based on the dynamic color lookup table, the first color image is subjected to color conversion processing to generate a second color image.

2. The method according to claim 1, characterized in that, Before determining the dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes: The first color image is input into the scene detection network, and the scene detection network performs semantic scene recognition processing on the first color image to output the scene label of the first color image.

3. The method according to claim 1 or 2, characterized in that, Before determining the dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes: Based on the first dataset, the network parameters of the first backbone network are adjusted to obtain the converged first backbone network, and the converged first backbone network is determined as the scene detection network. The scene detection network is used to perform semantic scene recognition on the first color image to obtain scene labels for the first color image; The first backbone network is an image classification network, and the first dataset includes multiple sets of first training data. Each set of first training data includes a sample color image and a sample scene label corresponding to the sample color image.

4. The method according to claim 1, characterized in that, Before determining the dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes: When the image sensor has the function of outputting raw images, the raw image of the first color image acquired by the image sensor is obtained; When the image sensor does not have the function of outputting the original image, the first color image is input into the backward network, and the image reconstruction processing of the first color image is performed by the backward network to output the original image of the first color image.

5. The method according to claim 1 or 4, characterized in that, Before determining the dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes: Based on the second dataset, the network parameters of the second backbone network are adjusted to obtain the converged second backbone network, which is then identified as the backward network. The feedforward network is used to reconstruct the first color image to obtain the original image of the first color image; The second backbone network is a convolutional neural network, and the second dataset includes multiple sets of second training data. Each set of second training data includes a sample color image and a sample original image of the sample color image.

6. The method according to claim 1, characterized in that, The step of determining a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image includes: The first color image, its scene label, and the original image are input into a dynamic color lookup table generation network. The network then performs multimodal feature fusion processing on the first color image, its scene label, and the original image using the dynamic color lookup table, and outputs a dynamic color lookup table that matches the shooting scene of the first color image.

7. The method according to claim 1 or 6, characterized in that, Before determining the dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image, the method further includes: Based on the third dataset, the network parameters of the third backbone network are adjusted to obtain the converged third backbone network. The converged third backbone network is then identified as the dynamic color lookup table generation network. The dynamic color lookup table generation network is used to generate a dynamic color lookup table that matches the shooting scene of the first color image. The third backbone network is a convolutional neural network. The third dataset includes multiple sets of third training data. Each set of third training data includes a sample color image, a sample scene label corresponding to the sample color image, a sample original image of the sample color image, and a sample color lookup table that matches the shooting scene of the sample color image. The sample color lookup table is used to perform color conversion on the sample color image to obtain a reference color image.

8. The method according to claim 7, characterized in that, Before adjusting the network parameters of the third backbone network based on the third dataset to obtain the converged third backbone network, and before determining the converged third backbone network as the dynamic color lookup table generation network, the method further includes: Based on the fourth dataset, the network parameters of the fourth backbone network are adjusted to obtain the converged fourth backbone network, which is then determined as the feedforward network. The fourth dataset includes multiple sets of fourth training data, each set of fourth training data including a sample color image, a sample scene label corresponding to the sample color image, and a reference color image corresponding to the sample color image. Based on the network structure and parameters of the feedforward network, a third backbone network is constructed.

9. The method according to claim 1, characterized in that, The first color image includes at least one of the following: a high dynamic range image and a standard dynamic range image.

10. An image processing apparatus, characterized in that, The device includes: The processing module is configured to determine a dynamic color lookup table matching the shooting scene of the first color image based on the first color image, the scene label of the first color image, and the original image; wherein the scene label is used to characterize the semantic scene category of the first color image; and based on the dynamic color lookup table, perform color conversion processing on the first color image to generate a second color image.

11. The apparatus according to claim 10, characterized in that, The processing module is further configured to acquire the original image of the first color image acquired by the image sensor when the image sensor has the original image output function; and to input the first color image into the backhaul network when the image sensor does not have the original image output function, and to perform image reconstruction processing on the first color image through the backhaul network to output the original image of the first color image.

12. The apparatus according to claim 10 or 11, characterized in that, The processing module is further configured to adjust the network parameters of the second backbone network according to the second dataset to obtain a converged second backbone network, and to determine the converged second backbone network as a feedforward network; wherein, the feedforward network is used to reconstruct the first color image to obtain the original image of the first color image; the second backbone network is a convolutional neural network, and the second dataset includes multiple sets of second training data, each set of second training data including a sample color image and a sample original image of the sample color image.

13. The apparatus according to claim 10, characterized in that, The processing module is further configured to adjust the network parameters of the third backbone network according to the third dataset to obtain a converged third backbone network, and to determine the converged third backbone network as a dynamic color lookup table generation network; wherein, the dynamic color lookup table generation network is used to generate a dynamic color lookup table that matches the shooting scene of the first color image; the third backbone network is a convolutional neural network, the third dataset includes multiple sets of third training data, each set of third training data includes a sample color image, a sample scene label corresponding to the sample color image, a sample original image of the sample color image, and a sample color lookup table that matches the shooting scene of the sample color image, the sample color lookup table is used to perform color conversion on the sample color image to obtain a reference color image.

14. The apparatus according to claim 13, characterized in that, The processing module is further configured to adjust the network parameters of the fourth backbone network according to the fourth dataset to obtain a converged fourth backbone network, and to determine the converged fourth backbone network as the feedforward network; wherein, the fourth dataset includes multiple sets of fourth training data, each set of fourth training data includes a sample color image, a sample scene label corresponding to the sample color image, and a reference color image corresponding to the sample color image; and to construct a third backbone network based on the network structure and network parameters of the feedforward network.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1 to 9.

Images