Image processing method and electronic device

Abstract

Embodiments of the present application apply to the field of artificial intelligence, and provide an image processing method and an electronic device. The method comprises: determining a plurality of image regions of a first image and a priority corresponding to each image region in the plurality of image regions, the plurality of image regions constituting the first image, and there being no overlap between the plurality of image regions; performing color adjustment on each image region according to the priority corresponding to each image region to determine a target image, wherein the granularity of the color adjustment on each image region is negatively correlated with the priority of each image region. Based on the technical method of the present application, the occupation of processing resources by color adjustment can be reduced, and the user has a higher satisfaction degree for the adjusted image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to an image processing method and an electronic device. Background Technology

[0002] Color adjustment plays a crucial role in photography and image processing. Appropriate color adjustments can enhance the artistic quality of a work, compensate for shortcomings during shooting, and adapt to the visual style requirements of different output scenarios.

[0003] How to reduce the processing resource consumption of color adjustment while ensuring user satisfaction with the color-adjusted image is an urgent problem to be solved. Summary of the Invention

[0004] This application provides an image processing method and electronic device that can reduce the resource consumption of color adjustment and enable users to have a high degree of satisfaction with the adjusted image.

[0005] In a first aspect, an image processing method is provided, comprising: determining a plurality of image regions of a first image and a priority corresponding to each of the plurality of image regions, wherein the plurality of image regions constitute the first image and there is no overlap between the plurality of image regions; and adjusting the color of each image region according to the priority corresponding to each image region to determine a target image, wherein the granularity of the color adjustment of each image region is negatively correlated with the priority of each image region.

[0006] The image processing method provided in this application divides a first image into multiple image regions of different priorities. Fine-grained color adjustment is performed on high-priority regions, while coarse-grained color adjustment is performed on low-priority regions. Fine-grained color adjustment consumes more processing resources. Priority indicates importance. Fine-grained color adjustment on higher-priority (more important) image regions results in higher user satisfaction with the adjusted image. Conversely, coarse-grained color adjustment on lower-priority (less important) image regions has less impact on user satisfaction and reduces the resource consumption of color adjustment.

[0007] In some possible implementations, determining multiple image regions of the first image and the priority of each image region in the multiple image regions includes: performing saliency detection on the first image to determine salient regions and non-salient regions other than the salient regions in the first image, wherein the priority of the salient regions is higher than the priority of the non-salient regions, and the multiple image regions include the salient regions and the non-salient regions.

[0008] Different priorities represent different levels of salience. The salience of an image refers to the ability of certain regions in an image to attract visual attention relative to other regions. Salient regions have stronger visual appeal and can be considered higher-priority image regions.

[0009] Furthermore, by prioritizing salient regions in the first image, the determination of image regions does not depend on user operations, which reduces the reliance of the image processing process on user actions and improves the user experience.

[0010] In some possible implementations, the method further includes: performing object segmentation on the first image to determine at least one object region in the first image, where different object regions correspond to different objects, and the edge of each object region is the edge of the object corresponding to the object region; performing saliency detection on the first image to determine salient regions and non-salient regions other than the salient regions in the first image includes: performing saliency detection on the first image to obtain an initial salient region in the first image; determining the salient region based on the initial salient region and the at least one object region, wherein the salient region includes at least one target object region among the at least one object region, and each target object region overlaps with the initial salient region; and determining the regions in the first image other than the salient regions as the non-salient regions.

[0011] The first image may contain multiple objects. A user may be interested in one or more of these objects, and their attention will primarily focus on those objects. The initial salient region edges obtained through salientity detection may not actually be the edges of an object; parts of the object the user is interested in may lie outside the initial salient region. Based on the initial salient region obtained from salientity detection of the first image and the multiple object regions obtained from object segmentation of the first image, a salient region is determined, ensuring that the object the user is interested in is completely located within the salient region. This makes the salient region more accurate, thereby improving user satisfaction with the target image.

[0012] In some possible implementations, determining the salient region based on the initial salient region and the at least one object region includes: for any one of the at least one object region, determining the overlap area between the arbitrary object region and the initial salient region; and determining the arbitrary object region as the target object region if the overlap area between the arbitrary object region and the initial salient region is greater than or equal to the product of a preset area ratio and the area of ​​the arbitrary object region.

[0013] Based on the relationship between the area ratio of the portion of the object region located in the initial salient region and the preset area ratio, it is determined whether the salient region includes the object region. This avoids object regions that only partially overlap with the initial salient region being used as target object regions, reducing the impact of the accuracy of the initial salient region on the accuracy of the salient region, improving the accuracy of the salient region, and reducing the processing resource consumption of color adjustment.

[0014] In some possible implementations, the step of performing object segmentation on the first image to determine at least one object region in the first image includes: determining location information based on the first image, the location information representing multiple location points uniformly distributed in the first image, and the number of the multiple location points being greater than or equal to a preset number; processing the location information and the first image through an object segmentation model to determine the at least one object region, wherein at least one object recorded in the at least one object region includes objects located at each of the multiple location points in the first image.

[0015] During object segmentation of the first image, the location information and the first image are processed using an object segmentation model. The location information represents multiple location points evenly distributed in the first image. Thus, at least one object region output by the object segmentation model records at least one object, including the object located at each location point represented by the location information in the first image. This avoids objects in the first image being missed from being recorded outside of at least one object in at least one object region output by the object segmentation model, ensuring that at least one object region records at least one pair of objects that are all objects in the first image, making the at least one object region more accurate.

[0016] In some possible implementations, adjusting the color of each image region according to its priority to determine the target image includes: dividing each image region into multiple image blocks according to its priority, wherein the size of each image block in each image region is negatively correlated with its priority; determining a representative color for each image block in any given image region; and adjusting the color of each image block in the given image region according to its representative color to determine the target image.

[0017] In the first image, each image patch is color-adjusted based on its representative color. The size of each image patch is negatively correlated with the priority of the image region it belongs to, thus allowing for fine-grained color adjustment of high-priority image regions and coarse-grained color adjustment of low-priority image regions. In the adjusted target image, each image patch can have the same color.

[0018] In some possible implementations, adjusting the color of each image block in the multiple image blocks of the arbitrary image region based on the representative color of each image block in the multiple image blocks of the arbitrary image region to determine the target image includes: based on the first image, determining multiple color mapping relationships through a color mapping model, each of the multiple color mapping relationships corresponding to a different priority, each color mapping relationship representing the relationship between each first color range in multiple first color ranges and its corresponding first mapping parameter, the number of first color ranges in each color mapping relationship being positively correlated with the priority of each color mapping relationship, and the multiple first color ranges in any color mapping relationship covering the color range represented by the color space; adjusting the color of each image block in the multiple image blocks of the arbitrary image region based on the representative color of each image block in the multiple image blocks of the arbitrary image region, the priority of the arbitrary image region, and the multiple color mapping relationships to determine the target image.

[0019] The color range represented by a color space can be understood as all colors in the color space. By processing the first image through a color mapping model, multiple color mapping relationships are determined, making these multiple first mapping relationships compatible with the first image, thereby improving the user's satisfaction with the target image obtained through color adjustment.

[0020] The number of first color ranges in each color mapping relationship is negatively correlated with the priority of that color mapping relationship; different first color ranges correspond to different first mapping parameters. By setting more first color ranges in higher-priority color mapping relationships and fewer first color ranges in lower-priority color mapping relationships, the color adjustment of higher-priority image regions is more refined, while the color adjustment of lower-priority image regions is coarser.

[0021] In some possible implementations, adjusting the color of each image block in the multiple image blocks of the arbitrary image region based on the representative color of each image block in the multiple image blocks of the arbitrary image region, the priority of the arbitrary image region, and the multiple color mapping relationships includes: determining a corresponding target color mapping relationship from the multiple color mapping relationships according to the priority of the arbitrary image region; determining a target first mapping parameter corresponding to the target first color range to which the representative color of any image block in the multiple image blocks of the arbitrary image region belongs based on the target color mapping relationship; determining a target adjustment color corresponding to the representative color of the arbitrary image block according to the target first mapping parameter; and adjusting the color of the arbitrary image block to the determined target adjustment color corresponding to the arbitrary image block.

[0022] For each image block, the color of the image block is adjusted according to the target color mapping relationship of the image region to which the image block belongs, based on the target first mapping parameter to which the color of the image block belongs in the target color mapping relationship, so that the target image can be obtained.

[0023] In some possible implementations, the color mapping model is further used to determine a color correction matrix; adjusting the color of each image block in the multiple image blocks of the arbitrary image region based on the representative color of each image block in the multiple image blocks of the arbitrary image region, the priority of the arbitrary image region, and the multiple color mapping relationships includes: determining the intermediate color corresponding to the representative color of any image block in the multiple image blocks of the arbitrary image region according to the color correction matrix; determining the corresponding target color mapping relationship from the multiple color mapping relationships according to the priority of the arbitrary image region; determining the target first mapping parameter corresponding to the target first color range to which the intermediate color of the arbitrary image block belongs based on the target color mapping relationship; determining the target adjustment color corresponding to the intermediate color of the arbitrary image block according to the target first mapping parameter; and adjusting the color of the arbitrary image block to the determined target adjustment color corresponding to the arbitrary image block.

[0024] A color correction matrix can be used for coarse-grained color adjustment. Multiple color mapping relationships can be used for fine-grained color adjustment. After coarse-grained color adjustment, the range of color adjustment during fine-grained color adjustment using color mapping relationships is smaller; that is, the value range of the first color mapping parameter in the color mapping relationship is smaller. Determining the first color mapping parameter within a smaller value range ensures that, compared to a color mapping model that only outputs information for fine-grained color adjustment, the color mapping matrix outputs information for both coarse-grained and fine-grained color adjustment. This results in shorter processing time and less resource consumption for the color mapping model, improving image processing efficiency.

[0025] In some possible implementations, determining multiple color mapping relationships based on the first image using a color mapping model includes: determining multiple second mapping parameters based on the first image using the color mapping model, wherein each of the multiple second mapping parameters corresponds one-to-one with a second color range in a set of multiple second color ranges, and each set of multiple second color ranges is obtained by dividing the color range represented by the color space according to different granularities; selecting N second color range sets from the multiple second color range sets in descending order of granularity, and determining a first color mapping relationship based on the second color ranges in the N second color range sets and the corresponding second mapping parameters, wherein the first color mapping relationship corresponds to a first priority, and N is an integer greater than or equal to 1; selecting M second color range sets from the multiple second color range sets in descending order of granularity, and determining a second color mapping relationship based on the second color ranges in the M second color range sets and the corresponding second mapping parameters, wherein the second color mapping relationship corresponds to a second priority, the second priority being higher than the first priority, and M being an integer greater than N.

[0026] Based on multiple second color ranges in at least one set of second color ranges with the largest granularity, multiple first color ranges in multiple color mapping relationships are determined. Then, based on the second mapping parameters corresponding to each of the multiple second color ranges, a first mapping parameter corresponding to each first color range is determined, thereby establishing multiple color mapping relationships. Selecting a set of second color ranges from the multiple sets of second color ranges in descending order of granularity can be understood as selecting a set of second color ranges in descending order of adjustment fineness. Therefore, the more sets of second color ranges selected, the finer the color adjustment can be based on the color mapping relationships determined by the selected sets of second color ranges.

[0027] By selecting different numbers of second color range sets from multiple second color range sets, the color mapping relationships corresponding to multiple priorities are determined, making the determination of color mapping relationships corresponding to multiple priorities simpler, reducing the consumption of processing resources for determining color mapping relationships corresponding to multiple priorities, and improving image processing efficiency.

[0028] Compared to the color mapping model, which outputs multiple first mapping parameters corresponding to multiple color mapping relationships, the color adjustment model outputs information representing multiple second mapping parameters that correspond one-to-one with the second color ranges in multiple sets of second color ranges. The color adjustment model outputs less information, so it requires less computation and consumes fewer processing resources, thus improving the efficiency of image processing.

[0029] In some possible implementations, determining the second color mapping relationship based on the second color ranges in the M sets of second color ranges and their corresponding second mapping parameters includes: determining multiple first color ranges in the second color mapping relationship based on the second color ranges in the M sets of second color ranges, and determining M second mapping parameters corresponding to each first color range in the second color mapping relationship, wherein the multiple first color ranges in the second color mapping relationship are obtained by dividing the color range represented by the color space according to the upper and lower limits of each second color range in the M sets of second color ranges; and taking the sum of the M second mapping parameters corresponding to each first color range in the second color mapping relationship as the first mapping parameter corresponding to each first color range in the second color mapping relationship.

[0030] In some possible implementations, determining multiple second mapping parameters based on the first image and through the color mapping model includes: inputting the first image into the color mapping model for processing, outputting multiple weight parameters, wherein the multiple weight parameters correspond one-to-one with multiple sets of second color ranges, and multiple second color ranges in each set of second color ranges correspond one-to-one with multiple preset third mapping parameters, wherein the second mapping parameter corresponding to the second color range in each set of second color ranges is the product of the third mapping parameter corresponding to the second color range and the weight parameter corresponding to the second color range set.

[0031] The first image is processed using a color mapping model. The output of the color mapping model is the weight parameter corresponding to each of the multiple second color range sets. The amount of data output by the color mapping model is small, the amount of computation required for processing by the color adjustment model is small, and the consumption of processing resources is less, which can improve the efficiency of image processing.

[0032] In some possible implementations, the color mapping model is obtained by adjusting the parameters of an initial color mapping model based on the difference between the training image and the label image. The initial training image is obtained by color-adjusting the initial image according to the training color mapping relationship. The training color mapping relationship is determined based on multiple training second mapping parameters. The multiple training second mapping parameters are determined based on the initial image and through the initial color mapping model. The multiple training second mapping parameters correspond one-to-one with the second color ranges in the multiple sets of second color ranges.

[0033] In some possible implementations, the method is applied to an electronic device, and the method further includes: cropping a second image captured by the camera of the electronic device to obtain the first image, wherein the color adjustment is used to achieve white balance.

[0034] If the image before cropping is a white-balanced image, the first image is only a portion of the original image and may not meet white balance requirements. Adjusting the colors of the first image can restore white balance, thereby improving the user experience.

[0035] If the image before cropping is an uncolored image, then color adjustment is applied to the first image obtained after cropping, while other areas of the original image are not color-adjusted. This improves image processing efficiency and enhances user experience. The second image can be understood as an uncolored image.

[0036] Secondly, a color adjustment model training method is provided. This method involves acquiring a training image and a label image, where the label image is obtained by color-adjusting the training image. Based on the training image, multiple training second mapping parameters are determined using an initial color mapping model. These parameters correspond one-to-one with second color ranges in multiple sets of second color ranges, each set being obtained by dividing the color space representation into different granularities. The training image is then color-adjusted according to these sets of second color ranges and the training second mapping parameters to obtain a trained adjusted image. Finally, the parameters of the initial color mapping model are adjusted based on the difference between the trained adjusted image and the label image to obtain the final color mapping model.

[0037] Thirdly, an image processing apparatus is provided, including a unit for performing the method of the first or second aspect. This apparatus may be a terminal device or a chip within a terminal device.

[0038] Fourthly, an electronic device is provided, including one or more processors and a memory coupled to the one or more processors, the memory being used to store computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform the methods of the first aspect and / or the second aspect.

[0039] Fifthly, a chip system is provided, the chip system being applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the methods of the first aspect and / or the second aspect.

[0040] In a sixth aspect, a computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the methods of the first aspect and / or the second aspect.

[0041] In a seventh aspect, a computer program product is provided, the computer program product comprising: computer program code, wherein when the computer program code is run on an electronic device, the electronic device performs the methods of the first aspect and / or the second aspect. Attached Figure Description

[0042] Figures 1 to 3 This is a schematic diagram of the graphical user interface provided in an embodiment of this application;

[0043] Figure 4 This is a schematic structural diagram of an image processing system provided in an embodiment of this application;

[0044] Figure 5 This is a schematic structural diagram of another image processing system provided in the embodiments of this application;

[0045] Figure 6 This is a schematic flowchart of an object segmentation model provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram of an area determined by an image processing system according to an embodiment of this application;

[0047] Figure 8 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0048] Figure 9 This is a schematic diagram of the color range provided in the embodiments of this application;

[0049] Figure 10 This is a schematic flowchart illustrating a training method for a neural network model provided in an embodiment of this application;

[0050] Figure 11 This is a schematic structural diagram of the electronic device provided in the embodiments of this application;

[0051] Figure 12 This is a schematic diagram of the software system of the electronic device provided in the embodiments of this application;

[0052] Figure 13 This is a schematic structural diagram of an image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0054] To facilitate understanding, before introducing the image processing method provided in the embodiments of this application, the terms or nouns involved in the embodiments of this application will be briefly introduced.

[0055] (1) Artificial Intelligence

[0056] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0057] (2) Machine Learning

[0058] Machine learning is an important branch of artificial intelligence, and deep learning is an important branch of machine learning. Deep learning refers to using multi-layer neural network structures to learn representations of various things in the real world that can be directly used for computer calculations (such as objects in images, sounds in audio, etc.) from big data.

[0059] (3) Neural Network

[0060] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0061]

[0062] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For xs The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0063] (4) Deep Neural Networks

[0064] Deep neural networks (DNNs), also known as multi-layer neural networks, can be understood as neural networks with multiple hidden layers. Based on the position of the layers, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0065] In deep neural networks, more hidden layers allow the network to better depict complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by vectors from many layers).

[0066] (5) Convolutional Neural Networks

[0067] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units on the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0068] When a convolutional neural network has multiple convolutional layers, the initial convolutional layers tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by later convolutional layers become more and more complex, such as high-level semantic features. The higher the level of the features, the more suitable they are for the problem to be solved.

[0069] (6) Loss Function

[0070] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0071] (7) Backpropagation algorithm

[0072] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0073] (8) White Balance

[0074] White balance is simply the balance of white. It can also be understood as restoring white objects to their true white color under any light source. To compensate for color casts that occur when shooting under specific light sources, the corresponding complementary color is enhanced, thus achieving white balance. By adjusting the colors and correcting the color temperature of the image captured by the camera, the colors of the subject can be restored, making the colors of images captured under different light sources similar to those seen by the human eye, thus achieving white balance.

[0075] White can be understood as a bright color without a specific hue. White contains all the color components of the visible spectrum, and these color components are roughly equal in intensity, thus producing the visual effect of white. In other words, white represents a balanced mixture of all the colors of the visible spectrum, resulting in a bright, colorless visual effect.

[0076] (9) Pixel value

[0077] Image pixel values ​​are typically represented as integers or floating-point numbers, representing the intensity of different color channels. In the red-green-blue (RGB) color model, each color channel is usually represented by an 8-bit integer, with a value range of 0-255 (256 values ​​in total). Therefore, the pixel value of a pixel can be represented by a triplet consisting of three 8-bit integers. In the triplet representing the pixel value, the three elements correspond to the three colors: red, green, and blue. The value of each element represents the brightness of the corresponding color; for example, (255, 0, 0) represents pure red. Each element in the triplet represents a color component. Among the color components, the smaller the value, the lower the brightness; the larger the value, the higher the brightness.

[0078] For grayscale images, pixel values ​​can be grayscale values.

[0079] (10) Color correction matrix

[0080] A color correction matrix (CCM) is primarily used to correct the colors in an image to reproduce the effects perceived by the human eye. A CCM is a 3×3 matrix used to map the original RGB color space to a new RGB color space that better matches human perception. This process is performed within the RGB domain. While a CCM can prioritize ensuring that the most important colors appear "correct" to the human eye, it may not guarantee optimal performance for all colors under all conditions.

[0081] The color corrected by CCM can be represented as:

[0082]

[0083] Where r, g, and b are the red, green, and blue color components in the color after CCM correction, respectively; r', g', and b' are the red, green, and blue color components in the original color before CCM correction; and parameter c... 11 c 12 c 13 c 21 c 22 c 23 c 31 c 32 c 33 The resulting matrix is ​​CCM.

[0084] When using Color Correction (CCM) to correct an image, for each original color in the image, the CCM is multiplied by the original color vector representing that original color, and the resulting product is the corrected color vector representing the corrected color. Both the original and corrected color vectors are column vectors. CCM correction based on CCM can be understood as color adjustment based on CCM.

[0085] (11) Color lookup table

[0086] A color lookup table (LUT) is essentially a pre-calculated table that stores the correspondence between input colors and output colors. When a color value needs to be converted, the corresponding output value is simply looked up in the LUT, without the need for complex calculations.

[0087] The LUT used in color correction can also be called a 3D LUT (3DLUT). For the original color with red, blue, and green components r', g', and b' respectively, the red, blue, and green components r, g, and b in the LUT-corrected color can be represented as follows:

[0088] r=r'+δr

[0089] g=g'+δg

[0090] b=b'+δb

[0091] Where δr is the offset value of the red component corresponding to the original color (r', g', b') recorded in the LUT, δg is the offset value of the green component corresponding to the original color (r', g', b') recorded in the LUT, and δb is the offset value of the blue component corresponding to the original color (r', g', b') recorded in the LUT.

[0092] LUTs can also be represented using parameters from other color spaces. For example, a LUT can also record the offsets of each color in the hue, saturation, and lightness components in the hue-saturation-lightness (HSL, or hue saturation value, HSV) color space.

[0093] When using a LUT to correct colors in an image, the original colors in the image are used as the input colors of the LUT. The output color corresponding to the input color can be determined based on the LUT. LUT correction can be understood as color adjustment based on the LUT.

[0094] LUTs can be used for image enhancement, retouching, color transformation, and post-production color grading and rendering in film and television works.

[0095] By finely adjusting the color tones of a photo, one can imbue it with a unique atmosphere and emotion. For example, enhancing warm tones can create a cozy and romantic scene, while adding cool tones can give the image a cold, solemn, and mysterious beauty. Appropriate color adjustments can elevate the artistry of an image. Utilizing parameters such as contrast and saturation can make the image more visually impactful and captivating.

[0096] When shooting is limited by lighting and environmental conditions, resulting in overly bright or underly dark situations, adjusting the exposure and brightness parameters can effectively correct the problem.

[0097] For image acquisition devices such as cameras, due to the limitations of hardware sensors, the image output by the sensor will differ from the image seen by the human eye, resulting in a difference between the color of objects in the captured image and the actual color of the subject as perceived by the human eye. If white balance deviation and color distortion occur during shooting, they can be corrected through color adjustment.

[0098] Figure 1 Image (a) shows a GUI for an electronic device, which is a camera interface 1210. The camera interface 1210 includes a preview frame 1211, a camera icon 1212, and an album icon 1213.

[0099] The preview box 1211 is used to display images captured in real time by the camera of the electronic device. The image in the preview box can also be called a preview image. When the user clicks the camera icon 1212 on the camera interface 1210, the electronic device adjusts the colors of the latest image captured by the camera and stores the color-adjusted image in the album.

[0100] When the electronic device detects that the user has clicked the album icon 1213 in the fourth camera interface 1210, it displays... Figure 1 The fourth album interface 1220 is shown in (b) of the diagram. The fourth album interface 1220 may include a captured image 1221. The captured image 1221 may be a color-adjusted image that was last stored in the album.

[0101] After cropping the image, you can also adjust the colors of the cropped image.

[0102] Figure 2Image (a) illustrates a graphical user interface (GUI) for an electronic device, specifically a first album interface 1110. Upon detecting a user clicking on any one of the thumbnails on the first album interface 1110, the electronic device can display, as shown below... Figure 2 The second album interface 1120 is shown in (b) above. The second album interface 1120 includes a full-size image 1121 corresponding to the thumbnail clicked by the user. The second album interface 1120 also includes an auto-cropping icon 1122.

[0103] When the electronic device detects that the user clicks the auto-cropping icon 1122, it can perform image processing on the full-size image 1121 to obtain the cropped image.

[0104] When an electronic device processes an image to obtain a cropped image, it can be displayed. Figure 2 The third album interface 1130 is shown in (c). The third album interface 1130 may include cropped images 1131.

[0105] Image 1121 can be an image that has undergone color adjustment but not yet. For image 1121 that has undergone color adjustment, since the color adjustment is applied to the entire image, the cropped image may not meet the user's color requirements. Therefore, after cropping image 1121, color adjustment can be performed again. The cropped image 1131 in the third album interface 1130 can be an image obtained by cropping image 1121 and then adjusting its color.

[0106] Color adjustments can be achieved using neural network models.

[0107] Image processing using color adjustment models can produce color-adjusted images. However, color adjustment consumes significant processing resources. When electronic devices have limited processing resources, it takes a long time to produce the color-adjusted image, resulting in long waiting times and a poor user experience.

[0108] In order to ensure user satisfaction with the color-adjusted image while reducing the processing resource consumption of color adjustment, this application provides an image processing system and an image processing method.

[0109] In the image processing method provided in this application embodiment, in some cases, when it is detected that a user clicks the auto-crop icon 1122 on the second album interface 1120, or when it is detected that a user clicks the camera icon 1212 on the camera interface 1210, the electronic device can display... Figure 3 The description settings interface 1310 is shown.

[0110] The description settings interface 1310 includes an input box 1311 and an OK icon 1313. The user can enter descriptive text in the input box 1311. The description settings interface 1310 may also include multiple alternative texts 1312a to 1312l. The user can click on any of the alternative texts as the description text. Alternatively, if the electronic device detects that a user has clicked on an alternative text, it can fill in the clicked alternative text in the input box 1311 as the text in the description text.

[0111] Upon detecting that the user has clicked the OK icon 1313, the electronic device can adjust the color of the image according to the description text that requires color adjustment.

[0112] Figure 4 This is a structural schematic diagram of an image processing system provided in an embodiment of this application.

[0113] The image processing system 300 may include an object segmentation model 311, a color mapping model 330, a region processing module 340, an image patch processing module 350, and a color adjustment module 360. The object segmentation model 311 and the color mapping model 330 may be trained neural network models.

[0114] The object segmentation model 311 can be used to segment objects in a first image based on descriptive information to obtain at least one descriptive object region. The descriptive information is used to describe the descriptive object. The descriptive information can be understood as the information provided by the user to describe the descriptive object in the first image. Each descriptive object region can record one descriptive object. The descriptive object can be an object that the user considers important.

[0115] Object segmentation can be understood as detecting objects recorded in a first image and locating the objects and their boundaries in the image. In other words, by processing the first image and descriptive information using object segmentation model 311, at least one descriptive object region can be obtained. The shape of the descriptive object region can be regular or irregular. For example, the edges of the descriptive object region can be the edges of the descriptive objects recorded in the first image.

[0116] The descriptive information may be determined based on user input. For example, the descriptive information may include information input by the user. The objects recorded in each of the at least one descriptive object regions obtained by the object segmentation model 311 through processing the first image and the descriptive information are the descriptive objects described by the descriptive information.

[0117] Description information may include one or more of the following: description text, description area, and description point.

[0118] Descriptive text can be text used to describe the object being described. It can be user-selected preset text, such as descriptive statements from poetic Tang poems, Song lyrics, Yuan songs, modern poems, or prose. Alternatively, it can be a user's description of the object based on their preferences, such as keywords or statements obtained through keyword searches or editing.

[0119] Descriptive text can be used to describe the characteristics of the object itself. For example, descriptive text can include text describing gender, type of object, name, etc. Descriptive text may also be used to describe the characteristics of items carried by the object, such as hairstyle, clothing, color of clothes, and other items held. Descriptive text can also indirectly describe the object by describing other related objects. For example, descriptive text can describe the relative or absolute position of the object, or the relationship between the object and other objects. For instance, descriptive text could be "a child feeding a puppy," where the description of the relationship between the puppy and the child indicates that the "child" is the object of description.

[0120] The descriptive point can be a point located on the descriptive object in the first image.

[0121] The description region may overlap with all or part of the area of ​​the described object in the first image. For example, the area of ​​the portion of the described object within the description region may be larger than the area of ​​the portion of the described object outside the description region; that is, the described object is primarily located within the description region. For example, the described object may be entirely located within the description region of the first image.

[0122] The shape of the description area can be regular or irregular. When the description area has a preset regular shape, its position in the first image can be described by text. The preset regular shape can be a circle, ellipse, triangle, or rectangle, etc. When the preset shape is rectangular, the description area can be called a description box.

[0123] The description region can also be represented by a description mask. A mask can be used to occlude all or part of the image being processed, thereby controlling the area or process of image processing. The description mask can be represented as a multi-valued image or a binary matrix, etc.

[0124] A descriptive mask is used to represent the approximate location of a described object in a first image. For example, a descriptive mask may include multiple location points or at least one region. A mask, also called a reference mask, can be represented as an image. The shape of the region in the descriptive mask can be regular or irregular.

[0125] The multiple location points may include the click location where the user clicks on a descriptive object in the first image displayed on the electronic device, or multiple points at a distance less than or equal to the click location.

[0126] The at least one region may include a circle with the click location where the user clicks on the descriptive object in the first image displayed on the electronic device as the center and a preset distance as the radius. Alternatively, the at least one region may include the area containing points on both sides of a curve that the user slides on the descriptive object in the first image displayed on the electronic device, and the distance between these points and the curve is less than a preset distance.

[0127] The accuracy of describing the region represented by the mask and / or the region described, i.e. the magnitude of the difference between the region represented by the mask and / or the region described and the region where the target image is located in the first image, affects the accuracy of describing the object region.

[0128] To obtain a more accurate object region, the image processing system 300 may further include a preprocessing module. The preprocessing module is used to determine description region information based on user input information. The description region information may include description regions and / or description masks. In the description region information, the description mask may represent at least one description region. The description region represented by the description region information records the description object.

[0129] User input information can be determined based on user actions. The user can operate on the electronic device based on the position of the described object in the first image, thus allowing the electronic device to determine the user input information. User input information can represent one or more of the following: input point, input line, input area, etc.

[0130] An input point can represent the location where a user clicks on the first image displayed on an electronic device. An input line can be the trajectory of a user's finger sliding across the first image displayed on the electronic device. An input area can be the region selected by the user on the first image displayed on the electronic device.

[0131] When the user input information includes input points, the region represented by the description region information determined by the preprocessing module may include the input points.

[0132] When the user input includes an input line, the area represented by the description area information determined by the preprocessing module may overlap with the input line. For example, the ratio of the length of the portion of the input line located within the area represented by the description area information to the total length of the input line may be greater than or equal to a preset length ratio.

[0133] When the user input information includes an input area, the area represented by the description area information determined by the preprocessing module may overlap with the input area. For example, the ratio of the area of ​​the portion of the input area located within the area represented by the description area information to the area of ​​the input area may be greater than or equal to a preset area ratio.

[0134] The preprocessing module may include an object detection model. The object detection model can be used to detect objects in the first image. Object detection can be understood as detecting all objects in an image. That is, by processing the first image using the object detection model, at least one object recognition region can be obtained. The shape of the object recognition region can be regular. For example, the object recognition region may include the smallest region of the object recorded by that object recognition region. The object recognition region can be represented, for example, by a bounding box.

[0135] The object detection model, by processing the first image, can also obtain the category of the object recorded in each object recognition region. In other words, the object detection model can be used to generate bounding boxes for objects in an image and determine the category to which each bounding box represents the object. For example, the object detection model could be a recognizeanything model (RAM).

[0136] The preprocessing module determines a description region based on multiple object recognition regions output by the object detection model and user input information. The description region can be a region among the multiple object recognition regions output by the object detection model that meets preset conditions. These preset conditions can include at least one of the following: an input point, coincidence with an input line, and coincidence with an input region. For example, the preset conditions can include an input point, a length ratio of the portion of the input line located within the object recognition region to the length of the input line being greater than or equal to a preset length ratio, and an area ratio of the portion of the input region located within the object recognition region to the area of ​​the input region being greater than or equal to a preset area ratio.

[0137] The object segmentation model 311 is used to encode the first image and description information to achieve cross-modal information fusion of the image and description information, making it easier for users to select important objects.

[0138] The region processing module 341 can determine salient and non-salient regions based on at least one descriptive object region. The salient region includes the at least one descriptive object region. The non-salient region is the region in the first image other than the salient region. The salient region can also be called the region of interest. The non-salient region can also be called the region of non-interest.

[0139] Each object region can be a connected region. A salient region can be either connected or disconnected. In a connected region, any two points can be connected by a polyline that belongs entirely to that region.

[0140] The first image can be captured by a camera in an electronic device. The image processing system can be located within the electronic device. During the image acquisition process, the camera can obtain the depth of various objects in the scene.

[0141] The region processing module 341 can also acquire the depth corresponding to each descriptive object in the first image. The depth of each object represents the distance between the descriptive object and the electronic device. The region processing module 341 can also determine at least one target object region in at least one descriptive object region, wherein the target object recorded in the at least one target object region can be at least one descriptive object with the smallest depth or a depth less than a preset depth among the at least one descriptive objects. The region processing module 341 can also determine a salient region based on the at least one target object region. The salient region may include the at least one descriptive object region.

[0142] In other words, the region processing module 341 can select at least one target object with the smallest depth or a depth less than a preset depth from among the multiple descriptive objects recorded in at least one descriptive object region, and determine a significant region based on the target object region where the at least one target object is located.

[0143] The image patch processing module 350 is used to divide the first image into image patches based on salient and non-salient regions, and to determine the representative color of each image patch. The image patch data represents multiple image patches in the first image and the representative color of each image patch.

[0144] The salient and non-salient regions, as well as the first image, can be input into the image block processing module 350 respectively.

[0145] Alternatively, salient and non-salient regions, along with the first image, can be represented as image data. This image data can be used as input to the image block processing module 350. The image data can be represented using four RGBS channels.

[0146] The first image can be a color image, represented by a three-channel image. The three channels correspond to the red, green, and blue color channels. Each channel image can be understood as a grayscale image. The grayscale image of each channel represents the brightness of the color corresponding to that channel in the first image. For example, in the red channel image, areas of red objects in the first image will be displayed as brighter grayscale values, while areas of non-red objects will be relatively darker. In other words, the first image can be represented using RGB three channels.

[0147] Image data can be represented using four RGBS channels. In the four-channel image of the image data, three channels represent the first image, namely the channel images corresponding to the red, green, and blue color channels respectively, and the other channel represents the salient and non-salient regions in the first image. That is, in the RGBS four-channel image, the S channel can be understood as the saliency channel. The saliency channel image represents the salient regions and the non-salient regions outside the salient regions in the first image. For example, the pixel value of each pixel in the S channel image is used to indicate whether the pixel is located in a salient region. Pixels located in salient regions have a pixel value of 1, and pixels located in non-salient regions have a pixel value of 0.

[0148] The image block processing module 350 divides the first image into image blocks based on salient and non-salient regions, such that the size of the image block in the salient region is smaller than the size of the image block in the non-salient region.

[0149] For example, during the process of dividing the first image, the image block processing module 350 can divide salient regions according to a first preset size to obtain multiple image blocks; and can divide insignificant regions according to a second preset size to obtain multiple image blocks. The multiple image blocks may not overlap. The first preset size can be understood as the preset size corresponding to the salient regions, and the second preset size can be understood as the preset size corresponding to the insignificant regions.

[0150] Multiple image blocks obtained by dividing salient regions and multiple image blocks obtained by dividing non-salient regions can be combined to form a first image. The number of pixels identified by the first preset size can be one or more.

[0151] It should be understood that for salient or non-salient regions that are part of an image area, the shape of the image area edge may limit the filling of blocks of a preset size at the image boundary, resulting in some image blocks having a size smaller than the preset size corresponding to the image area.

[0152] In other words, the granularity of image patch division differs for salient and non-salient regions. Determining the representative color for image patches of different granularities can also be called quadlinear interpolation.

[0153] Four-dimensional linear interpolation is a method for estimating the values ​​of unknown data points in four-dimensional space. The three dimensions of the color space can be understood as the three dimensions of four-dimensional linear interpolation, and the data representing salient and non-salient regions can be understood as the other dimension of four-dimensional linear interpolation.

[0154] The representative color of an image patch can be determined based on the color of at least one pixel within the patch. The representative color of multiple colors can represent the central tendency of that data set. The central tendency can be the mean, median, or mode, etc.

[0155] In other words, the image patch processing module 350 can determine the data set trend of the color of at least one pixel in each image patch of the first image, and use it as the representative color of that image patch. The representative color of the image patch can be the average, median, mode, etc., of the color of at least one pixel in that image patch in the first image.

[0156] In a salient region, the number of pixels in each image block can be one or more.

[0157] When an image block contains only one pixel, the representative color of that image block is the color of the pixels included in that image block.

[0158] Color mapping model 330 is used to process the first image to obtain the parameters in the CCM and the weight parameters corresponding to the multiple sub-LUTs.

[0159] The color adjustment module 360 ​​is used to determine the target image based on image patch data, parameters in the CCM, multiple sub-LUTs, and the weight parameters corresponding to each sub-LUT. The target image can be understood as the image obtained after color adjustment of the first image.

[0160] When white balance is taken into account, the sum of the parameters in each row of the CCM is 1. The parameters in the CCM output by the color mapping model 330 can include two of the three parameters in each row of the CCM.

[0161] The color adjustment module 360 ​​can determine the CCM based on the parameters in the CCM, and then determine the intermediate color corresponding to the representative color of each image patch based on the CCM. In other words, after determining the CCM, the color adjustment module 360 ​​can multiply the CCM by the vector representing the representative color of the image patch to obtain the vector representing the intermediate color.

[0162] The color adjustment module 360 ​​can determine the color mapping relationship for each image region based on multiple sub-LUTs and the weight parameters corresponding to each sub-LUT. The image region can be a salient region or a non-salient region. Each color mapping relationship can be represented as at least one sub-LUT and the weight parameters corresponding to each sub-LUT. Each color mapping relationship can be understood as a LUT.

[0163] In multiple sub-LUTs, the multiple second color ranges in the second color range set corresponding to different sub-LUTs can be obtained by dividing the colors represented in the color space according to different granularities. The second color range set corresponding to each sub-LUT can be preset.

[0164] In the RGB color space, each color is composed of the intensities of red, green, and blue, with each color typically ranging from 0 to 255, totaling 256 levels. The RGB color space can represent colors using a three-dimensional Cartesian coordinate system. In the RGB color space, colors are distributed across a 256×256×256 cube. If offsets are set for each color component, 256×256×256 offsets need to be set, resulting in a large number of parameters to configure.

[0165] By dividing the cube into multiple smaller cubes, multiple second color ranges can be obtained. The same second mapping parameters can be set for each second color range, and each second mapping parameter can include the offset of each color component. This reduces the number of parameters that need to be set.

[0166] It should be understood that the division of the cube can be as uniform as possible. That is, the size of each second color range can be equal or approximately equal.

[0167] For example, the number of second color ranges in the second color range set corresponding to the multiple sub-LUTs can be 9×9×9, 17×17×17, 33×33×33, and 64×64×64, respectively.

[0168] Within each sub-LUT, the mapping parameters for each color within the same second color range are identical. The corresponding mapping parameters within the second color range can include offsets for multiple color components.

[0169] Mapping parameters in multiple sub-LUTs can be represented using the same color space. That is, the meanings of multiple color components in the mapping parameters of different sub-LUTs are the same.

[0170] The color mapping relationship corresponding to salient regions can be represented as multiple sub-LUTs and the weight corresponding to each sub-LUT. The color mapping relationship corresponding to non-salient regions can be represented as a predetermined number of sub-LUTs among the multiple sub-LUTs and the weight corresponding to the predetermined number of sub-LUTs. The predetermined number of sub-LUTs is the largest granularity sub-LUT among the multiple sub-LUTs.

[0171] The color adjustment module 360 ​​can determine the target color corresponding to the middle color of an image block based on the color mapping relationship of the image region where each image block is located.

[0172] The target color of the image patch can be represented as the sum of the intermediate color of the image patch and the target mapping parameter of the image patch. The target mapping parameter of the image patch can be represented as the result of weighted summing of the third mapping parameters corresponding to the second color range to which the intermediate color of the image patch belongs, within multiple sub-LUTs of the color mapping relationship corresponding to the image region where the image patch is located, according to the weight parameters corresponding to the sub-LUTs. The third mapping parameter corresponding to each second color range can be preset.

[0173] For example, for salient regions, the color mapping relationship can include sub-LUT 1 and sub-LUT 2, and the third mapping parameter corresponding to the second color range to which the intermediate color of the image patch belongs in sub-LUT 1 is a. 11 The third mapping parameter corresponding to the second color range to which the middle color of the image patch belongs in sub-LUT 2 is a. 21 If the weight parameters corresponding to sub-LUT 1 and sub-LUT 2 are w1 and w2 respectively, then the target mapping parameters of the image patch can be expressed as w1×a 11 +w2×a 21 .

[0174] The color mapping relationship corresponding to the salient region includes a more fine-grained sub-LUT, which makes the color adjustment of the salient region more precise.

[0175] The color mapping relationships differ between salient and non-salient regions. Different image regions within an image use different color mapping relationships, and each color mapping relationship can be understood as a LUT. Therefore, the color mapping relationships used for color adjustment of an image can be understood as a 4-dimensional (4D) LUT (4D LUT).

[0176] Figure 4 The image processing system 300 is illustrated using an example where the description information is based on user input. Adjusting the colors of the first image based on user input relies heavily on user interaction, resulting in a poor user experience. Figure 5Another image processing system 400 is shown, which reduces the reliance on user operation for color adjustment of the first image.

[0177] Figure 5 This is a schematic structural diagram of an image processing system provided in an embodiment of this application.

[0178] The image processing system 400 includes an object segmentation model 312, a saliency detection model 320, a color mapping model 330, a region processing module 342, an image patch processing module 350, and a color adjustment module 360. The object segmentation model 312, the saliency detection model 320, and the color mapping model 330 can all be trained neural network models.

[0179] The object segmentation model 312 can be used to process the first image to obtain at least one object region. Each object can be recorded as one object. Object segmentation can be understood as detecting the recorded objects in the first image, locating the objects in the image, and the boundaries of the objects. That is, by processing the first image using the object segmentation model 312, at least one object region can be obtained. The shape of the object region can be regular or irregular. For example, the edge of the object region can be the edge of the object recorded in the object region in the first image.

[0180] In addition to the first image, the input to the object segmentation model 312 may also include location information. That is, the object segmentation model 312 can be used to process the first image and the location information to obtain at least one object region. The location information represents multiple location points uniformly distributed in the first image. The number of these multiple location points can be greater than or equal to a preset number. The preset number can be a fixed value or the product of a preset density and the first image.

[0181] A uniform grid can be set in the first image. The grid can include multiple grid lines. These multiple grid lines can include multiple horizontal lines and multiple vertical lines. The distance between two adjacent horizontal lines can be equal. The location information represents multiple location points, which are the intersections of the multiple grid lines.

[0182] Location information can also be understood as a type of descriptive information. However, compared to... Figure 4 The descriptive information described in the previous example differs from that in this example. The object described by the location information is no longer a specific object in the first image, but rather all objects in the first image. In other words, the objects recorded in each object region can be understood as the objects described by the location information. The location information can be represented by descriptive points or by descriptive masks.

[0183] The object segmentation model 312 detects objects in the first image based on location information, thus avoiding the omission of objects. The image segmentation performed on the first image by the object segmentation model 312 based on location information can be understood as fine-grained object segmentation.

[0184] The saliency detection model 320 is used to perform saliency detection on the first image to obtain the initial salient region in the first image.

[0185] Saliency detection, also known as image attention prediction or visual saliency detection, refers to the process of using intelligent algorithms to simulate the characteristics of the human visual system, predict human visual fixation points and eye movements, and extract salient regions in an image. A salient region in an image can also be understood as a region of interest (ROI). Saliency detection is a key image analysis technique in the field of computer vision.

[0186] The saliency detection model 320 can be an inverse saliency pyramid reconstruction network (InSPyReNet) or a neural network model with other structures.

[0187] like Figure 7 As shown, the initial salient region output by the saliency detection model 320 may not have edges that represent the edges of an object. Parts of the object of interest to the user may lie outside this initial salient region. If priority is determined based on the initial salient region, it may result in lower-granularity color adjustments being made to parts of the object of interest to the user, leading to lower user satisfaction with the color-adjusted image.

[0188] The region processing module 342 is used to determine the salient region and the non-salient region outside the salient region based on the initial salient region and at least one object region.

[0189] A salient region may include at least one target object region that overlaps with the initial salient region within the object region.

[0190] For example, the ratio of the area of ​​the overlapping portion of the target object region and the initial salient region to the area of ​​the target object region can be greater than or equal to a preset area ratio. That is, if a portion of an object region exceeding a preset area ratio is located within the initial salient region, it can be determined that the salient region includes that object region. Objects within the target object region can be referred to as salient objects.

[0191] The region processing module 342 can be used to traverse at least one object region output by the object segmentation model 312, compare each object region with an initial salient region, and determine whether the object region is a target object region. Based on at least one target object region, a salient region can be determined, and the salient region includes the at least one target object region.

[0192] The initial salient region can be connected or disconnected. Each object region can be a connected region. Multiple object regions may or may not overlap.

[0193] Each target object region can be understood as a connected sub-region within a salient region.

[0194] When there are multiple object regions, regardless of whether there is overlap between the multiple object regions, it is possible to determine whether each object region is the target object region.

[0195] Based on an initial salient region and at least one object region, a salient region and non-salient regions outside the salient region are determined, ensuring that all objects of interest to the user are located within the salient region. Adjusting the colors of the first image based on the salient region and its components improves the user experience.

[0196] The processing result of the first image and region processing module 342 can be represented as image data. In the RGBS four channels of the image data, the channel image of the S channel can also be called a fine-grained saliency map.

[0197] The image processing system 400 processes the first image independently of user-input description information. The image processing system 400 performs fine-grained color adjustments on prominent objects in the first image, while performing coarser color adjustments on other areas.

[0198] The object segmentation model 311 in the image processing system 300 and the object segmentation model 312 in the image processing system 312 can both be the segment anything model (SAM), U-Net, or other deep neural network models.

[0199] SAM refers to a general image segmentation model that possesses general knowledge about objects, allowing it to generate masks for objects in an image based on descriptive information. This descriptive information can also be called cue information or prompts. Figure 6 Taking SAM as an example, the structure of the object segmentation model is explained. The target object region output by the object segmentation model 311 can also be understood as an object region.

[0200] Different object regions can be represented by different images. Alternatively, different object regions can be represented as regions of different colors within the same image. Or, object regions can be represented by data forms other than images. Each location data point represents the position of all pixels within the object region corresponding to that location data.

[0201] For example, in Figure 6 The first image shown records a walking person. Given the descriptive information as the text "person," the object segmentation model can output an object region that records the walking person.

[0202] When the descriptive information is location information, the object segmentation model is for... Figure 6 The first image output shown can have three object regions. In the first image, one object region records the objects as walking people, one object region records the objects as complete buildings, and one object region records the objects as buildings obscured by walking people.

[0203] The edges of an object region can be the edges of the objects recorded in that object region within the first image, and there may be no overlap between object regions. If there is occlusion between objects recorded in the first image, the object recorded in each object region can be understood as the unoccluded portion. For example, for an object region recording a building occluded by a walking person, the object recorded in that object region is the unoccluded portion of the building occluded by the walking person.

[0204] Object segmentation models can also perform fine-grained object segmentation. Objects recorded in some of the multiple object regions output by the object segmentation model can also be components of objects recorded in other object regions outside of those regions. For example, besides... Figure 6 Beyond the three object regions shown, the object segmentation model for Figure 6 The object area of ​​the first image output shown may also include an object area recording a person's hands, an object area recording a person's legs, etc.

[0205] In other words, the multiple object regions output by the object segmentation model may or may not overlap.

[0206] Figure 6 This is a schematic structural diagram of an object segmentation model provided in an embodiment of this application.

[0207] The object segmentation model 310 can be either the object segmentation model 311 or the object segmentation model 312 in the image processing system 300.

[0208] It may include a first image feature extraction model 410, a descriptive information feature extraction model 420, and a feature processing model 430.

[0209] The first image feature extraction model 410 is used to extract features from the first image to obtain image features. Image features can also be called image embeddings.

[0210] The descriptive information feature extraction model 420 is used to extract features from descriptive information to obtain descriptive information features.

[0211] The feature processing model 430 is used to process descriptive information features and image features to generate at least one object region.

[0212] The object segmentation model 310 can process data from multiple modalities, such as the first image and descriptive information, to determine the target object. The data type of the first image is an image. The data type of the descriptive information can include text, or data types used to represent one or more of the following: location points, masks, regions of preset shapes, etc.

[0213] When the object segmentation model 310 is SAM, the structures of the descriptive information feature extraction model 420 and the feature processing model 430 in the object segmentation model 310 can be found in [reference needed]. Figure 4 .

[0214] The descriptive information feature extraction model 420 may include a first information feature extraction model 421 and a second information feature extraction model 422.

[0215] The first information feature extraction model 421 can be used to process the description mask in the description information to obtain the first information features.

[0216] The second information feature extraction model 422 can be used to extract features from one or more pieces of information in the descriptive information, such as descriptive text, descriptive regions, and descriptive points, to obtain second information features. The second information feature extraction model 422 can be understood as an encoder.

[0217] When the descriptive information is descriptive text, the second information feature can be called text embedding.

[0218] The descriptive information features may include first information features and / or second information features.

[0219] The feature processing model 430 may include a fusion model 431 and a feature processing sub-model 432.

[0220] The fusion model 431 is used to fuse the first information features and image features to obtain fused features.

[0221] The methods for fusing the first information feature and the image feature can be either concatenating the first information feature and the image feature, or adding the first information feature and the image feature together.

[0222] The fusion method that adds the primary information features and image features can be achieved through linear fusion or nonlinear fusion. Linear fusion includes operations such as addition and averaging. Nonlinear fusion includes attention mechanisms, pyramid pooling, deconvolution, etc.

[0223] When the descriptive information includes a descriptive mask, the feature processing sub-model 432 is used to process the second information features and the fused features to obtain at least one object region.

[0224] In the absence of a description mask in the description information, the feature processing sub-model 432 is used to process the second information features to obtain at least one object region.

[0225] The first image feature extraction model 410 can also be called an image encoder. The feature extraction of the first image by the first image feature extraction model 410 can also be understood as the encoding of the first image.

[0226] The feature processing sub-model 432 can also be called the feature restoration model or mask decoder. The processing of the second information feature (or the second information feature and the fused feature) by the feature processing sub-model 432 can also be understood as the feature processing sub-model 432 decoding based on the second information feature (or the second information feature and the fused feature).

[0227] Object regions can be represented using masks. Each object region can be understood as the area in the first image where the object it represents is located. The edges of the object region can be understood as the edges of the object in the first image. That is, in the first image, there are no pixels outside the object region that correspond to the object it represents.

[0228] The first image feature extraction model 410, the first information feature extraction model 421, the second information feature extraction model 422, and the feature processing sub-model 432 can all be convolutional neural networks.

[0229] Figure 4 The image processing system 300 shown Figure 5 The image processing system 400 shown can be applied to Figure 8 The image processing method shown.

[0230] The following is combined Figure 8The image processing method provided in the embodiments of this application will be described in detail. The execution subject of the method provided in the embodiments of this application can be an electronic device or a software / hardware module in an electronic device capable of image processing. For ease of explanation, the following embodiments will use an electronic device as an example.

[0231] Figure 8 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. The image processing method includes steps S610 to S620.

[0232] Step S610: Determine multiple image regions of the first image and the priority of each image region. The multiple image regions constitute the first image, and there is no overlap between the multiple image regions.

[0233] The first image can be an image received from another electronic device, an image stored in the electronic device, or an image captured by the electronic device using a camera.

[0234] In some embodiments, a descriptive object region in the first image can be determined based on descriptive information. The object region records the descriptive object represented by the descriptive information. The edges of the descriptive object region are the edges of the descriptive objects recorded in the descriptive object region. The descriptive information is determined based on user input information.

[0235] In other words, descriptive information can be determined based on user input. By processing the first image and descriptive information using an object segmentation model, the descriptive object region containing the descriptive information can be obtained.

[0236] Based on the described object region, the priority of multiple image regions and each object region can be determined.

[0237] For example, different priorities can be set for the description object region and other regions outside the description object region. The description object region has a higher priority than other regions. That is, the description object region and other regions outside the description object region can be understood as different image regions.

[0238] Alternatively, among multiple descriptive object regions, the descriptive object region containing at least one descriptive object with the lowest depth or a depth less than a preset depth can be identified as the target object region. The target object region and other regions can be assigned different priorities. The target object region has a higher priority than other regions.

[0239] In the first image, the descriptive information can represent one or more descriptive objects. If there are multiple descriptive objects in the first image, there can also be multiple descriptive object regions. These multiple descriptive object regions can have the same priority.

[0240] Based on the area being described, salient and non-salient areas can also be determined. A salient area may include at least one area being described. Alternatively, a salient area may include at least one target object area. Areas other than salient areas are non-salient areas.

[0241] The descriptive information is determined based on user input. Determining image regions of different priorities based on the descriptive information representing the object also relies heavily on user input. Inaccurate user input can lead to inaccurate image region segmentation, resulting in lower user satisfaction with the target image obtained through color adjustment based on these regions.

[0242] In other embodiments, saliency detection can be performed on the first image to obtain multiple image regions and a priority corresponding to each image region.

[0243] Different priorities represent different levels of salience. The salience of an image refers to the ability of certain regions in an image to attract visual attention relative to other regions. Regions with higher priority have stronger visual appeal.

[0244] The finer the granularity of color adjustments, the higher the user's satisfaction with the resulting image. However, finer granularity also leads to greater resource consumption for color adjustments.

[0245] When users observe an image, their attention is primarily focused on areas of high salience. Applying finer-grained color adjustments to these areas ensures user satisfaction with the adjusted image. Conversely, using coarser-grained color adjustments to less salience reduces the processing resource consumption, achieving a balance between user satisfaction and resource usage.

[0246] Furthermore, the determination of image regions does not depend on user operations, reducing the reliance of the image processing process on user actions and improving the user experience.

[0247] In some embodiments, the detection results of saliency detection on the first image may include an initial salient region.

[0248] An initial salient region can be designated as a salient region, while other regions outside the initial salient region can be designated as insignificant regions. Different priorities can be assigned to salient and insignificant regions. The priority of salient regions can be higher than that of insignificant regions.

[0249] The first image may contain multiple objects. A user may be interested in one or more of these objects, and their attention will primarily focus on those objects. The initial salient region edges obtained through salientity detection may not actually be the edges of an object; parts of the object the user is interested in may lie outside the initial salient region. In other words, the parts of the object the user is interested in may have undergone coarse-grained color adjustments, resulting in lower user satisfaction with the color-adjusted target image.

[0250] To further improve user satisfaction with the target image obtained through color adjustment, object segmentation can be performed on the first image before step S620 to determine at least one object region in the first image. Different object regions correspond to different objects; that is, different objects can be recorded in different object regions. The edge of each object region is the edge of the object corresponding to that object region. The edge of an object can also be understood as the edge of that object in the first image.

[0251] A salient region can be determined based on an initial salient region and at least one object region. The salient region includes at least one target object region. The target object region is an object region within the at least one object region that overlaps with the initial salient region.

[0252] Based on the initial salient region obtained by salient detection of the first image and the multiple object regions obtained by object segmentation of the first image, the salient region is determined so that the object of interest to the user is completely located in the salient region, that is, the salient region is more accurate, thereby improving the user's satisfaction with the target image.

[0253] Since the edge of the initial salient region may not be the edge of an object, the area containing some objects that users do not care about may also overlap with the initial salient region.

[0254] To further reduce the processing resource consumption of color adjustment, any object region can be determined as the target object region if the overlapping area between any object region and the initial salient region is greater than or equal to the product of the preset area ratio and the area of ​​the any object region.

[0255] In other words, for a given object region, if the ratio between the area of ​​the portion of the object region located within the initial salient region and the area of ​​the object region itself is greater than or equal to a preset area ratio, then the object region can be designated as a target object region. A salient region can be determined based on at least one target object region among at least one object region; the salient region includes the at least one object region. The preset area ratio can be a preset value greater than 0 and less than 1. For example, the preset area ratio could be 0.8 or 0.9, etc.

[0256] Based on the relationship between the area ratio of the portion of the object region located in the initial salient region and the preset area ratio, it is determined whether the salient region includes the object region. This avoids object regions that only partially overlap with the initial salient region being used as target object regions, reducing the impact of the accuracy of the initial salient region on the accuracy of the salient region, improving the accuracy of the salient region, and reducing the processing resource consumption of color adjustment.

[0257] By processing the first image using an object segmentation model, at least one object region can be obtained.

[0258] To avoid missing objects recorded in the first image in the object segmentation results, location information can be determined based on the first image. The location information and the first image are then processed using an object segmentation model to obtain at least one object region. The location information represents multiple location points uniformly distributed in the first image, and the number of these multiple location points is greater than or equal to a preset number. Each of the at least one object regions records at least one object, including the object located at each location point represented by the location information in the first image.

[0259] During the object segmentation process of the first image, the location information and the first image are processed using an object segmentation model. The location information represents multiple location points evenly distributed in the first image, thereby avoiding the omission of objects in the first image from being outside the at least one object recorded in the at least one object region output by the object segmentation model. This ensures that the at least one object recorded in the at least one object region is all objects in the first image, making the at least one object region more accurate.

[0260] For example, when the area of ​​the first image is less than a preset area, the number of the plurality of location points can be a preset number. When the area of ​​the first image is greater than or equal to the preset area, the number of the plurality of location points can be the product of a preset density and the area of ​​the first image. The product of the preset density and the preset area can be greater than or equal to the preset number.

[0261] In other embodiments, saliency detection is performed on the first image, and the detection result may include multiple initial salient regions and non-salient regions, with different initial salient regions used for different degrees of saliency. The number of multiple initial salient regions may be greater than or equal to 2.

[0262] Multiple image regions can include multiple initial salient regions and non-salient regions. The priority of multiple initial salient regions is higher than that of non-salient regions, and the priority of each initial salient region is positively correlated with the salience of that initial salient region.

[0263] Alternatively, multiple salient regions can be determined based on at least one object region obtained by object segmentation of the first image and multiple initial salient regions, with different salient regions corresponding to different priorities. Each salient region may include at least one object region that overlaps with it. For example, each salient region may include object regions whose overlap ratio with the salient region is greater than or equal to a preset area ratio. The overlap ratio between the object region and the salient region represents the ratio of the area of ​​the object region within the salient region to the area of ​​the object region itself. Non-salient regions may be areas outside of these multiple salient regions.

[0264] The following explanation uses the case where there is only one significant region as an example.

[0265] Step S620: Adjust the color of each image region according to the priority of each image region to determine the target image. The granularity of the color adjustment for each image region is negatively correlated with the priority of each image region.

[0266] To adjust the colors of the first image, a color mapping model can be used to process the first image and obtain a color mapping relationship. Based on the color mapping relationship, the target color corresponding to the color in the first image can be determined, thus obtaining the target image.

[0267] The granularity of color adjustment for each image region can be represented by the size of the image blocks into which the image region is divided. During the color adjustment process of the first image, the first image can be divided into multiple image blocks. The color adjustment of the first image can then be performed on the color of each individual image block.

[0268] In step S620, each image region can be divided into multiple image blocks according to its priority. The size of each image block in an image region is negatively correlated with the priority of that image region.

[0269] The size of an image patch is negatively correlated with the priority of the image region pair it belongs to; in other words, the area of ​​an image patch is negatively correlated with the priority of the image region pair it belongs to. That is, the higher the priority of an image region, the smaller the size of the image patch.

[0270] For any multiple image blocks in an image region, the representative color of each image block in that image region can be determined.

[0271] Based on the representative color of each image block in a given image region, the color of each image block in the given image region is adjusted to determine the target image.

[0272] In the target image, each image patch can have the same color. That is, the target color of each image patch can be determined based on its representative color. The target image is then obtained by adjusting the color of each pixel in each color patch to match the target color of that image patch.

[0273] In the first image, each image block is color-adjusted based on its representative color. The size of each image block is negatively correlated with the priority of the image region in which it is located, thus allowing for fine-grained color adjustment of high-priority image regions and coarse-grained color adjustment of low-priority image regions.

[0274] The size of the image patch varies in image regions corresponding to different priorities. In other words, the color adjustment for the first image is non-uniform.

[0275] The granularity of color adjustment for each image region can also be expressed as the number of candidate colors in the color mapping relationship. During the color adjustment process of the first image, the adjusted color can be determined from multiple candidate colors.

[0276] By processing the first image using a color mapping model, multiple color mapping relationships can be determined, with different color mapping relationships corresponding to different priorities.

[0277] In some embodiments, each color mapping relationship can represent a correspondence between multiple first color ranges and multiple candidate colors. The number of first color ranges in the color mapping relationship corresponding to each priority can be positively correlated with that priority. The number of candidate colors in the color mapping relationship corresponding to each priority can be positively correlated with that priority.

[0278] For each color in an image region, the target color corresponding to the target color range of that region is determined based on the color mapping relationship. Adjusting the color in the image region to the target color yields the target image. The color mapping relationship corresponding to the image region can be a color mapping relationship with the same priority as the image region itself.

[0279] The color corresponding to an image region can be the color of a pixel within that region, or it can be a representative color of an image block within that region. If the color corresponding to an image region is the color of a pixel, that pixel's color can be adjusted to the target color. If the color corresponding to an image region is a representative color of an image block, the color of each pixel within that image block can be set to the target color.

[0280] In multiple color mapping relationships, the fewer the number of first color ranges, the larger the range of each first color range, the fewer the number of candidate colors, and the lower the similarity between multiple colors belonging to the same color range. Conversely, fewer candidate colors mean more initial colors will be adjusted to the same candidate color. In other words, the more candidate colors there are, the lower the accuracy of the adjusted target image's color; conversely, the more candidate colors there are, the higher the accuracy of the adjusted target image's color.

[0281] Higher-priority image regions have more candidate colors in their mapping relationships, allowing for more precise color adjustments, while lower-priority regions receive coarser color adjustments. Higher-priority regions are those that users pay more attention to in the first image; finer color adjustments to these regions result in higher user satisfaction with the adjusted target image. Coarser color adjustments to lower-priority regions reduce processing resource consumption. Furthermore, the fewer colors in lower-priority regions within the target image reduce storage resource consumption.

[0282] In other embodiments, each color mapping relationship can represent the relationship between multiple first color ranges and multiple first mapping parameters. In each color mapping relationship, the number of first color ranges can be positively correlated with the priority corresponding to that color mapping relationship. The size of the range interval of the first color ranges in each color mapping relationship can be negatively correlated with the priority corresponding to that color mapping relationship. Multiple first color ranges in any color mapping relationship cover the color ranges represented by the color space. Furthermore, there is no overlap between any two first color ranges in the same color mapping relationship.

[0283] For example, based on the first image, a first color mapping relationship corresponding to a first priority and a second color mapping relationship corresponding to a second priority can be determined through a color mapping model, where the second priority is higher than the first priority. The first color mapping relationship may include the mapping relationship between P first color ranges and the mapping parameters corresponding to each first color range, and the second color mapping relationship may include the mapping relationship between Q first color ranges and the mapping parameters corresponding to each second color range, where Q and P are both positive integers, and Q is greater than P.

[0284] A color space can be understood as representing the range of colors within that color space. For the RGB color space, the range of colors it represents can be understood as a 256×256×256 cube in a three-dimensional Cartesian coordinate system.

[0285] Based on the color of each image region, the priority of any given image region, and the multiple color mapping relationships, the color of each image block in the multiple image blocks within that given image region can be adjusted to determine the target image.

[0286] The color in an image region can be the color of a pixel within that region, or it can be a representative color of a block of images within that region. The color of a pixel is its pixel value.

[0287] By processing the first image using a color mapping model, multiple color mapping relationships are determined, making these multiple first mapping relationships compatible with the first image, thereby improving user satisfaction with the target image obtained through color adjustment.

[0288] The size of the first color range in each color mapping relationship is negatively correlated with the priority of that color mapping relationship. In other words, the number of first color ranges in each color mapping relationship is positively correlated with the priority of that color mapping relationship. Within each color mapping relationship, there is no overlap between different first color ranges.

[0289] Different first color ranges correspond to different first mapping parameters.

[0290] By setting a smaller range of the first color range in higher-priority color mapping relationships and a larger range of the first color range in lower-priority color mapping relationships, the color adjustment of higher-priority image areas is made more precise, while the color adjustment of lower-priority image areas is made coarser.

[0291] The following explanation uses the example of how each color mapping relationship can represent the correspondence between multiple first color ranges and multiple first mapping parameters.

[0292] For each image region, after determining multiple color mapping relationships, the target color mapping relationship corresponding to the image region can be determined from among the multiple color mapping relationships according to the priority of the image region.

[0293] The target color mapping relationship corresponding to the image region can be a color mapping relationship with the same priority as the image region.

[0294] Based on the target color mapping relationship, the target first mapping parameter corresponding to the target color range to which the color in the image region belongs can be determined. According to the target first mapping parameter, the target adjustment color corresponding to the color in the image region can be determined. The color in the image region is then adjusted to the target adjustment color, thereby obtaining the target image.

[0295] When the color in the image region is the same as the color of the pixels in the image region, for each pixel in the first image, the target adjustment color corresponding to the target first color range to which the pixel's color in the first image belongs can be determined according to the target color mapping relationship corresponding to the image region to which the pixel belongs. Adjusting the color of each pixel in the first image to its corresponding target adjustment color yields the target image.

[0296] If the color in the image region is the same as the color of the pixels in the image region, then based on the target color mapping relationship corresponding to that image region, determine the target adjustment color corresponding to the pixel's color, and adjust the pixel's color to the target adjustment color. By traversing all pixels in the first image, the target image can be obtained.

[0297] When the color in an image region is the representative color of an image block within that region, for each image block in the first image, the target adjustment color corresponding to the target first color range to which the representative color of that image block belongs can be determined based on the target color mapping relationship of the image region to which that image block belongs. Adjusting the color of each image block in the first image to the target adjustment color corresponding to that image block yields the target image.

[0298] Adjusting the color of each image block in the first image to the target adjustment color corresponding to that image block can be understood as setting the color of each pixel in that image block to the target adjustment color.

[0299] The output of a color mapping model can be used not only to determine color mapping relationships but also to determine color conformation calibration (CCM). CCM correction can be performed before color adjustments are made based on the color mapping relationships.

[0300] Based on the Color Coding Model (CCM), the intermediate color corresponding to the color of an image region can be determined. Then, based on the target color mapping relationship corresponding to the image region in multiple color mapping relationships, the target adjustment color corresponding to the intermediate color of the image region can be determined. Adjusting the colors of each image region to the target adjustment color yields the target image.

[0301] The first image is input into the color mapping model for processing. The output of the color mapping model may include the color mapping model (CCM), or the output of the color mapping model may include parameters from the CCM. The CCM can be determined based on the parameters from the CCM.

[0302] A color correction matrix can be used for coarse-grained color adjustment. Multiple color mapping relationships can be used for fine-grained color adjustment. Compared to a color mapping model that only outputs information for fine-grained color adjustment, a color mapping matrix outputs information for both coarse-grained and fine-grained color adjustments. This results in shorter processing time, lower resource consumption, and improved image processing efficiency.

[0303] The output of the color mapping model can represent the set of second mapping parameters corresponding to each of multiple sets of second color ranges. Each set of second color ranges is obtained by dividing the color range represented by the color space according to different granularities. Each set of second mapping parameters includes multiple second mapping parameters. Different second mapping parameters correspond to different second color ranges. That is, there is a one-to-one correspondence between the multiple second mapping parameters in each set of second mapping parameters and the multiple second color ranges in the set of second color ranges corresponding to that set of second mapping parameters.

[0304] Within each set of second color ranges, there is no overlap between any two distinct second color ranges.

[0305] It should be understood that different sets of second color ranges are obtained by dividing the color ranges represented by the same color space.

[0306] The color range represented by the color space is divided to obtain a second set of color ranges. This division can be uniform. Uniformity can also be understood as approximately uniformity.

[0307] Each set of second color ranges corresponds to a set of second mapping parameters, which can be understood as a sub-LUT. Each second mapping parameter can be understood as including the offsets used for color adjustment across multiple color components. Within the same second color range, the second mapping parameters used when adjusting the image's colors are the same, and are the second mapping parameters corresponding to that second color range.

[0308] In multiple sub-LUTs, the second mapping parameters corresponding to the second color range in the set of second color ranges with larger granularity are used to perform coarse-grained color adjustment on the image. After performing coarse-grained color adjustment on the image, fine-grained adjustment can be performed on the image based on the second mapping parameters corresponding to the second color range in the set of second color ranges with smaller granularity.

[0309] Therefore, a color mapping relationship can be represented by one or more sets of second color ranges with the largest granularity and the second mapping parameter corresponding to each second color range in the one or more sets of second color ranges with the largest granularity.

[0310] Following the order of granularity from largest to smallest, L sets of second color ranges can be selected from these multiple sets. Based on the second color ranges in these L sets and their corresponding second mapping parameters, the color mapping relationship can be determined. The value of L is positively correlated with the priority of the determined color mapping relationship, where L is a positive integer. That is, for color mapping relationships with higher priority, more sets of second color ranges are selected during the determination process.

[0311] In other words, for the first color mapping relationship corresponding to the first priority, N sets of second color ranges can be selected from multiple sets of second color ranges in descending order of granularity, where N is an integer greater than or equal to 1. Based on the second color ranges in these N sets and their corresponding second mapping parameters, the first color mapping relationship can be determined.

[0312] For the second color mapping relationship corresponding to the second priority, M sets of second color ranges can be selected from multiple sets of second color ranges in descending order of granularity. The second color mapping relationship can be determined based on the second color ranges in these M sets and their corresponding second mapping parameters. The second priority is higher than the first priority, and M is an integer greater than N.

[0313] Each color mapping relationship can include at least one sub-LUT. Each color mapping relationship can be understood as a LUT. When adjusting the color of an image according to a certain color mapping relationship, the L second mapping parameters corresponding to the color in the image are determined based on the second color range to which the color in the image belongs in the L sets of second color ranges of the color mapping relationship. The adjustment amount of the color in the image can be expressed as the sum of the L second mapping parameters. That is, the target adjusted color corresponding to the color in the image is the sum of the color in the image and the L second mapping parameters.

[0314] Based on multiple second color ranges in at least one set of second color ranges with the largest granularity, multiple first color ranges in multiple color mapping relationships are determined. Then, based on the second mapping parameters corresponding to each of the multiple second color ranges, a first mapping parameter corresponding to each first color range is determined, thereby establishing multiple color mapping relationships. Selecting a set of second color ranges from the multiple sets of second color ranges in descending order of granularity can be understood as selecting a set of second color ranges in descending order of adjustment fineness. Therefore, the more sets of second color ranges selected, the finer the color adjustment can be based on the color mapping relationships determined by the selected sets of second color ranges.

[0315] Compared to the color mapping model, which represents multiple first mapping parameters corresponding to multiple color mapping relationships, the color mapping model outputs information representing multiple second mapping parameters that correspond one-to-one with the second color ranges in multiple sets of second color ranges. The color adjustment model outputs less information, so the color adjustment model requires less computation and consumes less processing resources, thus improving the efficiency of image processing.

[0316] In some embodiments, the output of the color mapping model may include a set of second mapping parameters.

[0317] The total number of second mapping parameters in the multiple sets of second mapping parameters is large, and the time required to process the first image using the color mapping model is still long, and the processing resources consumed are still considerable.

[0318] In other embodiments, to reduce the resource consumption of image processing, shorten processing time, and improve processing efficiency, the output of the color mapping model may include multiple weight parameters. These multiple weight parameters can correspond one-to-one with multiple sets of second color ranges. That is, different weight parameters can correspond to different sub-LUTs.

[0319] In each sub-LUT, the third mapping parameter can be preset. Multiple second color ranges in each second color range set can correspond one-to-one with multiple preset third mapping parameters. The second mapping parameter corresponding to a second color range in each second color range set is the product of the third mapping parameter corresponding to the second color range and the weight parameter corresponding to the second color range set. Each third mapping parameter can include the offset of each color component in the color space. The color space can include multiple color components.

[0320] Following the order of granularity from largest to smallest, L sets of second color ranges can be selected from these multiple sets. Based on the second color ranges and their corresponding third mapping parameters in these L sets, as well as the L weight parameters corresponding to each of the L sets, the color mapping relationship can be determined. The value of L is positively correlated with the priority of the determined color mapping relationship, where L is a positive integer. That is, for color mapping relationships with higher priority, more sets of second color ranges can be selected.

[0321] In other words, following the order of granularity from largest to smallest, N sets of second color ranges can be selected from these multiple sets of second color ranges. Based on the second color ranges in these N sets and their corresponding third mapping parameters, as well as the N weight parameters corresponding to each of the N sets, the first color mapping relationship corresponding to the first priority can be determined. Following the order of granularity from largest to smallest, M sets of second color ranges can be selected from these multiple sets of second color ranges. Based on the second color ranges in these M sets and their corresponding third mapping parameters, as well as the M weight parameters corresponding to each of the M sets, the second color mapping relationship corresponding to the second priority can be determined. N is a positive integer, M is an integer greater than N, and the second priority is higher than the first priority.

[0322] In some embodiments, the color mapping relationship can be represented as at least one set of second color ranges selected in the process of determining the color mapping relationship and a second mapping parameter corresponding to the second color range in each second color set, or it can be represented as at least one set of second color ranges, a second mapping parameter corresponding to the second color range in each second color set, and a weight parameter corresponding to each set of second color ranges.

[0323] Taking the second color mapping relationship as an example, the second color mapping relationship can be represented as the M sets of second color ranges and the third mapping parameter corresponding to the second color range in each second color set. The second mapping parameter corresponding to the second color range is the product of the third mapping parameter corresponding to the second color range and the weight parameter corresponding to the set of second color ranges to which the second color range belongs. That is, the M second mapping parameters corresponding to each first color range in the M second color mapping relationships can all be used as the first mapping parameter in the second color mapping relationship. Alternatively, the M third mapping parameters corresponding to each first color range in the M second color mapping relationships and the weight parameter corresponding to the M second color mapping relationships can be used as the first mapping parameter.

[0324] When adjusting the color of an image region with the same priority as the second color mapping relationship, the m-th target third mapping parameter corresponding to the image region's color can be determined based on the m-th second color range within the m-th set of M second color ranges. Multiplying the m-th third target mapping parameter by the weight parameter corresponding to the m-th second color range set yields the m-th target second mapping parameter. Based on the M target second mapping parameters, the target adjustment color corresponding to the image region's color can be determined. The target adjustment color corresponding to the image region's color can be expressed as the sum of the image region's color and the M target second mapping parameters.

[0325] The color of an image region can be the color of a pixel in the first image, a representative color value of an image block in the image region, or the intermediate color of the image region.

[0326] In other embodiments, multiple first color ranges in the second color mapping relationship can be determined based on M sets of second color ranges. These multiple first color ranges in the second color mapping relationship are obtained by dividing the color space representation into color ranges based on the upper and lower limits of each second color range in the M sets of second color ranges.

[0327] Each first color range is located within a second color range in each set of second color ranges. Furthermore, other first color ranges adjacent to a certain first color range are located in different second color ranges than that first color range in at least one of the multiple sets of second color ranges.

[0328] like Figure 9 As shown, the M sets of second color ranges can be second color range set 901 and second color range set 902. For a certain color component, second color range set 901 can include two second color ranges. After normalizing the color component values, the upper and lower limits of the two second color ranges are 0, 0.5, and 1, respectively. The two third mapping parameters corresponding to second color range set 901 are a... 11 and a 12 The second color range set 902 can include three second color ranges. The upper and lower limits of two of the second color ranges are 0, 1 / 3, 2 / 3, and 1, respectively, where " / " represents a division sign. The three third mapping parameters corresponding to the second color range set 902 are a... 21 a 22 and a 23 .

[0329] Based on the second color range set 901 and the second color range set 902, it can be determined that the number of multiple first color ranges in the second color mapping relationship can be four.

[0330] Based on the weight parameter corresponding to each of the M second color range sets, and the third mapping parameter corresponding to the second color range to which each first color range belongs in each second color range set in the second color mapping relationship, the first mapping parameter corresponding to each first color range can be determined.

[0331] For the second color mapping relationship, the first mapping parameter corresponding to each first color range can be represented as the sum of M second mapping parameters. This allows us to establish a mapping relationship between each first color range and its corresponding first mapping parameter in the second mapping relationship.

[0332] When the output of the color mapping model includes multiple weight parameters, for the second color mapping relationship, the first mapping parameter corresponding to each first color range can be represented as the result of weighted summing of the third mapping parameters corresponding to the M second color ranges to which the first color range belongs, according to the weight parameters corresponding to the set of second color ranges to which each second color range belongs. During the weighted summation calculation, the weight of the third mapping parameter corresponding to each second color range is the weight parameter corresponding to the set of second color ranges to which that second color range belongs.

[0333] In other words, the product of the M third mapping parameters corresponding to each first color range in the second color mapping relationship and the weight parameters corresponding to the set of second color ranges in which the second color range corresponding to the third mapping parameter is located can all be used as the first mapping parameters.

[0334] like Figure 9 As shown, the weight parameter corresponding to the second color range set 901 is w1, and the weight parameter corresponding to the second color range set 902 is w2. The four first mapping parameters corresponding to the four first color ranges in the second color mapping relationship determined by the second color range sets 901 and 902 are w1×a. 11 +w2×a 21 w1×a 11 +w2×a 22 w1×a 12 +w2×a 22 w1×a 12 +w2×a 23 .

[0335] The greatest common divisor among the number of second color ranges in different sets of second color ranges can be less than a preset value.

[0336] If the sum of the number of second color ranges in the two color range sets remains unchanged, the smaller the greatest common divisor between the number of second color ranges in the two color range sets, the more first color ranges are determined based on the two color range sets.

[0337] Step S610 can be performed after receiving a color adjustment operation from the user. The user's color adjustment operation can be a photo-taking operation instructing the electronic device to take a picture, or a cropping operation instructing the electronic device to crop the image. The photo-taking operation can be the user clicking the album icon 1213 in the fourth photo-taking interface 1210. The cropping operation can be the user clicking the automatic cropping icon 1122 in the second album interface 1120.

[0338] Step S610 can also be performed after receiving the user's color adjustment operation and obtaining the description information. The electronic device can determine the description information based on the user's first operation. The first user operation may include one or more of the following: entering description text in the input box 1311, clicking on alternative text, or clicking on the OK icon 1312.

[0339] The first user operation may further include an operation whereby the user selects one or more of an input point, input line, or input region on the first image within a second interface including the first image. The descriptive information may include one or more of the input point, input line, or input region selected by the user in the first user operation. Alternatively, the descriptive information may also include a descriptive region determined based on one or more of the input point, input line, or input region. The descriptive region may include the object recognition region where the input point is located in the image to be processed. The descriptive information may also include an object recognition region that overlaps with the input line and / or input region.

[0340] For example, the user's color adjustment operation can also be a photo-taking operation instructing the electronic device to take a picture when the device's automatic photo cropping function is enabled. When the automatic photo cropping function is enabled, the electronic device can crop the latest image captured by the camera upon detecting a photo-taking operation. The cropped image can then be used as the first image.

[0341] In other words, the first image can be a cropped version of the second image. The second image can be an image received from other electronic devices, an image stored in the electronic device, or an image captured by the electronic device using a camera. The image displayed on the second interface can be either the first image or the second image.

[0342] Before step S610, the second image captured by the camera of the electronic device can be cropped to obtain the first image.

[0343] Color adjustments can be used to achieve white balance.

[0344] White balance is simply the balance of white. It can also be understood as restoring white objects to their true white color under any light source. To compensate for color casts that occur when shooting under specific light sources, the corresponding complementary color is enhanced, thus achieving white balance. By adjusting the colors and correcting the color temperature of the image captured by the camera, the colors of the subject can be restored, making the colors of images captured under different light sources similar to those seen by the human eye, thus achieving white balance.

[0345] White can be understood as a bright color without a specific hue. White contains all the color components of the visible spectrum, and these color components are roughly equal in intensity, thus producing the visual effect of white. In other words, white represents a balanced mixture of all the colors of the visible spectrum, resulting in a bright, colorless visual effect.

[0346] The second image can be an image that has not been adjusted for white balance.

[0347] Alternatively, the second image can be an image obtained by color adjustment of the image captured by the camera, an image that has already achieved white balance. In other words, the second image can be an image that has undergone white balance adjustment or an image with white balance settings.

[0348] The white balance achieved in the second image is relative to the first image as a whole. For a specific area of ​​the second image, white balance may not be achieved. Therefore, for the first image, which is a portion of the second image, color adjustments can be made again to obtain the target image, ensuring white balance and thus improving the user experience.

[0349] Through steps S610 to S620, the first image is divided into multiple image regions with different priorities. Fine-grained adjustments are made to high-priority regions, while coarse-grained adjustments are made to low-priority regions. This reduces the processing resource consumption of color adjustments and increases user satisfaction with the adjusted image.

[0350] Figure 8 The image processing method described can be based on one or more neural network models, including object segmentation models, saliency detection models, and color mapping models. The training process of the neural network model is explained below.

[0351] Figure 10 This is a schematic flowchart illustrating a training method for a neural network model. Figure 10 The method shown includes steps S710 to S730.

[0352] Step S710: Obtain training data, which includes training samples and sample labels.

[0353] Step S720: Based on the training samples, determine the training information through the initial model.

[0354] Step S730: Adjust the parameters of the initial model based on the differences between the training information and the sample labels.

[0355] The adjusted initial model is the neural network model obtained through training.

[0356] It should be understood that the amount of training data can be one or more. The difference between the training information and the sample labels can be represented as the loss value.

[0357] For saliency detection models, the training method for the neural network model can be called the saliency detection model training method. Training samples can include the first training image, sample labels can be labeled salient regions, and training information can be training salient regions. The initial model can be the initial saliency detection model. The adjusted initial model can also be the saliency detection model.

[0358] In step S720, the first training image is processed by the initial saliency detection model to obtain the training salient region.

[0359] For object segmentation models, the training method for the neural network model can be called the object segmentation model training method. Training samples can be training segmentation samples, which include a third training image. Sample labels can be at least one labeled object region in the third training image; training information can be called a training region. The initial model can be an initial object segmentation model, and the adjusted initial model can also be an object segmentation model. The edge of each labeled object region is the edge of the object recorded in that labeled object region in the third training image.

[0360] In step S720, the training samples are processed by the initial object segmentation model to obtain the training region.

[0361] Optionally, the training segmentation samples may include training location information, which represents at least one training location point in the third training image. The objects recorded in at least one labeled object region include objects located at the at least one training location point in the third training image. That is, at least one labeled object region includes objects located at the at least one training location point.

[0362] The object segmentation model can be a SAM (Segmentation Aspect Model). When the object segmentation model is SAM, its training can be divided into three stages. The first stage can be understood as... Figure 10 The method shown is used to achieve this.

[0363] In the first stage of training, multiple first training segmentation samples and a first label region corresponding to each first training segmentation sample can be obtained. Each first training segmentation sample includes a first training segmentation image and first training description information. The first training segmentation sample can be the training sample in step S710, and the first label region can be the label information in step S710.

[0364] The edges of the first label region are the edges of the objects recorded in the first label region in the first training segmentation image. For example, the objects in the first label region can be foreground objects in the first training segmentation image, objects located in a prominent position in the first training segmentation image, or objects with a large area in the first training segmentation image, etc. The objects in the first label region can be the main (or can be understood as prominent) objects in the first training segmentation image.

[0365] The first training description information may include one or more of the following: first training description text, first training description region, and first training description points. The shape of the first training description region may be a preset regular shape or other shapes. For example, a first training description region with a preset regular shape can be represented by a box, while a first training description region with other shapes can be represented by a mask. The number of first training description regions and first training description points in the first training description information can both be one or more.

[0366] The first training description information can be used to describe objects located in the first label region in the first training segmentation image. That is, the first label region corresponding to the first training segmentation sample is the region of the object described by the first training description information in the first training segmentation image of the first training segmentation sample. The number of first label regions corresponding to each first training segmentation sample can be one or more.

[0367] In step S720, the first training segmentation sample can be processed using the initial object segmentation model to obtain the first training region. In step S730, the parameters of the initial object segmentation model can be adjusted according to the difference between the first training region corresponding to each first training segmentation sample and the first label region corresponding to that first training segmentation sample, so as to obtain the first-stage adjusted object segmentation model.

[0368] The difference between the first training region corresponding to each first training segmentation sample and the first label region corresponding to that first training segmentation sample can be represented as the loss value.

[0369] The first stage of training can be understood as the manual assistance stage. By manually annotating the regions corresponding to the main objects described by the first training description information in the first training segmentation image (i.e., the first label regions), the object segmentation model adjusted in the first stage is able to segment the main objects in the image.

[0370] After the first phase of training, the second and third phases of training can be carried out.

[0371] In the second phase of training, multiple second training segmentation samples can be obtained. These second training segmentation samples include second training segmentation images and second training descriptive information. The second training segmentation images may be the same as or different from the first training segmentation images.

[0372] The second training description information is used to describe objects in the second training segmentation image. The second training description information may include one or more of the following: second training description text, second training description region, and second training description points. The shape of the second training description region may be a preset regular shape or other shapes.

[0373] The second training segmentation sample is processed using the object segmentation model adjusted in the first stage to obtain the second training region corresponding to the second training segmentation sample.

[0374] Manually label the regions in the second training segmentation image that correspond to objects described in the second training text, excluding those recorded in the second training region. The edges of the manually labeled regions are the edges of the objects. Based on the second training region and the manually labeled regions, determine the second label region. The second label region includes both the second training region and the manually labeled region.

[0375] In the second stage, the objects recorded in the manually labeled regions can be understood as secondary (or non-prominent) objects in the second training segmentation image.

[0376] Based on the difference between the second training region and the second label region corresponding to each second training segmentation sample, the parameters of the object segmentation model adjusted in the first stage are obtained to obtain the object segmentation model adjusted in the second stage. The second label region corresponding to each second training segmentation sample includes the second training region and the region manually labeled based on the second training region. The difference between the second training region and the second label region corresponding to each second training segmentation sample can be represented as the loss value.

[0377] The second phase of training can be understood as the semi-automatic phase.

[0378] In the third stage of training, multiple third training segmentation samples can be obtained. These third training segmentation samples include third training segmentation images and third training descriptive information. The third training segmentation images may be the same as or different from the third training segmentation images.

[0379] The third training description information is used to describe objects in the second training segmentation image. The third training description information may include one or more of the following: third training description text, third training description region, and third training description points. The shape of the third training description region may be a preset regular shape or other shapes.

[0380] The object segmentation model adjusted in the second stage is used to process each third training segmentation sample multiple times. Each processing yields the third training region corresponding to the third training segmentation sample under that processing.

[0381] After multiple processing steps, a stable region is selected from the third training region corresponding to the third training segmentation sample as the third label region corresponding to the third training segmentation sample.

[0382] Based on the difference between the third training region and the third label region corresponding to each third training segmentation sample, the parameters of the object segmentation model adjusted in the second stage are modified. The object segmentation model adjusted in the third stage is the final object segmentation model. The difference between the third training region and the third label region corresponding to each third training segmentation sample can be the loss value.

[0383] The third stage of training can be understood as the fully automated stage. The first and second stages can be considered pre-training stages, while the third stage is the training stage. This third stage allows for fine-tuning of the parameters of the object segmentation model adjusted in the second stage.

[0384] Training traditional neural network models requires a large amount of labeled data. However, training the object segmentation model through three stages reduces the amount of labeled data required, thus reducing manual labor costs.

[0385] For color mapping models, the training method for the neural network model can be called the color mapping model training method. Training samples can include second training images, and sample labels can include label images.

[0386] The label image can be obtained by manually adjusting the colors of the second training image.

[0387] The second training image and the label image can be images from the training dataset. The training dataset may include a large-scale portrait photo retouching dataset (such as PPR10K with human-region mask and group-level consistency). In some embodiments, the label image can have a preset style. Therefore, color adjustments performed using a color mapping model can impart that style to the adjusted image.

[0388] The initial model can be an initial color mapping model, and the adjusted initial model can also be a color mapping model.

[0389] In some embodiments, in step S720, the second training image is processed by the initial color mapping model to determine multiple training color mapping relationships corresponding to multiple priorities.

[0390] In these multiple training color mapping relationships, each color mapping relationship can represent the correspondence between multiple training first color ranges and multiple training candidate colors. The number of training first color ranges in the training color mapping relationship corresponding to each priority can be positively correlated with that priority. That is, the number of candidate colors in the color mapping relationship corresponding to each priority can be positively correlated with the priority of that image region.

[0391] Furthermore, the size of the training first color range corresponding to each priority can be negatively correlated with that priority. The training first color range in the training color mapping relationship corresponding to each priority can be preset.

[0392] Alternatively, each color mapping can represent a correspondence between multiple training first color ranges and multiple training first mapping parameters. The number of training first color ranges in the training color mapping corresponding to each priority can be positively correlated with that priority. In other words, the number of training first mapping parameters in the color mapping corresponding to each priority can be positively correlated with the priority of that image region.

[0393] By adjusting the colors of the second training image according to multiple training color mapping relationships, multiple adjusted training images can be obtained. The training information may include these multiple adjusted training images. That is, in step S730, the parameters of the initial color mapping model can be adjusted according to the differences between the multiple adjusted training images and the label image to obtain the color mapping model.

[0394] For example, the initial color mapping model may include an initial sub-model corresponding to each priority. The parameters of the initial color mapping model are adjusted based on the differences between multiple training adjusted images and label images. This adjustment can be achieved by adjusting the initial sub-model corresponding to a given priority based on the differences between the label image and the training adjusted images obtained by color-adjusting the training color mapping relationships corresponding to each priority of the second training image. The color mapping model may include a sub-model corresponding to each priority. The sub-model corresponding to each priority is the adjusted initial sub-model for that priority.

[0395] In other embodiments, an initial color mapping model processes the second training image, and based on the output of the initial color mapping model, multiple sets of trained second mapping parameters can be determined. These multiple sets of trained second mapping parameters correspond one-to-one with multiple sets of second color ranges. Each set of trained second mapping parameters includes multiple trained second mapping parameters.

[0396] In other words, based on the output of the initial color mapping model, multiple trained second mapping parameters can be determined that correspond one-to-one with the second color ranges in multiple sets of second color ranges.

[0397] Based on the set of training second mapping parameters corresponding to each second color range set, the training adjustment image corresponding to the second training image can be determined. For the color of each pixel in the second training image, based on the target second color range to which that color belongs in each second color range set, the training target second mapping parameters corresponding to each target second color range are determined. The training adjustment color corresponding to the color of that pixel can be represented as the sum of the color of that pixel and each training target second mapping parameter.

[0398] For example, a training color mapping relationship can be determined based on the training second mapping parameter set corresponding to each second color range set. Each training first color range in the training color mapping relationship can be determined based on the multiple second color range sets. The training first mapping parameter corresponding to each training first color range in the training color mapping relationship can be the sum of the second mapping parameters corresponding to the second color range to which the training first color range belongs in each second color range set.

[0399] Based on the training color mapping relationship, the second training image can be adjusted to obtain the training adjusted image.

[0400] Multiple training first color ranges can be obtained by dividing the color space representation range according to the upper and lower limits of each second color range in the multiple second color range sets.

[0401] The multiple training first color ranges used in the training process, and the inference process, i.e. Figure 8In the image processing method shown, multiple first color ranges in the highest priority color mapping relationship can be the same.

[0402] By adjusting the color of each pixel in the second training image to the corresponding training adjustment color, a training adjustment image can be obtained. Therefore, in step S730, based on the difference between the training adjustment image and the label image, the parameters of the initial color mapping model are adjusted to obtain the color mapping model.

[0403] The output of the initial color mapping model can include multiple trained second mapping parameters. Alternatively, the output of the initial color mapping model can include trained weight parameters corresponding to each of multiple second color range sets.

[0404] The training second mapping parameter corresponding to each second color range can be represented as the product of the preset third mapping parameter corresponding to the second color range and the weight parameter corresponding to the set of second color ranges to which the second color range belongs. In other words, the training first mapping parameter corresponding to each training first color range in the training color mapping relationship can be the sum of the products of the third mapping parameter corresponding to the second color range to which the training first color range belongs in each set of second color ranges and the weight parameter corresponding to the set of second color ranges.

[0405] The output of the initial color mapping model can also represent the training CCM. For example, the output of the initial color mapping model can include the training CCM parameters. The training CCM can be determined based on the training CCM parameters.

[0406] Based on the trained CCM, the training intermediate color corresponding to the color of each pixel in the second training image can be determined. Based on the trained color mapping relationship, the training adjustment color corresponding to each training intermediate color can be determined. Adjusting the color of each pixel in the second training image to its corresponding training adjustment color yields the training adjustment image.

[0407] It should be understood that Figure 8 and Figure 10 The methods shown can be performed by the same or different electronic devices.

[0408] For example, execute Figure 10 The electronic device used in the method shown can be a training device. The training device could be, for example, a server or the cloud. Execution Figure 8 The electronic device used in the method described can be called an execution device. An execution device can be a terminal, such as a mobile phone, tablet, laptop, AR / VR device, in-vehicle terminal, etc., or it can be a server or cloud-based device.

[0409] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or variations based on the above examples, and such modifications or variations also fall within the scope of the embodiments of this application.

[0410] The above text combined Figures 1 to 10 The image processing system and image processing method of the embodiments of this application are described in detail below. Figures 11 to 13 This document describes in detail the apparatus embodiments of this application. It should be understood that the image processing apparatus in the embodiments of this application can execute the image processing methods described in the foregoing embodiments of this application. That is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0411] Figure 11 A hardware system for an electronic device applicable to this application is shown.

[0412] Electronic device 100 can be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application embodiment does not limit the specific type of electronic device 100.

[0413] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0414] Figure 11 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0415] It should be noted that, Figure 11 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 11 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 11 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 11 Sub-components of some of the components shown. Figure 11 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0416] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices.

[0417] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0418] Electronic device 100 can implement display functions through GPU, display screen 194 and application processor.

[0419] A GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0420] The display screen 194 can be used to display images or videos. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0421] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0422] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits this electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0423] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard red-green-blue (RGB), YUV, or other image signal format. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0424] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0425] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0426] An NPU (Neural Processing Unit) is a processor that borrows from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, to rapidly process input information and continuously learn. NPUs can enable intelligent cognitive functions in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0427] The external storage interface 120 can be used to connect an external memory card, such as a secure digital (SD) card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0428] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (e.g., sound playback and image playback). The data storage area may store data created during the use of electronic device 100 (e.g., audio data and phonebook). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, and universal flash storage (UFS). Processor 110 executes various processing methods of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0429] Touch sensor 180K, also known as a touch device, can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a touch screen. Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, and in a different location from display screen 194.

[0430] Figure 11 The hardware system of the electronic device 100 has been described in detail. The software system of the electronic device 100 is described below. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes a layered architecture as an example to exemplarily describe the software system of the electronic device 100.

[0431] like Figure 12 As shown, a layered architecture software system is divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the Android Runtime and system libraries, and the kernel layer.

[0432] The application layer may include applications such as camera, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, and photo album. Applications such as camera and photo album can be used to execute the image processing methods provided in this application.

[0433] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer may include some predefined functions.

[0434] For example, the application framework layer includes a window manager, content providers, a view system, a phone manager, a resource manager, and a notification manager.

[0435] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0436] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0437] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0438] The system library can include multiple functional modules, such as: surface manager, media libraries, 3D graphics processing library, and 2D graphics engine.

[0439] The kernel layer is the layer between hardware and software. It can include driver modules such as display drivers, camera drivers, audio drivers, and sensor drivers.

[0440] It should be noted that the above is passed Figure 11 An example is given to illustrate the structure of an electronic device. Figure 12 An example of a software architecture diagram for an electronic device is provided; however, this application does not impose any limitations on it.

[0441] Figure 13 This is a schematic diagram of the image processing apparatus provided in the embodiments of this application.

[0442] The image processing apparatus 1000 includes a processing unit 1010.

[0443] In some embodiments, the image processing device 1000 may perform Figure 8 The image processing method shown. The image processing apparatus 1000 may include a processing unit 1010.

[0444] The processing unit 1010 is used to determine a plurality of image regions of a first image and the priority of each image region in the plurality of image regions, wherein the plurality of image regions constitute the first image and there is no overlap between the plurality of image regions.

[0445] The processing unit 1010 is further configured to perform color adjustment on each image region according to the priority corresponding to each image region in order to determine the target image, wherein the granularity of the color adjustment on each image region is negatively correlated with the priority of each image region.

[0446] Optionally, the processing unit 1010 is specifically configured to perform saliency detection on the first image to determine salient regions and non-salient regions other than the salient regions in the first image, wherein the salient regions have a higher priority than the non-salient regions, and the plurality of image regions include the salient regions and the non-salient regions.

[0447] Optionally, the processing unit 1010 is further configured to perform object segmentation on the first image to determine at least one object region in the first image, wherein different object regions correspond to different objects, and the edge of each object region is the edge of the object corresponding to the object region.

[0448] The processing unit 1010 is further configured to perform saliency detection on the first image to obtain an initial salient region in the first image.

[0449] The processing unit 1010 is further configured to determine the salient region based on the initial salient region and the at least one object region, wherein the salient region includes at least one target object region among the at least one object region, and each target object region overlaps with the initial salient region.

[0450] The processing unit 1010 is further configured to determine the region in the first image other than the salient region as the non-salient region.

[0451] Optionally, the processing unit 1010 is further configured to, for any one of the at least one object region, determine the overlap area between the arbitrary object region and the initial salient region.

[0452] The processing unit 1010 is further configured to determine the arbitrary object region as the target object region when the overlapping area between the arbitrary object region and the initial salient region is greater than or equal to the product of a preset area ratio and the area of ​​the arbitrary object region.

[0453] Optionally, the processing unit 1010 is further configured to determine location information based on the first image, wherein the location information represents a plurality of location points evenly distributed in the first image, and the number of the plurality of location points is greater than or equal to a preset number.

[0454] The processing unit 1010 is further configured to process the location information and the first image using an object segmentation model to determine the at least one object region, wherein the at least one object recorded in the at least one object region includes objects located at each of the plurality of location points in the first image.

[0455] Optionally, the processing unit 1010 is further configured to divide each image region into multiple image blocks according to the priority corresponding to each image region, wherein the size of the image block in each image region is negatively correlated with the priority corresponding to each image region.

[0456] The processing unit 1010 is further configured to, for any plurality of image blocks in an image region, determine the representative color of each image block in the plurality of image blocks in the arbitrary image region.

[0457] The processing unit 1010 is further configured to adjust the color of each image block in the multiple image blocks in the arbitrary image region according to the representative color of each image block in the multiple image blocks in the arbitrary image region, so as to determine the target image.

[0458] Optionally, the processing unit 1010 is further configured to, based on the first image, determine a plurality of color mapping relationships through a color mapping model, wherein each of the plurality of color mapping relationships corresponds to a different priority, each color mapping relationship represents the relationship between each of the plurality of first color ranges and the corresponding first mapping parameter, the number of first color ranges in each color mapping relationship is positively correlated with the priority of each color mapping relationship, and the plurality of first color ranges in any color mapping relationship cover the color range represented by the color space.

[0459] The processing unit 1010 is further configured to adjust the color of each image block in the multiple image blocks in the arbitrary image region based on the representative color of each image block in the multiple image blocks in the arbitrary image region, the priority of the arbitrary image region, and the multiple color mapping relationships, so as to determine the target image.

[0460] Optionally, the processing unit 1010 is further configured to determine a corresponding target color mapping relationship from the plurality of color mapping relationships based on the priority of any one of the image regions.

[0461] The processing unit 1010 is further configured to, based on the target color mapping relationship, determine the target first mapping parameter corresponding to the target first color range to which the representative color of any one of the multiple image blocks in the arbitrary image region belongs.

[0462] The processing unit 1010 is further configured to determine the target adjustment color corresponding to the representative color of any image block based on the target first mapping parameter.

[0463] The processing unit 1010 is further configured to adjust the color of any one of the image blocks to a determined target adjustment color corresponding to the any one of the image blocks.

[0464] Optionally, the color mapping model is also used to determine the color correction matrix.

[0465] The processing unit 1010 is further configured to determine, based on the color correction matrix, the intermediate color corresponding to the representative color of any one of the multiple image blocks in the arbitrary image region.

[0466] The processing unit 1010 is further configured to determine the corresponding target color mapping relationship from the plurality of color mapping relationships according to the priority of any one of the image regions.

[0467] The processing unit 1010 is further configured to determine, based on the target color mapping relationship, the target first mapping parameter corresponding to the target first color range to which the middle color of any image block belongs.

[0468] The processing unit 1010 is further configured to determine the target adjustment color corresponding to the intermediate color of any image block based on the target first mapping parameter.

[0469] The processing unit 1010 is further configured to adjust the color of any one of the image blocks to a determined target adjustment color corresponding to the any one of the image blocks.

[0470] Optionally, the processing unit 1010 is further configured to, based on the first image, determine a plurality of second mapping parameters through the color mapping model, wherein the plurality of second mapping parameters correspond one-to-one with the second color ranges in a plurality of second color range sets, and each of the plurality of second color range sets is obtained by dividing the color ranges represented by the color space according to different granularities.

[0471] The processing unit 1010 is further configured to select N second color range sets from the plurality of second color range sets in descending order of granularity, and determine a first color mapping relationship based on the second color ranges in the N second color range sets and the corresponding second mapping parameters, wherein the first color mapping relationship corresponds to a first priority, and N is an integer greater than or equal to 1.

[0472] The processing unit 1010 is further configured to select M second color range sets from the plurality of second color range sets in descending order of granularity, and determine a second color mapping relationship based on the second color ranges in the M second color range sets and the corresponding second mapping parameters. The second color mapping relationship corresponds to a second priority, and the second priority is higher than the first priority. M is an integer greater than N.

[0473] Optionally, the processing unit 1010 is further configured to determine multiple first color ranges in the second color mapping relationship based on the second color ranges in the M sets of second color ranges, and to determine M second mapping parameters corresponding to each first color range in the second color mapping relationship, wherein the multiple first color ranges in the second color mapping relationship are obtained by dividing the color range represented by the color space according to the upper limit and lower limit values ​​of each second color range in the M sets of second color ranges.

[0474] The processing unit 1010 is further configured to use the sum of the M second mapping parameters corresponding to each first color range in the second color mapping relationship as the first mapping parameter corresponding to each first color range in the second color mapping relationship.

[0475] Optionally, the processing unit 1010 is further configured to input the first image into the color mapping model for processing, and output multiple weight parameters, wherein the multiple weight parameters correspond one-to-one with multiple sets of second color ranges, and the multiple second color ranges in each set of second color ranges correspond one-to-one with multiple preset third mapping parameters, and the second mapping parameter corresponding to the second color range in each set of second color ranges is the product of the third mapping parameter corresponding to the second color range and the weight parameter corresponding to the second color range set.

[0476] Optionally, the color mapping model is obtained by adjusting the parameters of the initial color mapping model based on the difference between the training image and the label image. The initial training image is obtained by adjusting the color of the initial image according to the training color mapping relationship. The training color mapping relationship is determined based on multiple training second mapping parameters. The multiple training second mapping parameters are determined based on the initial image and through the initial color mapping model. The multiple training second mapping parameters correspond one-to-one with the second color range in the multiple sets of second color ranges.

[0477] Optionally, the image processing device is located in the electronic device. The processing unit 1010 is further configured to crop the second image captured by the camera of the electronic device to obtain the first image.

[0478] In other embodiments, the image processing device 1000 may perform... Figure 10 The neural network model training method is shown. For example, the image processing apparatus 1000 can execute a color mapping model training method. That is, the trained neural network model can be a color mapping model.

[0479] The processing unit 1010 is used to acquire training images and label images, wherein the label image is obtained by color adjustment of the training image.

[0480] The processing unit 1010 is used to determine multiple training second mapping parameters based on the training image and through an initial color mapping model. The multiple training second mapping parameters correspond one-to-one with the second color ranges in multiple sets of second color ranges. Each set of second color ranges is obtained by dividing the color ranges represented by the color space according to different granularities.

[0481] The processing unit 1010 is also used to adjust the color of the training image according to multiple sets of second color ranges and multiple training second mapping parameters to obtain a training adjusted image.

[0482] The processing unit 1010 is also used to adjust the parameters of the initial color mapping model based on the difference between the training adjustment image and the label image.

[0483] The adjusted initial color mapping model is the color mapping model. The color mapping model can be applied to... Figure 8 The image processing method shown.

[0484] Optionally, the processing unit 1010 is further configured to determine a training mapping relationship based on a plurality of second color range sets and a plurality of training second mapping parameters. The training mapping relationship represents the correspondence between a plurality of training color ranges and a plurality of training first mapping parameters. The plurality of training color ranges are obtained by dividing the color space representation into color ranges based on the upper and lower limits of each second color range in the plurality of second color range sets.

[0485] The processing unit 1010 is further configured to use the sum of the training second mapping parameters corresponding to each training color range in the plurality of second color range sets as the training first mapping parameter corresponding to the training color range.

[0486] The training second mapping parameters corresponding to each training color range in each set of second color ranges can be understood as the training second mapping parameters corresponding to the second color range to which the training color range belongs in the set of second color ranges.

[0487] Optionally, the processing unit 1010 is further configured to input the training image into an initial color mapping model for processing. The output of the initial color mapping model includes multiple weight parameters, each weight parameter corresponding to a set of multiple second color ranges. Each set of multiple second color ranges corresponds to a set of multiple preset third mapping parameters. The training second mapping parameter corresponding to a second color range in each set of second color ranges is the product of the third mapping parameter corresponding to that second color range and the weight parameter corresponding to that second color range set.

[0488] It should be noted that the image processing device 1000 described above is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, and there is no specific limitation on this.

[0489] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0490] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0491] This application also provides a chip, which includes a data interface and one or more processors. When the one or more processors execute instructions, they read instructions stored in memory through the data interface to implement the image processing method and / or neural network model training method described in the above method embodiments.

[0492] This application also provides an image processing system, including an execution device and a training device.

[0493] The one or more processors can be general-purpose processors or special-purpose processors. For example, the one or more processors can be central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0494] The chip can be used as a component of a terminal device or other electronic device. For example, the chip can be located in electronic device 100.

[0495] Processors and memory can be configured separately or integrated together. For example, processors and memory can be integrated onto a system-on-a-chip (SoC) in a terminal device. That is, the chip can also include memory.

[0496] The memory may store a program, which can be run by a processor to generate instructions, causing the processor to execute the image processing method and / or neural network model training method described in the above method embodiments according to the instructions.

[0497] Optionally, the memory may also store data. Optionally, the processor may also read data stored in the memory, which may be stored at the same memory address as the program, or the data may be stored at a different memory address than the program.

[0498] For example, the memory can be used to store related programs of the image processing method provided in the embodiments of this application, and the processor can be used to call the related programs of the image processing method stored in the memory to implement the image processing method of the embodiments of this application. For example, multiple image regions of a first image and the priority corresponding to each of the multiple image regions are determined, the multiple image regions constitute the first image, and there is no overlap between the multiple image regions; according to the priority corresponding to each image region, color adjustment is performed on each image region to determine the target image, wherein the granularity of the color adjustment of each image region is negatively correlated with the priority of each image region.

[0499] For example, the memory can be used to store related programs of the color mapping model training method provided in the embodiments of this application, and the processor can be used to call the related programs of the color mapping model training method stored in the memory to implement the color mapping model training method of the embodiments of this application. For example, a training image and a label image are acquired, where the label image is obtained by color adjustment of the training image; based on the training image, multiple training second mapping parameters are determined through an initial color mapping model, wherein the multiple training second mapping parameters correspond one-to-one with the second color ranges in multiple sets of second color ranges, and each set of second color ranges is obtained by dividing the color ranges represented by the color space according to different granularities; the training image is color-adjusted according to the multiple sets of second color ranges and the multiple training second mapping parameters to obtain a training adjusted image; the parameters of the initial color mapping model are adjusted according to the difference between the training adjusted image and the label image to obtain the color mapping model.

[0500] This chip can be installed in electronic devices.

[0501] This application also provides a computer program product that, when executed by a processor, implements the image processing method described in any of the method embodiments of this application.

[0502] The computer program product can be stored in memory. For example, it is a program that undergoes preprocessing, compilation, assembly, and linking processes to eventually be converted into an executable object file that can be executed by a processor.

[0503] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the image processing method described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0504] The computer-readable storage medium is, for example, memory. Memory can be volatile or non-volatile, or it can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0505] The embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0506] In the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance, or a specific order or sequence. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0507] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0508] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0509] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0510] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0511] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0512] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0513] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0514] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Multiple image regions of a first image are determined, along with the priority of each image region. These multiple image regions constitute the first image, and there is no overlap between them. Based on the priority of each image region, each image region is divided into multiple image blocks, and the size of each image block in each image region is negatively correlated with the priority of each image region. For any multiple image blocks in an image region, determine the representative color of each image block in the multiple image blocks in the image region; Based on the first image, multiple color mapping relationships are determined through a color mapping model. Each color mapping relationship corresponds to a different priority. Each color mapping relationship represents the relationship between each first color range and its corresponding first mapping parameter. The number of first color ranges in each color mapping relationship is positively correlated with the priority of each color mapping relationship. The multiple first color ranges in any color mapping relationship cover the color range represented by the color space. Based on the representative color of each image block in a plurality of image blocks within any given image region, the priority of the given image region, and the plurality of color mapping relationships, the color of each image block in a plurality of image blocks within the given image region is adjusted to determine the target image.

2. The method according to claim 1, characterized in that, The determination of multiple image regions of the first image and the priority of each image region includes: The first image is subjected to saliency detection to determine salient regions and non-salient regions other than the salient regions in the first image. The salient regions have a higher priority than the non-salient regions. The plurality of image regions include the salient regions and the non-salient regions.

3. The method according to claim 2, characterized in that, The method further includes: The first image is segmented to determine at least one object region in the first image. Different object regions correspond to different objects, and the edge of each object region is the edge of the object corresponding to the object region. The step of performing saliency detection on the first image to determine salient regions and non-salient regions other than the salient regions includes: Saliency detection is performed on the first image to obtain the initial salient regions in the first image; The salient region is determined based on the initial salient region and the at least one object region, wherein the salient region includes at least one target object region among the at least one object region, and each target object region overlaps with the initial salient region; The region in the first image other than the salient region is determined as the non-salient region.

4. The method according to claim 3, characterized in that, The step of determining the salient region based on the initial salient region and the at least one object region includes: For any one of the at least one object regions, determine the overlap area between the arbitrary object region and the initial salient region; If the overlapping area between any one object region and the initial salient region is greater than or equal to the product of a preset area ratio and the area of ​​any one object region, then the any one object region is determined as the target object region.

5. The method according to claim 3 or 4, characterized in that, The step of performing object segmentation on the first image to determine at least one object region in the first image includes: Location information is determined based on the first image, wherein the location information represents multiple location points evenly distributed in the first image, and the number of the multiple location points is greater than or equal to a preset number; The location information and the first image are processed by an object segmentation model to determine at least one object region, wherein at least one object recorded in the at least one object region includes objects located at each of the plurality of location points in the first image.

6. The method according to any one of claims 1-4, characterized in that, The adjustment of the color of each image block in the multiple image blocks of the arbitrary image region, based on the representative color of each image block in the multiple image blocks of the arbitrary image region, the priority of the arbitrary image region, and the multiple color mapping relationships, includes: Based on the priority of any one of the image regions, determine the corresponding target color mapping relationship from the plurality of color mapping relationships; Based on the target color mapping relationship, determine the target first mapping parameter corresponding to the target first color range to which the representative color of any one of the multiple image blocks in any one image region belongs; Based on the first target mapping parameters, determine the target adjustment color corresponding to the representative color of any image patch; The color of any one of the image blocks is adjusted to the target adjustment color corresponding to that image block.

7. The method according to any one of claims 1-4, characterized in that, The color mapping model is also used to determine the color correction matrix; The adjustment of the color of each image block in the multiple image blocks of the arbitrary image region, based on the representative color of each image block in the multiple image blocks of the arbitrary image region, the priority of the arbitrary image region, and the multiple color mapping relationships, includes: Based on the color correction matrix, determine the intermediate color corresponding to the representative color of any one of the multiple image blocks in any one image region; Based on the priority of any one of the image regions, determine the corresponding target color mapping relationship from the plurality of color mapping relationships; Based on the target color mapping relationship, determine the target first mapping parameters corresponding to the target first color range to which the middle color of any image block belongs; Based on the first target mapping parameters, determine the target adjustment color corresponding to the intermediate color of any image patch; The color of any one of the image blocks is adjusted to the target adjustment color corresponding to that image block.

8. The method according to any one of claims 1-4, characterized in that, Based on the first image, multiple color mapping relationships are determined through a color mapping model, including: Based on the first image, multiple second mapping parameters are determined through the color mapping model. Each of the multiple second mapping parameters corresponds one-to-one with a second color range in a set of multiple second color ranges. Each set of multiple second color ranges is obtained by dividing the color range represented by the color space according to different granularities. In descending order of granularity, N sets of second color ranges are selected from the plurality of sets of second color ranges. Based on the second color ranges in the N sets of second color ranges and the corresponding second mapping parameters, a first color mapping relationship is determined. The first color mapping relationship corresponds to a first priority. N is an integer greater than or equal to 1. In descending order of granularity, M second color range sets are selected from the plurality of second color range sets. Based on the second color ranges in the M second color range sets and the corresponding second mapping parameters, a second color mapping relationship is determined. The second color mapping relationship corresponds to a second priority, and the second priority is higher than the first priority. M is an integer greater than N.

9. The method according to claim 8, characterized in that, The step of determining the second color mapping relationship based on the second color ranges in the M sets of second color ranges and the corresponding second mapping parameters includes: Based on the second color range in the M sets of second color ranges, determine multiple first color ranges in the second color mapping relationship, and determine M second mapping parameters corresponding to each first color range in the second color mapping relationship. The multiple first color ranges in the second color mapping relationship are obtained by dividing the color range represented by the color space according to the upper limit and lower limit values ​​of each second color range in the M sets of second color ranges. The sum of the M second mapping parameters corresponding to each first color range in the second color mapping relationship is used as the first mapping parameter corresponding to each first color range in the second color mapping relationship.

10. The method according to claim 8, characterized in that, Based on the first image, multiple second mapping parameters are determined using the color mapping model, including: The first image is input into the color mapping model for processing. The color mapping model outputs multiple weight parameters, which correspond one-to-one with multiple sets of second color ranges. Each set of second color ranges corresponds one-to-one with multiple preset third mapping parameters. The second mapping parameter corresponding to the second color range in each set of second color ranges is the product of the third mapping parameter corresponding to the second color range and the weight parameter corresponding to the second color range.

11. The method according to claim 8, characterized in that, The color mapping model is obtained by adjusting the parameters of the initial color mapping model based on the difference between the training adjusted image and the label image. The training adjusted image is obtained by adjusting the color of the initial image according to the training color mapping relationship. The training color mapping relationship is determined based on multiple training second mapping parameters. The multiple training second mapping parameters are determined based on the initial image and through the initial color mapping model. The multiple training second mapping parameters correspond one-to-one with the second color range in the multiple second color range sets.

12. The method according to any one of claims 1-4, 9, 10, and 11, characterized in that, The method is applied to an electronic device, and the method further includes: The second image captured by the camera of the electronic device is cropped to obtain the first image.

13. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 12.

14. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 12.

Images