Image enhancement model training method and device, image enhancement method and device, equipment and medium

Abstract

The invention provides an image enhancement model training method and device, an image enhancement method and device, equipment and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining an image data pair containing a target image and a to-be-enhanced source image; processing the image data by adopting an image enhancement model to obtain a mapping coefficient of each pixel point in the source image; wherein the mapping coefficient is used for indicating a mapping relation of color parameters between corresponding pixel points in the source image and the target image; based on the mapping coefficient of each pixel point, performing mapping processing on the corresponding pixel point in the source image to obtain a mapping image; an image enhancement model is trained based on the mapped image and a tagged image associated with the image data pair. Therefore, the continuity and the consistency of the image quality can be kept, and the use experience of a user in a shooting scene is improved. Optionally, the image enhancement model can be independent of an ISP module and can be independently developed and tested, and the flexibility of the method is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image enhancement model training and image enhancement method, device, equipment and medium. Background Art

[0002] Electronic devices (including but not limited to smartphones, XR (Extended Reality) devices, in-vehicle systems, etc.) are often equipped with multiple cameras to meet users' imaging needs for objects in different locations. However, due to factors such as manufacturing process, cost, and environment, the imaging quality between different cameras often varies greatly. As consumers' demand for high-definition image quality and smooth switching continues to increase, image quality smoothness processing is often required in application scenarios such as multi-camera switching and multi-camera simultaneous operation.

[0003] In the related art, a series of correction modules in the ISP (Image Signal Processor), such as automatic exposure and automatic white balance, are used to adjust imaging parameters such as brightness and color balance so that images acquired by different cameras are visually consistent.

[0004] However, the above method is constrained by the correction module and the parameter adjustment space is limited, resulting in the inability to maintain the continuity of image quality when the camera is switched. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, this application proposes a training and image enhancement method, device, equipment and medium for an image enhancement model, so as to realize the use of an image enhancement model to maintain the continuity and consistency of image quality and improve the user experience in shooting scenarios. Optionally, the image enhancement model can be independent of the ISP module and can be independently developed and tested to enhance the flexibility of the method.

[0007] In one aspect, an embodiment of the present application provides a method for training an image enhancement model, including:

[0008] Acquire an image data pair; wherein the image data pair includes a target image and a source image to be enhanced;

[0009] The image data pair is processed using an image enhancement model to obtain a mapping coefficient of each pixel in the source image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the source image and the target image;

[0010] Based on the mapping coefficients of the pixels, mapping is performed on the corresponding pixels in the source image to obtain a mapping image;

[0011] The image enhancement model is trained based on the mapped image and the annotated image associated with the image data pair.

[0012] Another aspect of the present application provides an image enhancement method, including:

[0013] In response to a first camera in the electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously, acquiring a first image captured by the first camera and a second image captured by the second camera;

[0014] Processing the first image and the second image using an image enhancement model to obtain a mapping coefficient of each pixel in the second image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the first image and the second image;

[0015] The mapping coefficients of the pixels are used to perform mapping processing on the corresponding pixels in the second image to obtain an enhanced image.

[0016] In another aspect, the present application provides a training device for an image enhancement model, comprising:

[0017] An acquisition module, used for acquiring an image data pair; wherein the image data pair includes a target image and a source image to be enhanced;

[0018] A first processing module, configured to process the image data pair using an image enhancement model to obtain a mapping coefficient of each pixel in the source image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the source image and the target image;

[0019] A second processing module, configured to perform mapping processing on corresponding pixel points in the source image based on the mapping coefficients of the pixel points to obtain a mapped image;

[0020] A training module is used to train the image enhancement model based on the mapped image and the labeled image associated with the image data pair.

[0021] In another aspect, an embodiment of the present application provides an image enhancement device, comprising:

[0022] An acquisition module, configured to acquire a first image captured by the first camera and a second image captured by the second camera in response to a switch from a first camera to a second camera in the electronic device, or a simultaneous shooting by the first camera and the second camera;

[0023] a first processing module, configured to process the first image and the second image using an image enhancement model to obtain a mapping coefficient of each pixel in the second image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the first image and the second image;

[0024] The second processing module is used to use the mapping coefficients of the pixels to perform mapping processing on the corresponding pixels in the second image to obtain an enhanced image.

[0025] In another aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the image enhancement model as described in the aforementioned aspect is implemented, or the image enhancement method as described in the aforementioned other aspect is implemented.

[0026] In another aspect of the present application, an embodiment provides a chip, which includes an interface circuit and a processing circuit coupled to each other, wherein the interface circuit is used to input or output a signal, and the processing circuit is configured to execute the training method of the image enhancement model as described in the aforementioned aspect, or to execute the image enhancement method as described in the aforementioned other aspect.

[0027] In yet another aspect of the present application, an embodiment provides a non-temporary computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the training method of the image enhancement model as described in the aforementioned aspect is implemented, or, when executed, the image enhancement method as described in the aforementioned other aspect is implemented.

[0028] Another aspect of the present application provides a computer program product having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the training method of the image enhancement model as described in the aforementioned aspect, or, when executed, implements the image enhancement method as described in the aforementioned aspect.

[0029] The training of the image enhancement model and the image enhancement method, device, equipment and medium proposed in the present application are based on deep learning technology to predict the mapping coefficient of each pixel in the source image to be enhanced, and based on the mapping coefficient, adjust each pixel in the source image to obtain the adjusted source image (referred to as the mapping image in this application), so that the image enhancement model is trained based on the difference (such as color difference) between the mapping image and the annotated image (consistent with the picture content of the source image and consistent with the color parameters of the target image), so that the model can learn to obtain the actual mapping relationship of the color parameters between the source image and the target image, so that the source image is adjusted based on the mapping relationship, which can ensure that the adjusted source image and the target image are consistent in color parameters, and then the trained image enhancement model is applied to multi-camera switching scenes or multi-camera shooting scenes, which can maintain the continuity and consistency of image quality and improve the user experience in the shooting scene. Optionally, the image enhancement model can be independent of the ISP module and can be independently developed and tested to enhance the flexibility of the method.

[0030] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A flowchart of a method for training an image enhancement model provided in an embodiment of the present application;

[0033] Figure 2 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application;

[0034] Figure 3 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application;

[0035] Figure 4 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application;

[0036] Figure 5 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application;

[0037] Figure 6 This is a schematic diagram of the implementation principle of the embodiment of the present application;

[0038] Figure 7A schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of a training device for an image enhancement model provided in an embodiment of the present application;

[0040] Fig. 9 A schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application;

[0041] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0042] Fig.11 A schematic diagram of the structure of a chip proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0043] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] For multi-camera systems, since the hardware, field of view, ISP algorithm, image adjustment, etc. of each camera are independent of each other, the image quality (such as color, brightness and other color parameters) output by different cameras will be different. During the zoom process of camera switching, there will be a sudden change in image quality, affecting the user's subjective experience. Therefore, how to achieve consistency in color parameters between multiple cameras is an important issue.

[0045] Among them, the multi-camera system refers to installing multiple cameras on an electronic device to obtain more image information. This system has been widely used in smart phones, driverless vehicles, security monitoring and other fields. However, due to differences in hardware characteristics, installation locations, lighting conditions and other factors of each camera, the images collected by different cameras may have consistency issues in color parameters (such as brightness and color), resulting in the image quality not being smooth enough in application scenarios such as multi-camera switching and multi-camera simultaneous operation, which also brings troubles to the subsequent processing and application of the image. To address this problem, comprehensive optimization can be carried out from multiple levels such as hardware level, algorithm, and standardization.

[0046] In related technologies, most studies rely on a series of correction modules in ISP, such as automatic exposure and automatic white balance, to make images acquired by different cameras visually consistent by adjusting imaging parameters such as brightness and color balance. However, such methods are constrained by the correction modules, and the parameter adjustment space is very limited, resulting in poor flexibility.

[0047] Some studies have also used adaptive algorithms based on image statistics, such as histogram matching algorithms, to directly establish the mapping relationship between brightness and color between images. However, the traditional image processing techniques used in these studies are often sensitive to changes in factors such as camera hardware characteristics and lighting conditions, and are difficult to apply to a variety of scenarios.

[0048] In summary, the related technology has at least the following shortcomings or deficiencies:

[0049] 1. Poor independence and flexibility: Most related technologies rely on the correction module in the ISP, which is constrained by the correction module and cannot be developed and tested independently of other modules;

[0050] 2. Poor generalization: Most related technologies use some ISP algorithms with few adjustable parameters. They are limited by the complexity of the model and are difficult to adapt to complex data distribution. They have limited generalization ability and can only be used in specific scenarios.

[0051] 3. Poor robustness: The image processing technology of related technologies is often sensitive to noise and outliers in image data, and cannot handle the color consistency problem under complex environmental conditions well;

[0052] 4. Reliance on manual calibration: The deep learning methods of related technologies require a large amount of labeled data, that is, pixel-level annotation is required, which has high annotation costs and low annotation efficiency.

[0053] In order to solve at least one of the above-mentioned problems, the present application mainly proposes an image enhancement model training and image enhancement method, device, equipment and medium.

[0054] The following describes the image enhancement model training and image enhancement methods, devices, equipment and media of the embodiments of the present application with reference to the accompanying drawings.

[0055] Figure 1 A flowchart of a method for training an image enhancement model provided in an embodiment of the present application.

[0056] The embodiment of the present application takes the training method of the image enhancement model as an example, which is configured in the training device of the image enhancement model. The training device of the image enhancement model can be applied to any electronic device so that the electronic device can perform the training function of the image enhancement model. The electronic device can be any device with computing capabilities, such as a server, an XR device, a mobile terminal, an in-vehicle device, a vehicle, etc. The mobile terminal can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens, and / or display screens.

[0057] like Figure 1As shown, the training method of the image enhancement model may include the following steps S101 to S104:

[0058] Step S101, obtaining an image data pair; wherein the image data pair includes a target image and a source image to be enhanced.

[0059] Among them, the source image refers to the image to be enhanced, that is, the image whose color parameters are to be adjusted, that is, the source image is enhanced based on the color parameters of the target image (including but not limited to color, brightness, etc.) so that the color parameters of the processed source image are consistent with the color parameters of the target image.

[0060] In the embodiment of the present application, there is no restriction on the way of acquiring the image data pair. For example, the source image and the target image in the image data pair can be acquired from the training set or the test set. Alternatively, the source image and the target image in the image data pair can be acquired by different cameras in an electronic device equipped with multiple cameras. For example, when the first camera in the electronic device is switched to the second camera, the image acquired by the first camera before the camera switching can be used as the target image, and the image acquired by the second camera after the camera switching can be used as the source image; for another example, when the first camera and the second camera in the electronic device are shooting at the same time, the image acquired by one of the cameras can be used as the source image, and the image acquired by the other camera can be used as the target image. Alternatively, the source image and the target image in the image data pair can be generated based on a sample image. Alternatively, the source image and the target image in the image data pair can be artificially synthesized, and so on. The embodiment of the present application does not limit this.

[0061] Step S102, using an image enhancement model to process the image data pair to obtain a mapping coefficient of each pixel in the source image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the source image and the target image.

[0062] There is no restriction on the model structure of the image enhancement model. For example, the image enhancement model may be a generative model, or may be an HDRNet (High Dynamic Range Network) model, etc.

[0063] The color parameters include but are not limited to color, brightness and other parameters.

[0064] The mapping coefficient of each pixel point may be a multi-dimensional coefficient. For example, the mapping coefficient may be a 12-dimensional coefficient.

[0065] In an embodiment of the present application, an image enhancement model can be used to process image data pairs to obtain mapping coefficients for each pixel in the source image; wherein the mapping coefficient for each pixel is used to indicate the mapping relationship between the color parameters of the pixel in the source image and the corresponding pixel in the target image.

[0066] Exemplarily, the target image and the source image in the image data pair may be spliced ​​to obtain a spliced ​​image, and the spliced ​​image may be input into the image enhancement model for processing to obtain the mapping coefficient of each pixel in the source image.

[0067] Step S103 , mapping the corresponding pixel points in the source image based on the mapping coefficients of the pixel points to obtain a mapped image.

[0068] In the embodiment of the present application, mapping processing can be performed on corresponding pixel points in the source image according to the mapping coefficient of each pixel point in the source image to obtain a mapped image.

[0069] As an example, taking the mapping coefficient as a 12-dimensional coefficient, the following formula can be used to generate a mapping image:

[0070]

[0071] in, represents the pixel in the source image, represents the corresponding pixel in the mapped image, It can be expressed as a 9-dimensional linear transformation matrix of the 12-dimensional mapping coefficients corresponding to the pixel point. It can represent the 3-dimensional offset matrix of the 12-dimensional mapping coefficients corresponding to the pixel points.

[0072] Step S104: training the image enhancement model based on the mapped image and the annotated image associated with the image data pair.

[0073] The picture content of the annotated image is consistent with the picture content of the source image, and the color parameters of the annotated image match or are consistent with the color parameters of the target image.

[0074] As an example, a labeled image may be obtained by performing a color parameter transformation operation on a source image.

[0075] As another example, the annotated image may include a target image. Exemplarily, when the target image and the source image have the same picture content or shooting angle, the target image may be used as the annotated image.

[0076] As a possible implementation method, the image enhancement model can be trained based on the color difference between the mapped image and the annotated image.

[0077] The training method of the image enhancement model of the embodiment of the present application is based on deep learning technology to predict the mapping coefficient of each pixel in the source image to be enhanced, and based on the mapping coefficient, adjust each pixel in the source image to obtain the adjusted source image (referred to as the mapping image in this application), so that the image enhancement model is trained based on the difference (such as color difference) between the mapping image and the annotated image (consistent with the picture content of the source image and consistent with the color parameters of the target image), so that the model can learn to obtain the actual mapping relationship of the color parameters between the source image and the target image, so that based on the mapping relationship, the source image is adjusted to ensure that the adjusted source image and the target image are consistent in color parameters, and then the trained image enhancement model is applied to multi-camera switching scenes or multi-camera shooting scenes, which can maintain the continuity and consistency of image quality and improve the user experience in the shooting scene. Optionally, the image enhancement model can be independent of the ISP module and can be independently developed and tested to enhance the flexibility of the method.

[0078] The present application embodiment provides another training method for an image enhancement model. Figure 2 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application.

[0079] It should be noted that the training method of the image enhancement model can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0080] like Figure 2 As shown, the training method of the image enhancement model may include the following steps S201 to S206:

[0081] Step S201, obtaining an image data pair; wherein the image data pair includes a target image and a source image to be enhanced.

[0082] For the explanation of step S201, please refer to the relevant description in any embodiment of the present application, which will not be repeated here.

[0083] Step S202 , down-sampling the source image and the target image respectively to obtain a first sampled image corresponding to the source image and a second sampled image corresponding to the target image.

[0084] In the embodiment of the present application, the source image may be downsampled to obtain a first sampled image with a low resolution. Similarly, the target image may be downsampled to obtain a second sampled image with a low resolution.

[0085] Step S203: splicing the first sampled image and the second sampled image to obtain a spliced ​​image.

[0086] As an example, the number of color channels of the first sampling image and the second sampling image are both 3, namely, the three color channels of R (Red), G (Green), and B (Blue). The first sampling image and the second sampling image can be spliced ​​along the color channel dimension to obtain a spliced ​​image.

[0087] Step S204: Process the spliced ​​image using an image enhancement model to obtain a mapping coefficient for each pixel in the source image.

[0088] The mapping coefficient is used to indicate the mapping relationship between the color parameters of corresponding pixels in the source image and the target image.

[0089] In an embodiment of the present application, the spliced ​​image may be input into an image enhancement model for processing to obtain a mapping coefficient of each pixel in the source image.

[0090] In any embodiment of the present application, the mapping coefficient of each pixel in the source image can be obtained by following steps A to C:

[0091] Step A: The feature extraction network in the image enhancement model is used to extract global features and local features from the spliced ​​image.

[0092] Among them, global features reflect the overall properties of the stitched image, such as color distribution, shape contour, etc., which are usually invariant to the scale and rotation of the stitched image.

[0093] Among them, local features focus on the characteristics of specific key points or regions in the stitched image, such as corners or edges. These features help to identify specific objects and details in the stitched image.

[0094] As an example, the feature extraction network may include two feature extraction branches, namely a global feature extraction branch and a local feature extraction branch. In the present application, the global feature extraction branch in the feature extraction network may be used to perform feature extraction on the stitched image to obtain global features, and the local feature extraction branch in the feature extraction network may be used to perform feature extraction on the stitched image to obtain local features.

[0095] As another example, the feature extraction network may include three feature extraction branches, namely a first extraction branch, a second extraction branch and a third extraction branch. In the present application, first, the first extraction branch in the feature extraction network may be used to perform feature extraction on the spliced ​​image to obtain basic features. After that, the second extraction branch in the feature extraction network may be used to perform feature extraction on the basic features to obtain global features, and the third extraction branch in the feature extraction network may be used to perform feature extraction on the basic features to obtain local features.

[0096] Step B: Use feature fusion algorithm to fuse global features and local features to obtain fused features.

[0097] Step C: Use the prediction network in the image enhancement model to process the fusion features to obtain the coefficient matrix output by the prediction network; wherein the elements in the i-th and j-th columns in the coefficient matrix are used to indicate the mapping coefficients of the pixels in the i-th row and j-th column in the source image, and i and j are both positive integers.

[0098] In summary, global features can capture the overall structure and contextual information of the image, and local features can focus on the number of details and texture in the image. Combining global features and local features can enable the image enhancement model to better cope with occlusion, perspective changes and other interference factors in the image. Even if some parts are occluded, global features can still help the model understand the main content of the image and improve the robustness of the model.

[0099] Step S205 , mapping the corresponding pixel points in the source image based on the mapping coefficient of each pixel point to obtain a mapped image.

[0100] Step S206: training the image enhancement model based on the mapped image and the annotated image associated with the image data pair.

[0101] The explanation of steps S205 to S206 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0102] The training method of the image enhancement model of the embodiment of the present application downsamples the source image and the target image to obtain a low-resolution first sampling image and a second sampling image, splices the first sampling image and the second sampling image along the color channel dimension, and then inputs them into the image enhancement model for processing, which can take into account the prediction speed and prediction accuracy of the image enhancement model.

[0103] The present application embodiment provides another training method for an image enhancement model. Figure 3 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application.

[0104] It should be noted that the training method of the image enhancement model can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0105] like Figure 3 As shown, the training method of the image enhancement model may include the following steps S301 to S306:

[0106] Step S301, obtaining an image data pair; wherein the image data pair includes a target image and a source image to be enhanced.

[0107] Step S302: Process the image data pair using an image enhancement model to obtain a mapping coefficient for each pixel in the source image.

[0108] The mapping coefficient is used to indicate the mapping relationship between the color parameters of corresponding pixels in the source image and the target image.

[0109] Step S303: Based on the mapping coefficient of each pixel point, mapping processing is performed on the corresponding pixel points in the source image to obtain a mapped image.

[0110] The explanation of steps S301 to S303 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0111] Step S304, obtaining the gradient of the mapping coefficient of each pixel point in a set direction.

[0112] The set direction refers to a pre-set direction. Exemplarily, the set direction may include a horizontal direction and a vertical direction.

[0113] In the embodiment of the present application, the gradient of the mapping coefficient of each pixel in the source image in a set direction may be calculated.

[0114] As an example, taking the setting direction including the horizontal direction and the vertical direction as an example, for any pixel point in the source image, the difference between the mapping coefficient of the pixel point and the mapping coefficient of its neighboring pixel on the right can be calculated, that is, the gradient in the horizontal direction, and the difference between the mapping coefficient of the pixel point and the mapping coefficient of its neighboring pixel below can be calculated, that is, the gradient in the vertical direction.

[0115] Step S305: determining a target loss value according to the gradient of each pixel point and the difference between the mapped image and the annotated image.

[0116] In an embodiment of the present application, the value of the total loss function (referred to as the target loss value in this application) can be calculated based on the gradient of the mapping coefficient of each pixel in the source image in a set direction and the difference between the mapped image and the annotated image. The target loss value is positively correlated with the gradient of the pixel, that is, the larger the gradient of the pixel, the larger the target loss value, and vice versa, the smaller the gradient of the pixel, the smaller the target loss value; the target loss value is also positively correlated with the difference between the mapped image and the annotated image.

[0117] As an example, the target loss value can be calculated by following steps a to c:

[0118] Step a: Generate a first loss value based on the gradient of each pixel.

[0119] As an example, the gradient amplitudes of the mapping coefficients of all pixels in the source image in a specified direction may be summed to obtain a first loss value.

[0120] Step b: Determine a second loss value based on the color difference between the mapped image and the labeled image.

[0121] The second loss value is positively correlated with the color difference, that is, the greater the color difference, the greater the second loss value, and conversely, the smaller the color difference, the smaller the second loss value.

[0122] Step c: Determine a target loss value based on the first loss value and the second loss value.

[0123] As an example, the sum of the first loss value and the second loss value may be used as the target loss value.

[0124] As another example, a weighted sum may be performed on the first loss value and the second loss value to obtain a target loss value.

[0125] As another example, the average of the first loss value and the second loss value may be used as the target loss value.

[0126] Step S306: training the image enhancement model based on the target loss value.

[0127] In an embodiment of the present application, model parameters in the image enhancement model can be adjusted based on the target loss value to minimize the target loss value.

[0128] It should be noted that the above only takes the termination condition of model training as minimization of the target loss value as an example. In actual application, other termination conditions can also be set. For example, the termination conditions can also include: the training time reaches the set time, the training rounds reach the set rounds, and so on.

[0129] The training method of the image enhancement model of the embodiment of the present application comprehensively considers the gradient of the mapping coefficient of each pixel point in the source image in a set direction, as well as the difference between the mapped image and the annotated image, to train the image enhancement model, thereby maintaining the smoothness of the final output or adjusted image.

[0130] The present application embodiment provides another training method for an image enhancement model. Figure 4 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application.

[0131] It should be noted that the training method of the image enhancement model can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0132] like Figure 4 As shown, the training method of the image enhancement model may include the following steps S401 to S409:

[0133] Step S401, obtaining an image data pair; wherein the image data pair includes a target image and a source image to be enhanced.

[0134] Step S402: Process the image data pair using an image enhancement model to obtain a mapping coefficient of each pixel in the source image.

[0135] The mapping coefficient is used to indicate the mapping relationship between the color parameters of corresponding pixels in the source image and the target image.

[0136] Among them, color parameters include brightness and color.

[0137] Step S403: Based on the mapping coefficient of each pixel point, mapping processing is performed on the corresponding pixel points in the source image to obtain a mapped image.

[0138] Step S404, obtaining the gradient of the mapping coefficient of each pixel point in a set direction, and generating a first loss value according to the gradient of each pixel point.

[0139] The explanation of steps S401 to S404 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0140] Step S405: Generate a first sub-loss value according to the brightness difference between the mapped image and the annotated image.

[0141] Among them, the first sub-loss value is positively correlated with the brightness difference.

[0142] As an example, the difference in pixel values ​​between the mapped image and the annotated image in the same color channel can be calculated, and the brightness difference between the mapped image and the annotated image can be calculated based on the difference in pixel values ​​of multiple color channels (such as the three color channels of R, G, and B), so that the first sub-loss value corresponding to the brightness component can be generated based on the brightness difference.

[0143] For example, the label mapping image is I pred , label the image as I gt , you can extract I pred and I gt The corresponding pixel values ​​of the two images on the three RGB color channels can calculate the pixel-level difference for each color channel (R, G, B). For each pixel point, the pixel value differences of the three color channels of the pixel point can be added to obtain the total difference of the pixel point. The total differences of all pixels can be averaged or summed to obtain I pred and I gt The brightness difference between .

[0144] Step S406: Generate a second sub-loss value according to the color difference between the mapped image and the annotated image.

[0145] Among them, the second sub-loss value is positively correlated with the color difference.

[0146] As an example, the mapping image can be converted from a first color space (such as RGB color space) to a second color space corresponding to brightness (such as LAB (Lightness, a, b) color space, where LAB color space is a color space closer to human eye perception, L represents illuminance (Lightness, equivalent to brightness), a represents the color axis from green to red, and b represents the color axis from blue to yellow) to obtain a first converted image. Similarly, the labeled image can be converted from the first color space to the second color space to obtain a second converted image, so that the second sub-loss value corresponding to the color component can be generated according to the color difference between the first converted image and the second converted image.

[0147] Step S407, determining a second loss value according to the first sub-loss value and the second sub-loss value.

[0148] As an example, the sum of the first sub-loss value and the second sub-loss value may be used as the second loss value.

[0149] As another example, a weighted sum may be performed on the first sub-loss value and the second sub-loss value to obtain the second loss value.

[0150] As another example, the average of the first sub-loss value and the second sub-loss value may be used as the second loss value.

[0151] Step S408: determining a target loss value according to the first loss value and the second loss value.

[0152] For the explanation of step S408, please refer to the relevant description in any embodiment of the present application, and will not be repeated here.

[0153] As an example, the label mapping image is I pred , label the image as I gt , the target loss value is L, then:

[0154] L=L1(I pred ,I gt )+L1(LAB(I pred ),LAB(I gt ))+L TV ; (2)

[0155] Among them, L1(I pred ,I gt ) represents the first sub-loss value, L1(LAB(I pred ),LAB(I gt )) represents the second sub-loss value, L TV Represents the first loss value.

[0156] Step S409: training the image enhancement model based on the target loss value.

[0157] For the explanation of step S409, please refer to the relevant description in any embodiment of the present application, and will not be repeated here.

[0158] The training method of the image enhancement model of the embodiment of the present application comprehensively considers the gradient of the mapping coefficient of each pixel point in the source image in a set direction, as well as the brightness difference and color difference between the mapped image and the annotated image, to train the image enhancement model, thereby maintaining the smoothness of the final output or adjusted image and the consistency of color parameters.

[0159] The present application embodiment provides another training method for an image enhancement model. Figure 5 A flowchart of another method for training an image enhancement model provided in an embodiment of the present application.

[0160] It should be noted that the training method of the image enhancement model can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0161] like Figure 5As shown, the training method of the image enhancement model may include the following steps S501 to S506:

[0162] Step S501, obtaining a sample image.

[0163] There is no restriction on the method of acquiring sample images. For example, sample images can be acquired from an existing training set or test set, or sample images can be captured by a camera in an electronic device, or sample images can be artificially synthesized, etc. The embodiments of the present application do not impose any restrictions on this.

[0164] Step S502: cropping the sample image to obtain a first image block and a second image block; wherein the first image block is contained in the second image block, or the second image block is contained in the first image block.

[0165] In an embodiment of the present application, two image blocks can be randomly cropped from a sample image, where one image block is completely contained in the other image block, so as to simulate the FOV (Field of View) difference between different cameras, which are recorded as the first image block and the second image block in the present application.

[0166] As an example, the first image block is marked as Mark the second image block as

[0167] Step S503: performing a data augmentation operation on the first image block and the second image block to obtain an image data pair.

[0168] The image data pair includes a target image and a source image to be enhanced.

[0169] The data augmentation operation includes, but is not limited to, random distortion operation, random noise addition operation, random blur operation, color parameter transformation operation (such as brightness color transformation operation), etc.

[0170] In the embodiment of the present application, a data augmentation operation may be performed on the first image block and the second image block to obtain a target image and a source image in an image data pair.

[0171] In any embodiment of the present application, the image data pair can be obtained by following steps E to H:

[0172] Step E: performing a first data augmentation operation on one of the first image block and the second image block to obtain a processed image block.

[0173] The first data augmentation operation includes at least one of the following operations: a random distortion operation (used to simulate lens distortion), a random noise addition operation (used to simulate noise differences between images), and a random blur operation (used to simulate clarity differences between images).

[0174] Step F: Generate a first intermediate image and a second intermediate image according to the other image block of the first image block and the second image block and the image block processed in step E.

[0175] As an example, the first image block is marked as Mark the second image block as Can or One of the image blocks in performs the first data augmentation operation, and the other image block remains unchanged, and the result is and Among them, if Perform the first data augmentation operation, then and Remain unchanged; if Perform the first data augmentation operation, then and Remain unchanged.

[0176] At this time, you can As the first intermediate image, as the second intermediate image.

[0177] Step G: performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain a source image and a target image; wherein the second data augmentation operation includes a color parameter transformation operation.

[0178] As an example, a second data augmentation operation may be performed on the second intermediate image to obtain a target image, and the first intermediate image may be used as a source image.

[0179] At this time, in order to ensure that the picture content of the annotated image is consistent with the picture content of the source image, and the color parameters of the annotated image match the color parameters of the target image, the second data augmentation operation that is the same as the second intermediate image can be performed on the first intermediate image to obtain the annotated image.

[0180] As another example, a second data augmentation operation may be performed on the first intermediate image to obtain a source image, and the second intermediate image may be used as a target image.

[0181] At this time, in order to ensure that the picture content of the annotated image is consistent with the picture content of the source image and the color parameters of the annotated image match the color parameters of the target image, the first intermediate image can be directly used as the annotated image.

[0182] Step H: Generate an image data pair according to the source image and the target image, that is, the image data pair includes the source image and the target image.

[0183] Therefore, it is possible to automatically generate large-scale annotated training data through data augmentation technology, and the data distribution can be as close to the actual scene as possible, without the need for additional manual data annotation, thus reducing the annotation cost. In addition, by performing random transformation and amplification operations on the training data, the image enhancement model can learn more data of different styles during the training process, thereby improving the generalization ability of the image enhancement model.

[0184] Step S504: Process the image data pair using an image enhancement model to obtain a mapping coefficient for each pixel in the source image.

[0185] The mapping coefficient is used to indicate the mapping relationship between the color parameters of corresponding pixels in the source image and the target image.

[0186] Step S505 , mapping the corresponding pixel points in the source image based on the mapping coefficient of each pixel point to obtain a mapped image.

[0187] Step S506: training the image enhancement model based on the mapped image and the annotated image associated with the image data pair.

[0188] The explanation of steps S504 to S506 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0189] The training method of the image enhancement model in the embodiment of the present application can automatically generate training data through data enhancement or data augmentation technology, which can reduce the cost of obtaining training data.

[0190] In any embodiment of the present application, by constructing a deep learning model, for a pair of input images taken by different cameras, the brightness and color of one image are adjusted to be consistent with the other image, so as to maintain the continuity of image quality when the camera is switched, and improve the user experience in the shooting scene. It can be applied to the following fields: multi-camera switching image quality smoothness, scalable multi-camera fusion imaging, panoramic imaging, data stream switching image quality smoothness and other technical fields.

[0191] The input of the solution provided in this application is a pair of images taken by two cameras, which are respectively denoted as source image I and src and the target image I dst Without loss of generality, I dst The brightness and color of the image are the standard, and the output of the solution provided by this application is a single image and I dst The image with the same brightness and color is denoted as Ipred . Corresponding to the camera switching process, I dst is the image before switching. After consistency mapping, the image after switching is src Change to I pred , thereby achieving a continuous transition of brightness and color and achieving smooth image quality. The following introduces the three core parts of the solution provided by this application: model, training data and loss function.

[0192] Part I: Model.

[0193] Since the output of the solution provided in this application is an image, some generative models can be used in model design. However, the training of such models is unstable and the quality of the generated results is difficult to guarantee. The solution provided in this application can refer to HDRNet. src For each pixel on the image, the model predicts 12 mapping coefficients and acts on the source image. The formula is:

[0194]

[0195] The solution provided in this application takes HDRNet as an example to design the model structure. In actual application scenarios, the model structure can be designed specifically to achieve a balance between speed and accuracy. Since the input of HDRNet is a single image with 3 channels, and the input of the solution provided in this application is two images, the two images need to be spliced ​​along the channel dimension and sent to the model with 6 channels. Therefore, the first layer of convolution of the model is different. The structure of other parts is consistent with HDRNet. By inputting low-resolution images into the model to save time and overhead, and using bilateral grids to restore high-resolution mapping coefficient maps with higher accuracy, the implementation principle can be as follows: Figure 6 shown.

[0196] Part 2: Training data.

[0197] The solution provided in this application can construct image data pairs that are as close to the actual scene as possible based on a public data set through a series of data augmentation operations, wherein the ground truth image can be constructed by brightness and color transformation. Assuming that the sample image obtained from the public data set is I, the training data is obtained by performing the following operations:

[0198] Data augmentation 1: Randomly crop two image blocks from the sample image, one of which is completely contained in the other image block, to simulate the FOV difference between different cameras. This step can get

[0199] Data augmentation 2: or Random distortion is performed to simulate lens distortion, random noise is added to simulate noise differences between images, and random blur is performed to simulate clarity differences between images. This series of steps can achieve the goal of being closer to the actual image data. After these steps, we can get and

[0200] Data augmentation 3: and Perform the same brightness and color transformation to obtain and In this way, the input image data pair can be constructed Ground Truth is This step can also be done only for Perform brightness and color transformation to construct the input image data pair Ground Truth is

[0201] Part III: Loss Function.

[0202] The solution provided in this application can adopt an end-to-end approach to directly perform supervised learning on the final output image (I gt is the automatically generated annotation image, I pred is the predicted image). The loss function includes three items, and the L1 loss function is used to calculate the brightness difference of the three color channels of RGB respectively. In order to provide color supervision information, the solution provided in this application converts the image from the RGB domain to the LAB domain and calculates the L1 loss of the color component. In addition, the solution provided in this application also uses a regularization loss function - TV loss (Total Variation loss) to constrain the intermediate result of the mapping coefficient, so as to maintain the smoothness of the final output image.

[0203] L=L1(I pred ,I gt )+L1(LAB(I pred ),LAB(I gt ))+L TV ; (2)

[0204] Compared with the prior art, the solution provided in this application has at least the following advantages:

[0205] 1. Good independence and flexibility: The solution provided by this application is independent of the ISP module and can be independently developed and tested;

[0206] 2. High versatility: The solution provided in this application processes two images in real time, so it can be migrated to any terminal and any sensor and has wide applicability;

[0207] 3. Strong generalization: The solution provided in this application adopts deep learning technology. By performing random transformation and amplification operations on training data, the model can learn more data of different styles during the training process, thereby improving the generalization ability of the model;

[0208] 4. Strong robustness: The solution provided in this application adopts an end-to-end learning method, and learns the abstract representation of data through multi-layer nonlinear transformation, which can better adapt to different data distributions and changes;

[0209] 5. Easy to train, no additional labeling cost required: The solution provided in this application can automatically generate large-scale labeled training data through data enhancement, and the data distribution is as close to the actual scene as possible.

[0210] The above are various embodiments corresponding to the training method of the image enhancement model. The present application also proposes an application method of the image enhancement model, namely, an image enhancement method.

[0211] Figure 7 A flowchart of an image enhancement method provided in an embodiment of the present application.

[0212] In any embodiment of the present application, the image enhancement method may be executed by an electronic device.

[0213] In any embodiment of the present application, the image enhancement method can be executed by a chip, which can be integrated into electronic devices such as mobile terminals or AI (Artificial Intelligence) servers. Among them, the chip includes image signal processing (Image Signal Processing, referred to as ISP), central processing unit (Central Processing Unit, referred to as CPU), application-specific integrated circuit (Application-Specific Integrated Circuit, referred to as ASIC), microprocessor (Digital Signal Processor, referred to as DSP), field programmable gate array (Field-Programmable Gate Array, referred to as FPGA), system on chip (System On A Chip, referred to as SOC), reduced instruction set computer (Reduced Instruction Set Computer, referred to as RISC), etc., which are not listed one by one this time.

[0214] like Figure 7 As shown, the image enhancement method may include the following steps S701 to S703:

[0215] Step S701, in response to the electronic device's first camera switching to the second camera, or the first camera and the second camera shooting simultaneously, a first image captured by the first camera and a second image captured by the second camera are acquired.

[0216] The electronic device is equipped with multiple cameras, wherein the multiple cameras include at least a first camera and a second camera.

[0217] In an embodiment of the present application, when it is detected that the first camera of the electronic device switches to the second camera, the first image captured by the first camera before the camera switching and the second image captured by the second camera after the camera switching are obtained. Alternatively, when it is detected that the first camera and the second camera are shooting at the same time, the first image captured by the first camera and the second image captured by the second camera can be obtained.

[0218] Step S702, using an image enhancement model to process the first image and the second image to obtain a mapping coefficient for each pixel in the second image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the first image and the second image.

[0219] The image enhancement model is a trained deep learning model. For example, the image enhancement model can be trained using any of the above method embodiments.

[0220] In an embodiment of the present application, an image enhancement model may be used to process the first image and the second image to obtain a mapping coefficient for each pixel in the second image.

[0221] As an example, the first image and the second image can be downsampled respectively to obtain a first sampling image corresponding to the first image and a second sampling image corresponding to the second image, and the first sampling image and the second sampling image can be spliced ​​to obtain a spliced ​​image. Thereafter, the spliced ​​image can be processed using an image enhancement model to obtain the mapping coefficient of each pixel in the second image.

[0222] Step S703: using the mapping coefficient of each pixel point, mapping processing is performed on the corresponding pixel points in the second image to obtain an enhanced image.

[0223] It should be noted that the implementation principle of step S703 is similar to that of step S103, which will not be described in detail here.

[0224] As an example, the image enhancement method provided in the present application can be applied to any scenario that requires multiple cameras to collect information to achieve consistency in color parameters (such as brightness and color) between multiple images to facilitate subsequent information processing.

[0225] These scenarios include but are not limited to the following:

[0226] 1. Multi-camera shooting scene: When using multiple cameras for shooting, in order to ensure the continuity and consistency of the shooting pictures, the brightness and color of each camera need to be processed consistently.

[0227] 2. Virtual reality and augmented reality scenarios: In VR and AR (Augmented Reality), multiple cameras may be required to simultaneously capture images of the real world and then perform image synthesis. In order to ensure the authenticity and coherence of the synthesized images, it is necessary to achieve brightness and color consistency between multiple cameras.

[0228] 3. Autonomous driving scenarios: Autonomous vehicles are usually equipped with multiple cameras to capture environmental information around the vehicle. In order to ensure that the vehicle can accurately identify environmental information, it is necessary to achieve brightness and color consistency among multiple cameras.

[0229] 4. Security monitoring scenario: In a security monitoring system, multiple cameras may be used for monitoring. In order to ensure the clarity and accuracy of the monitoring images, the brightness and color of each camera need to be processed consistently.

[0230] 5. Machine vision scenario: In machine vision applications, multiple cameras may be used for image acquisition. In order to ensure the accuracy of the image, the brightness and color of each camera need to be processed consistently.

[0231] 6. 3D (three-dimensional) reconstruction scene: In 3D reconstruction, multiple cameras are used to shoot objects from different angles, and then these images are combined into a 3D model through algorithms. In order to ensure the accuracy of 3D model construction, the brightness and color of each camera need to be processed consistently.

[0232] The image enhancement method of the embodiment of the present application can adjust the color parameters (such as brightness and color) of one of a pair of input images taken by different cameras to be consistent with the other image based on deep learning technology, so that when the camera is switched or multiple cameras are shooting at the same time, the continuity and consistency of the image quality can be maintained, thereby improving the user experience in the shooting scene.

[0233] In order to implement the above embodiment, the embodiment of the present application also proposes a training device for an image enhancement model.

[0234] Figure 8 A schematic diagram of the structure of a training device for an image enhancement model provided in an embodiment of the present application.

[0235] like Figure 8 As shown, the training device 800 of the image enhancement model may include: an acquisition module 810, a first processing module 820, a second processing module 830 and a training module 840.

[0236] The acquisition module 810 is used to acquire an image data pair; wherein the image data pair includes a target image and a source image to be enhanced;

[0237] The first processing module 820 is used to process the image data pair using an image enhancement model to obtain a mapping coefficient of each pixel in the source image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the source image and the target image;

[0238] The second processing module 830 is used to perform mapping processing on corresponding pixel points in the source image based on the mapping coefficient of each pixel point to obtain a mapping image;

[0239] The training module 840 is used to train the image enhancement model based on the mapped image and the labeled image associated with the image data pair.

[0240] Furthermore, in an implementation of an embodiment of the present application, the first processing module 820 is specifically used to: downsample the source image and the target image respectively to obtain a first sampling image corresponding to the source image and a second sampling image corresponding to the target image; splice the first sampling image and the second sampling image to obtain a spliced ​​image; and process the spliced ​​image using an image enhancement model to obtain a mapping coefficient of each pixel.

[0241] In one implementation of an embodiment of the present application, the first processing module 820 is specifically used to: use the feature extraction network in the image enhancement model to extract global features and local features from the stitched image; fuse the global features and local features to obtain fused features; use the prediction network in the image enhancement model to process the fused features to obtain a coefficient matrix output by the prediction network; wherein the elements in the i-th and j-th columns in the coefficient matrix are used to indicate the mapping coefficients of the pixel points in the i-th row and j-th column in the source image, and i and j are both positive integers.

[0242] In one implementation of an embodiment of the present application, the first processing module 820 is specifically used to: use the first extraction branch in the feature extraction network to perform feature extraction on the stitched image to obtain basic features; use the second extraction branch in the feature extraction network to perform feature extraction on the basic features to obtain global features; use the third extraction branch in the feature extraction network to perform feature extraction on the basic features to obtain local features.

[0243] In one implementation of an embodiment of the present application, the training module 840 is specifically used to: obtain the gradient of the mapping coefficient of each pixel point in a set direction; determine the target loss value based on the gradient of each pixel point and based on the difference between the mapped image and the annotated image; and train the image enhancement model based on the target loss value.

[0244] In one implementation of an embodiment of the present application, the training module 840 is specifically used to: generate a first loss value based on the gradient of each pixel point; determine a second loss value based on the color difference between the mapped image and the annotated image; and determine a target loss value based on the first loss value and the second loss value.

[0245] In one implementation of an embodiment of the present application, the training module 840 is specifically used to: generate a first sub-loss value based on the brightness difference between the mapped image and the annotated image; generate a second sub-loss value based on the color difference between the mapped image and the annotated image; and determine the second loss value based on the first sub-loss value and the second sub-loss value.

[0246] In one implementation of an embodiment of the present application, the training module 840 is specifically used to: obtain the pixel value difference between the mapped image and the annotated image in the same color channel; determine the brightness difference between the mapped image and the annotated image based on the pixel value difference of multiple color channels; and generate a first sub-loss value based on the brightness difference.

[0247] In one implementation of an embodiment of the present application, the training module 840 is specifically used to: convert the mapping image from a first color space to a second color space corresponding to brightness to obtain a first converted image; convert the annotated image from the first color space to the second color space to obtain a second converted image; and generate a second sub-loss value based on the color difference between the first converted image and the second converted image.

[0248] In one implementation of an embodiment of the present application, the acquisition module 810 is specifically used to: acquire a sample image; crop the sample image to obtain a first image block and a second image block; wherein the first image block is contained in the second image block, or the second image block is contained in the first image block; perform a data augmentation operation on the first image block and the second image block to obtain an image data pair.

[0249] In one implementation of the embodiment of the present application, the acquisition module 810 is specifically used to: perform a first data augmentation operation on one of the first image block and the second image block to obtain a processed image block; wherein the first data augmentation operation includes at least one of the following operations: a random distortion operation, a random noise addition operation, and a random blur operation; generate a first intermediate image and a second intermediate image according to the other image block of the first image block and the second image block and the processed image block; perform a second data augmentation operation on the first intermediate image or the second intermediate image to obtain a source image and a target image; wherein the second data augmentation operation includes a color parameter transformation operation; and generate an image data pair according to the source image and the target image.

[0250] In one implementation of the embodiment of the present application, the acquisition module 810 is specifically used to: use the first intermediate image as the source image; and perform a second data augmentation operation on the second intermediate image to obtain a target image.

[0251] In an implementation of the embodiment of the present application, the acquisition module 810 is further used to: perform a second data augmentation operation on the first intermediate image to obtain a labeled image.

[0252] In one implementation of the embodiment of the present application, the acquisition module 810 is specifically used to: use the second intermediate image as the target image; and perform a second data augmentation operation on the first intermediate image to obtain a source image.

[0253] In an implementation of the embodiment of the present application, the acquisition module 810 is further configured to: use the first intermediate image as a labeled image.

[0254] It should be noted that the aforementioned explanation of the embodiment of the training method of the image enhancement model is also applicable to the training device of the image enhancement model of this embodiment, and will not be repeated here.

[0255] In the training device of the image enhancement model of the embodiment of the present application, the mapping coefficient of each pixel in the source image to be enhanced is predicted based on deep learning technology, and based on the mapping coefficient, each pixel in the source image is adjusted to obtain the adjusted source image (referred to as the mapping image in this application), so that the image enhancement model is trained based on the difference (such as color difference) between the mapping image and the annotated image (consistent with the picture content of the source image and consistent with the color parameters of the target image), so that the model can learn to obtain the actual mapping relationship of the color parameters between the source image and the target image, so that the source image is adjusted based on the mapping relationship, and it can be ensured that the adjusted source image and the target image are consistent in color parameters, and then the trained image enhancement model is applied to multi-camera switching scenes or multi-camera shooting scenes, which can maintain the continuity and consistency of image quality and improve the user experience in the shooting scene. Optionally, the image enhancement model can be independent of the ISP module and can be independently developed and tested to enhance the flexibility of the method.

[0256] In order to implement the above embodiment, the embodiment of the present application also proposes an image enhancement device.

[0257] Fig. 9 A schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application.

[0258] like Fig. 9 As shown, the image enhancement device 900 may include: an acquisition module 910 , a first processing module 920 , and a second processing module 930 .

[0259] The acquisition module 910 is used to acquire a first image captured by the first camera and a second image captured by the second camera in response to the first camera in the electronic device switching to the second camera, or the first camera and the second camera shooting at the same time;

[0260] A first processing module 920 is used to process the first image and the second image using an image enhancement model to obtain a mapping coefficient of each pixel in the second image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the first image and the second image;

[0261] The second processing module 930 is used to use the mapping coefficient of each pixel point to perform mapping processing on the corresponding pixel points in the second image to obtain an enhanced image.

[0262] It should be noted that the above explanation of the embodiment of the image enhancement method is also applicable to the image enhancement device of this embodiment, and will not be repeated here.

[0263] In the image enhancement device of the embodiment of the present application, it is possible to adjust the color parameters (such as brightness and color) of one of a pair of input images taken by different cameras to be consistent with the other image based on deep learning technology, so that when the camera is switched or multiple cameras are shooting at the same time, the continuity and consistency of the image quality can be maintained, thereby improving the user experience in the shooting scenario.

[0264] In order to implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image enhancement model training method or image enhancement method as described in any of the above method embodiments.

[0265] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. For example, the electronic device 1000 may be a vehicle, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0266] Reference Fig.10 , the electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .

[0267] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1002 may include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.

[0268] The memory 1004 is configured to store various types of data to support operations on the electronic device 1000. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0269] The power component 1006 provides power to the various components of the electronic device 1000. The power component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.

[0270] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0271] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), and when the electronic device 1000 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.

[0272] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0273] The sensor assembly 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the electronic device 1000, the relative positioning of components, such as the display and keypad of the electronic device 1000, and the sensor assembly 1014 can also detect the position change of the electronic device 1000 or a component of the electronic device 1000, the presence or absence of user contact with the electronic device 1000, the orientation or acceleration / deceleration of the electronic device 1000, and the temperature change of the electronic device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0274] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0275] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0276] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0277] In order to implement the above embodiments, the present application also proposes a chip, including: the chip includes an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output signals, and the processing circuit is configured to execute the image enhancement model training method or image enhancement method provided in any of the above embodiments.

[0278] Fig.11 is a schematic diagram of the structure of a chip proposed in an embodiment of the present application. Fig.11 The structure of the chip 1100 is shown, but is not limited to this.

[0279] Chip 1100 includes a processing circuit 1101, which is configured to execute any of the above image enhancement model training methods or image enhancement methods.

[0280] In some embodiments, the chip 1100 further includes one or more interface circuits 1102. Optionally, the interface circuit 1102 is connected to the memory 1103. The interface circuit 1102 can be used to receive signals from the memory 1103 or other devices, and the interface circuit 1102 can be used to send signals to the memory 1103 or other devices. For example, the interface circuit 1102 can read instructions stored in the memory 1103 and send the instructions to the processing circuit 1101.

[0281] In some embodiments, the interface circuit 1102 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 1101 performs other steps.

[0282] In some embodiments, terms such as interface circuit, interface, transceiver pin, and transceiver may be used interchangeably.

[0283] In some embodiments, the chip 1100 further includes one or more memories 1103 for storing instructions. Alternatively, all or part of the memory 1103 may be outside the chip 1100.

[0284] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the image enhancement model training method or image enhancement method as described in any of the aforementioned method embodiments.

[0285] In order to implement the above embodiments, the present application also proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, it implements the image enhancement model training method or image enhancement method as described in any of the above method embodiments.

[0286] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0287] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0288] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0289] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0290] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0291] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0292] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0293] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A training method for an image enhancement model, characterized in that: include: Acquire an image data pair; wherein the image data pair includes a target image and a source image to be enhanced; The image data pair is processed using an image enhancement model to obtain a mapping coefficient of each pixel in the source image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the source image and the target image; Based on the mapping coefficients of the pixels, mapping is performed on the corresponding pixels in the source image to obtain a mapping image; The image enhancement model is trained based on the mapped image and the annotated image associated with the image data pair.

2. The method according to claim 1, characterized in that The adopting the image enhancement model to process the image data pair to obtain the mapping coefficient of each pixel in the source image includes: Downsampling the source image and the target image respectively to obtain a first sampled image corresponding to the source image and a second sampled image corresponding to the target image; splicing the first sampling image and the second sampling image to obtain a spliced ​​image; The stitched image is processed using the image enhancement model to obtain a mapping coefficient of each pixel.

3. The method according to claim 2, characterized in that The step of using the image enhancement model to process the spliced ​​image to obtain the mapping coefficient of each pixel point includes: Using the feature extraction network in the image enhancement model to extract global features and local features from the spliced ​​image; Fusing the global feature and the local feature to obtain a fused feature; Processing the fusion features using the prediction network in the image enhancement model to obtain a coefficient matrix output by the prediction network; The elements in the i-th and j-th columns of the coefficient matrix are used to indicate the mapping coefficients of the pixels in the i-th row and j-th column of the source image, and i and j are both positive integers.

4. The method according to claim 3, characterized in that The extracting global features and local features from the spliced ​​image using the feature extraction network in the image enhancement model includes: Using the first extraction branch in the feature extraction network to extract features from the spliced ​​image to obtain basic features; Using the second extraction branch in the feature extraction network to extract the basic features to obtain the global features; The third extraction branch in the feature extraction network is used to extract the basic features to obtain the local features.

5. The method according to claim 1, characterized in that: The step of training the image enhancement model based on the mapped image and the annotated image associated with the image data pair comprises: Obtaining the gradient of the mapping coefficient of each pixel point in a set direction; Determining a target loss value according to the gradient of each pixel point and according to the difference between the mapped image and the annotated image; Based on the target loss value, the image enhancement model is trained.

6. The method according to claim 5, characterized in that The determining of the target loss value according to the gradient of each pixel point and the difference between the mapped image and the annotated image includes: Generate a first loss value according to the gradient of each pixel point; determining a second loss value according to a color difference between the mapped image and the annotated image; The target loss value is determined according to the first loss value and the second loss value.

7. The method according to claim 6, characterized in that The color parameter includes brightness and color, and determining the second loss value according to the color difference between the mapped image and the annotated image includes: generating a first sub-loss value according to a brightness difference between the mapped image and the annotated image; generating a second sub-loss value according to a color difference between the mapped image and the annotated image; The second loss value is determined according to the first sub-loss value and the second sub-loss value.

8. The method according to claim 7, characterized in that The step of generating a first sub-loss value according to a brightness difference between the mapped image and the labeled image comprises: Obtaining a difference in pixel values ​​between the mapped image and the annotated image in the same color channel; Determining a brightness difference between the mapped image and the annotated image according to pixel value differences of the plurality of color channels; The first sub-loss value is generated according to the brightness difference.

9. The method according to claim 7, characterized in that: The step of generating a second sub-loss value according to a color difference between the mapped image and the labeled image comprises: Converting the mapped image from a first color space to a second color space corresponding to the brightness to obtain a first converted image; Converting the annotated image from the first color space to the second color space to obtain a second converted image; A second sub-loss value is generated according to a color difference between the first converted image and the second converted image.

10. The method according to any one of claims 1 to 9, characterized in that The acquiring of the image data pair comprises: Get a sample image; Cropping the sample image to obtain a first image block and a second image block; wherein the first image block is contained in the second image block, or the second image block is contained in the first image block; A data augmentation operation is performed on the first image block and the second image block to obtain the image data pair.

11. The method according to claim 10, characterized in that The performing a data augmentation operation on the first image block and the second image block to obtain the image data pair includes: Performing a first data augmentation operation on one of the first image block and the second image block to obtain a processed image block; wherein the first data augmentation operation includes at least one of the following operations: a random distortion operation, a random noise addition operation, and a random blur operation; generating a first intermediate image and a second intermediate image according to the other of the first image block and the second image block and the processed image block; Performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image; wherein the second data augmentation operation includes a color parameter transformation operation; The image data pair is generated according to the source image and the target image.

12. The method according to claim 11, characterized in that The performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image includes: Using the first intermediate image as the source image; A second data augmentation operation is performed on the second intermediate image to obtain the target image.

13. The method according to claim 12, characterized in that The annotated image is obtained by using the following steps: The second data augmentation operation is performed on the first intermediate image to obtain the labeled image.

14. The method according to claim 11, characterized in that The performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image includes: Using the second intermediate image as the target image; A second data augmentation operation is performed on the first intermediate image to obtain the source image.

15. The method according to claim 14, characterized in that The annotated image is obtained by using the following steps: The first intermediate image is used as the labeled image.

16. An image enhancement method, characterized in that: include: In response to a first camera in the electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously, acquiring a first image captured by the first camera and a second image captured by the second camera; Processing the first image and the second image using an image enhancement model to obtain a mapping coefficient of each pixel in the second image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the first image and the second image; The mapping coefficients of the pixels are used to perform mapping processing on the corresponding pixels in the second image to obtain an enhanced image.

17. A training device for an image enhancement model, characterized in that: include: An acquisition module, used for acquiring an image data pair; wherein the image data pair includes a target image and a source image to be enhanced; A first processing module, configured to process the image data pair using an image enhancement model to obtain a mapping coefficient of each pixel in the source image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the source image and the target image; A second processing module, configured to perform mapping processing on corresponding pixel points in the source image based on the mapping coefficients of the pixel points to obtain a mapped image; A training module is used to train the image enhancement model based on the mapped image and the labeled image associated with the image data pair.

18. An image enhancement device, characterized in that: include: An acquisition module, configured to acquire a first image captured by the first camera and a second image captured by the second camera in response to a switch from a first camera to a second camera in the electronic device, or a simultaneous shooting by the first camera and the second camera; a first processing module, configured to process the first image and the second image using an image enhancement model to obtain a mapping coefficient of each pixel in the second image; wherein the mapping coefficient is used to indicate a mapping relationship between color parameters of corresponding pixels in the first image and the second image; The second processing module is used to use the mapping coefficients of the pixels to perform mapping processing on the corresponding pixels in the second image to obtain an enhanced image.

19. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 15 is implemented, or the method according to claim 16 is implemented.

20. A chip, characterized in that: The chip includes an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output a signal, and the processing circuit is used to implement the method according to any one of claims 1 to 15, or the method according to claim 16.

21. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 15, or the computer program implements the method according to claim 16.

22. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15, or implements the method according to claim 16.

Images