Image processing and model training method and device, equipment and medium

Abstract

The invention provides an image processing and model training method and device, equipment and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: responding to a first camera in electronic equipment, switching to a second camera, or carrying out the simultaneous shooting of the first camera and the second camera, and obtaining a global mapping parameter; wherein the global mapping parameter is used for indicating a color mapping relationship between a first image acquired by the first camera and a second image acquired by the second camera; and performing color migration processing on the second image by adopting the global mapping parameter to obtain a target image matched with the color parameter of the first image. Therefore, images obtained when the first camera is switched to the second camera or the first camera and the second camera are used for shooting at the same time have higher color consistency, and the consistency and stability of image quality after color migration are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image processing and model training method, device, equipment and medium. BACKGROUND

[0002] Electronic devices are generally equipped with multiple cameras to meet the imaging needs of users for objects at different positions. However, due to manufacturing processes, costs, and environmental factors, there are often large differences in imaging quality between cameras. With the increasing demand of users for high-definition image quality and seamless switching experience, in multi-camera switching, multi-camera simultaneous shooting and other application scenarios, it is urgent to perform consistency and smoothness processing on image quality (such as color, brightness, and other color parameters) to improve the visual effect of the image. SUMMARY

[0003] The present application aims to at least partially solve one of the technical problems in the related art.

[0004] To this end, the present application proposes an image processing and model training method, device, equipment and medium to achieve higher color consistency between images obtained when switching from a first camera to a second camera or when both are shooting at the same time, and to improve the consistency and stability of image quality after color transfer.

[0005] An embodiment of the present application proposes an image processing method, comprising:

[0006] In response to switching of a first camera to a second camera in an electronic device, or simultaneous shooting of the first camera and the second camera, a global mapping parameter is obtained; wherein the global mapping parameter is used to indicate a color mapping relationship between a first image collected by the first camera and a second image collected by the second camera;

[0007] The global mapping parameter is used to perform color transfer processing on the second image to obtain a target image matching the color parameters of the first image.

[0008] Another embodiment of the present application proposes a training method of an image color transfer model, comprising:

[0009] An image data pair is obtained; wherein the image data pair includes a target image and a source image to be color transferred;

[0010] An image color transfer model is used to process the image data pair to obtain a global mapping parameter; wherein the global mapping parameter is used to indicate a color mapping relationship between the source image and the target image;

[0011] perform color migration processing on the source image based on the global mapping parameter to obtain a migration image;

[0012] train the image color migration model based on the migration image and a labeled image associated with the image data pair.

[0013] Another aspect of the present disclosure provides an image processing apparatus, comprising:

[0014] an obtaining module configured to obtain a global mapping parameter in response to switching a first camera to a second camera in an electronic device or simultaneously capturing by the first camera and the second camera;

[0015] a migration module configured to perform color migration processing on the second image based on the global mapping parameter to obtain a target image matching color parameters of the first image.

[0016] Another aspect of the present disclosure provides an image color migration model training apparatus, comprising:

[0017] an obtaining module configured to obtain an image data pair, wherein the image data pair comprises a target image and a source image to be subjected to color migration;

[0018] a processing module configured to process the image data pair by using an image color migration model to obtain a global mapping parameter, wherein the global mapping parameter is used to indicate a color mapping relationship between the source image and the target image;

[0019] a migration module configured to perform color migration processing on the source image based on the global mapping parameter to obtain a migration image;

[0020] a training module configured to train the image color migration model based on the migration image and a labeled image associated with the image data pair.

[0021] Another aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image processing method according to the foregoing aspect, or implement the image color migration model training method according to the foregoing aspect.

[0022] In yet another aspect, an embodiment of the present application provides a chip, which comprises an interface circuit and a processing circuit coupled with each other, the interface circuit is configured to input or output a signal, and the processing circuit is configured to implement the image processing method according to the foregoing aspect, or implement the training method of the image color migration model according to the foregoing aspect.

[0023] In yet another aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is configured to implement the image processing method according to the foregoing aspect, or implement the training method of the image color migration model according to the foregoing aspect when executed by a processor.

[0024] In yet another aspect, an embodiment of the present application provides a computer program product, which stores a computer program, and the computer program is configured to implement the image processing method according to the foregoing aspect, or implement the training method of the image color migration model according to the foregoing aspect when executed by a processor.

[0025] The image processing and model training method, device, equipment and medium provided by the present application can predict global mapping parameters according to a first image collected by a first camera and a second image collected by a second camera in a scenario where the first camera is switched to the second camera or both of them are shooting at the same time, wherein all pixel points in the entire second image share the global mapping parameters, and for a high-resolution image, the parameter amount and the calculation requirement can be greatly reduced, so that the processing process is more efficient, and the color migration speed of the image is significantly improved. With the reduction of the parameter amount, the demand for computing resources is reduced, and the requirement for storage space is also reduced, which is particularly important for applications in mobile devices or other resource-constrained environments, and helps to prolong the battery life and improve the overall performance of the device. At the same time, the global mapping parameters provide a unified color mapping method, which ensures that the images obtained when the first camera is switched to the second camera or both of them are shooting at the same time have higher color consistency, avoids the discontinuity problem that may occur in pixel-by-pixel prediction, and thus improves the consistency and stability of the image quality after color migration.

[0026] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A flowchart of a first image processing method provided by an embodiment of the present application;

[0029] Figure 2 A flowchart of a second image processing method provided by an embodiment of the present application is shown in FIG. 3;

[0030] Figure 3 A flowchart of a third image processing method provided by an embodiment of the present application is shown in FIG. 4;

[0031] Figure 4 A structure diagram of an image color migration model provided by an embodiment of the present application is shown in FIG. 5;

[0032] Figure 5 A flowchart of a fourth image processing method provided by an embodiment of the present application is shown in FIG. 6;

[0033] Figure 6 A flowchart of a first image color migration model training method provided by an embodiment of the present application is shown in FIG. 7;

[0034] Figure 7 A flowchart of a second image color migration model training method provided by an embodiment of the present application is shown in FIG. 8;

[0035] Figure 8 A flowchart of a third image color migration model training method provided by an embodiment of the present application is shown in FIG. 9;

[0036] Figure 9 A flowchart of a fourth image color migration model training method provided by an embodiment of the present application is shown in FIG. 10;

[0037] Figure 10 A flowchart of an I pred acquisition method provided by an embodiment of the present application is shown in FIG. 11;

[0038] Figure 11 A structure diagram of an image processing device provided by an embodiment of the present application is shown in FIG. 12;

[0039] Figure 12 A structure diagram of an image color migration model training device provided by an embodiment of the present application is shown in FIG. 13;

[0040] Figure 13 A structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 14;

[0041] Figure 14 A structure diagram of a chip provided by an embodiment of the present application is shown in FIG. 15. DETAILED DESCRIPTION

[0042] Embodiments of the present application are described below in detail with reference to examples shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0043] For a multi-camera system, due to the independence of each camera's hardware, field of view angle, Image Signal Processor (ISP) algorithm, image tuning, etc., there are differences in the image quality (such as color, brightness, etc. color parameters) output by different cameras. In the zooming process of camera switching, the image quality will suddenly change, affecting the subjective experience of users. Therefore, how to realize the consistency of color parameters between multiple cameras is an important problem.

[0044] Among them, the multi-camera system refers to installing multiple cameras on an electronic device to obtain more image information. Such a system has been widely used in the fields of smart phones, unmanned vehicles, security monitoring, etc. However, due to the differences in hardware characteristics, installation positions, lighting conditions, etc. of each camera, the images captured by different cameras may have consistency problems in color parameters (such as brightness and color), resulting in uneven image quality in multi-camera switching, multi-camera opening, etc. Application scenarios, which also brings trouble to the subsequent processing and application of images. To solve this problem, comprehensive optimization can be carried out from multiple levels such as hardware level, algorithm, standardization, etc.

[0045] In related technologies, most of the research relies on a series of correction modules in the ISP, such as automatic exposure, automatic white balance, etc. By adjusting the brightness, color balance, etc. imaging parameters, the images obtained by different cameras are visually consistent. However, this method is constrained by the correction module, and the parameter adjustment space is very limited, resulting in poor flexibility.

[0046] Some researches also use adaptive algorithms based on image statistics, such as histogram matching algorithm, to directly establish the mapping relationship between image brightness and color. However, the traditional image processing techniques used in these researches are often sensitive to changes in camera hardware characteristics, lighting conditions, etc. and are difficult to adapt to diversified scenarios.

[0047] There are also some studies that use deep learning methods, such as the High Dynamic Range Network (HDRNet) that learns a pixel-by-pixel color matrix, where each pixel's color matrix contains 12 mapping coefficients. Assuming the size of the input image is 3840x2160, the total number of mapping coefficients is: 3840*2160*12=99532800 (nearly 100 million), which requires a very high computing resource. This high parameter quantity will lead to a decline in network performance (such as slow inference speed or high memory occupancy).

[0048] In summary, the related art has at least the following disadvantages or deficiencies:

[0049] 1. Poor independence and flexibility: Most related technologies rely on the correction module in the ISP to implement, which is constrained by the correction module and cannot be developed and tested independently of other modules.

[0050] 2. Poor generalization: Most related technologies use some ISP algorithms with fewer adjustable parameters, which are limited by the complexity of the model, making it difficult to adapt to complex data distribution and have limited generalization ability, and can only be used in specific scenarios.

[0051] 3. Poor robustness: The image processing technology of the related art is often sensitive to noise and outliers in image data, and cannot well handle the color consistency problem under complex environmental conditions.

[0052] 4. Dependent on manual calibration: The deep learning method of the related art requires a large amount of labeled data, i.e., pixel-level labeling, which has high labeling cost and low labeling efficiency.

[0053] 5. Unable to balance performance and effect performance: In order to ensure good effect performance, most network designs in the related art tend to be highly complex, which not only leads to performance decline and energy consumption increase, but also limits the widespread deployment and implementation of these technologies in practical applications. On the other hand, deployable and user-friendly networks often have poor mapping effects and cannot achieve smooth transition of image quality when switching cameras.

[0054] In order to solve at least one of the above problems, the present application mainly proposes an image processing and model training method, device, equipment and medium.

[0055] The image processing and model training method, device, equipment and medium of the embodiments of the present application are described below with reference to the accompanying drawings. Before specifically describing the embodiments of the present application, in order to facilitate understanding, first introduce the commonly used technical terms:

[0056] 3DLUT, short for 3D Look-Up Table, is a technology widely used in image processing, video editing, film color grading, display calibration, etc. Its core principle is to convert the input RGB (Red, Green, Blue) color value into the output RGB color value through a predefined 3D color mapping table, realizing the conversion of color space or the adjustment of color style. That is, 3DLUT regards RGB color space as a three-dimensional cube, and each input RGB color value (R, G, B) corresponds to a three-dimensional coordinate. Through the lookup table, the output color value (R', G', B') corresponding to the three-dimensional coordinate is found.

[0057] Figure 1 The flowchart of the first image processing method provided by the embodiments of the present application is shown.

[0058] It should be noted that the image processing method of the embodiments of the present application can be applied to an image processing device. In some possible embodiments, the image processing device can be configured in an electronic device or a chip, so that the electronic device or the chip can perform image processing functions. In addition, in some possible embodiments, the image processing device can also be software in an electronic device, etc.

[0059] In any one of the embodiments of the present application, the chip can be integrated into an electronic device. The chip includes a central processing unit (CPU), an image signal processor (ISP), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a system on a chip (SOC), a reduced instruction set computer (RISC), etc., which are not listed one by one.

[0060] The electronic device can be any device with computing capability, such as a terminal, an artificial intelligence (AI) server, and the like. The terminal is an entity on the user side for receiving or transmitting signals, such as a mobile phone. The terminal can also be referred to as a terminal device (terminal), user equipment (UE), a mobile station (MS), a mobile terminal (MT), and the like. The terminal can be a car, a smart car, a mobile phone, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, and the like. Embodiments of the present application do not limit the specific technology and specific device form of the terminal.

[0061] As shown in Figure 1 The image processing method can include the following steps S101-S102:

[0062] In step S101, a global mapping parameter is obtained in response to the first camera being switched to the second camera in the electronic device, or the first camera and the second camera being simultaneously photographed. The global mapping parameter is used to indicate the color mapping relationship between the first image collected by the first camera and the second image collected by the second camera.

[0063] The electronic device is equipped with multiple cameras, wherein the multiple cameras at least include a first camera and a second camera.

[0064] The global mapping parameter is used to indicate the mapping relationship of the color parameter between the first image collected by the first camera and the second image collected by the second camera, which is referred to as the color mapping relationship in the present application. That is, all pixel points share a set of global mapping parameters.

[0065] The color parameter includes but is not limited to color, brightness, and the like.

[0066] In the embodiments of the present application, in a case where it is monitored that the first camera of the electronic device is switched to the second camera, a first image captured by the first camera before the camera switching and a second image captured by the second camera after the camera switching can be acquired. Alternatively, in a case where it is monitored that the first camera and the second camera shoot at the same time, the first image captured by the first camera and the second image captured by the second camera can be acquired. Then, global mapping parameters can be predicted according to the first image and the second image, wherein the global mapping parameters are used to indicate the color mapping relationship between the first image and the second image.

[0067] As an example, the global mapping parameters can be predicted according to the first image and the second image based on a deep learning technology in the AI field. For example, a trained image color transfer model can be used to process the first image and the second image to obtain the global mapping parameters. For example, the first image and the second image can be spliced to obtain a spliced image, and the spliced image can be input into the image color transfer model for processing to obtain the global mapping parameters.

[0068] For example, the model structure of the image color transfer model is not limited, and for example, the image color transfer model can be a generative model, or can be a High Dynamic Range Network (HDRNet) model, an adaptive-3DLUT model, etc.

[0069] In step S102, the second image is color transferred using the global mapping parameters to obtain a target image matching the color parameters of the first image.

[0070] For example, the picture content of the target image is consistent with the picture content of the second image, and the color parameters of the target image match or are consistent with the color parameters of the first image. For example, “matching” means that the similarity is higher than a set similarity threshold.

[0071] In the embodiments of the present application, the second image can be color transferred using the global mapping parameters to obtain a target image matching or consistent with the color parameters of the first image.

[0072] It should be noted that, compared with the pixel-by-pixel prediction in the related art, which needs to calculate 12-dimensional mapping parameters for each pixel point, the global mapping parameters only need to be calculated once and can be applied to the entire image. For high-resolution images, the parameter amount and calculation demand can be greatly reduced, the processing process is more efficient, and the image color transfer speed is improved.

[0073] It should be noted that the image processing method provided in the present application can be applied to any scene requiring multiple cameras to collect information to achieve consistency of color parameters (such as brightness and color) between multiple images, so as to facilitate subsequent information processing.

[0074] These scenarios include but are not limited to the following scenarios:

[0075] 1. Multi-camera shooting scene: When using multiple cameras for shooting, in order to ensure the coherence and consistency of the shooting picture, the brightness and color of each camera need to be consistently processed.

[0076] 2. Virtual reality and augmented reality scene: In VR and AR, multiple cameras may be needed to capture images of the real world simultaneously, and then perform image synthesis. In order to ensure the authenticity and coherence of the synthesized image, the brightness and color consistency between multiple cameras needs to be achieved.

[0077] 3. Unmanned driving scene: Unmanned vehicles are usually equipped with multiple cameras to capture environmental information around the vehicle. In order to ensure that the vehicle can accurately identify environmental information, the brightness and color consistency between multiple cameras needs to be achieved.

[0078] 4. Security monitoring scene: In a security monitoring system, multiple cameras may be used for monitoring. In order to ensure the clarity and accuracy of the monitoring picture, the brightness and color of each camera need to be consistently processed.

[0079] 5. Machine vision scene: In machine vision applications, multiple cameras may be used for image acquisition. In order to ensure the accuracy of the image, the brightness and color of each camera need to be consistently processed.

[0080] 6. 3D (three-dimensional) reconstruction scene: In 3D reconstruction, multiple cameras are needed to shoot objects from different angles, and then algorithms are used to combine these images into a 3D model. In order to ensure the accuracy of the 3D model construction, the brightness and color of each camera need to be consistently processed.

[0081] The image processing method of the embodiments of the present application can, in the case of switching from the first camera to the second camera or simultaneously shooting with both cameras, predict global mapping parameters according to the first image collected by the first camera and the second image collected by the second camera, wherein all pixel points in the entire second image share this set of global mapping parameters. For high-resolution images, the parameter amount and the calculation requirement can be greatly reduced, making the processing process more efficient and significantly improving the color migration speed of the image. With the reduction of the parameter amount, not only the demand for computing resources is reduced, but also the requirement for storage space is reduced, which is particularly important for applications in mobile devices or other resource-constrained environments, helping to prolong the battery life and improve the overall performance of the device. At the same time, the global mapping parameters provide a unified color mapping method, ensuring higher color consistency between the images obtained when switching from the first camera to the second camera or simultaneously shooting with both cameras, avoiding the discontinuity problem that may occur in pixel-by-pixel prediction, thereby improving the consistency and stability of the image quality after color migration.

[0082] The embodiments of the present application provide another image processing method, Figure 2 The flowchart of the second image processing method provided by the embodiments of the present application is shown in FIG. 2.

[0083] It should be noted that the image processing method can be executed alone, or can be executed in combination with any one of the embodiments or the possible implementation manners in the embodiments, or can be executed in combination with any one of the technical solutions in the related art, and the embodiments of the present application do not limit this.

[0084] As shown in FIG. 1, Figure 2 The image processing method can include the following steps S201 to S205:

[0085] Step S201, in response to switching from the first camera to the second camera in the electronic device, or simultaneously shooting with the first camera and the second camera, acquiring a first image collected by the first camera and a second image collected by the second camera.

[0086] It should be noted that the explanation and description of step S201 can be referred to the related description in any one of the embodiments of the present application, which will not be repeated here.

[0087] Step S202, downsampling the first image to obtain a first sample image, and downsampling the second image to obtain a second sample image.

[0088] It should be noted that the calculation of the global mapping parameter directly on the high-resolution image occupies more computing resources and storage resources, thereby leading to poor performance. In view of this problem, in the present application, the high-resolution image can be down-sampled to a low-resolution, and then the global mapping parameter can be predicted according to the low-resolution image.

[0089] That is, in the embodiments of the present application, the high-resolution first image can be down-sampled to obtain a low-resolution first sample image, and similarly, the high-resolution second image can be down-sampled to obtain a low-resolution second sample image.

[0090] In step S203, the first sample image and the second sample image are spliced along the color channel dimension to obtain a spliced image.

[0091] For example, the number of color channels of the first sample image and the second sample image is 3, i.e. 3 color channels of R (Red), G (Green), and B (Blue), and the first sample image and the second sample image can be spliced along the color channel dimension to obtain a spliced image with 6 color channels.

[0092] Taking the size of the first sample image and the second sample image as an example, c x w x h is 3 x 136 x 136, and the size of the spliced image can be 6 x 136 x 136. Wherein, c refers to the color channel, w refers to the width, and h refers to the height.

[0093] In step S204, the global mapping parameter is predicted according to the spliced image.

[0094] In the embodiments of the present application, the global mapping parameter can be predicted according to the spliced image based on the deep learning technology in the AI field. For example, the spliced image can be input into a trained image color transfer model for processing to obtain the global mapping parameter.

[0095] Wherein, the model structure of the image color transfer model is not limited, for example, the image color transfer model can be a generative model, or it can be an HDRNet, an adaptive-3DLUT, etc.

[0096] In step S205, the global mapping parameter is used to perform color transfer processing on the second image to obtain a target image matching the color parameter of the first image.

[0097] It should be noted that the explanation of step S205 can refer to the related description in any embodiment of the present application, which will not be repeated here.

[0098] The image processing method of the present application downsamples a high-resolution image to a low resolution and then predicts global mapping parameters based on the low-resolution image. This significantly reduces computing resource requirements, lowers memory usage, and improves prediction speed. Furthermore, appropriate downsampling can remove unnecessary detail noise while preserving important global information, thereby improving prediction accuracy.

[0099] The present application embodiment provides another image processing method, Figure 3 A flowchart of the third image processing method provided in an embodiment of the present application.

[0100] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application do not limit this.

[0101] like Figure 3 As shown, the image processing method may include the following steps S301 to S306:

[0102] Step S301: In response to a first camera in an electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously, a first image captured by the first camera and a second image captured by the second camera are acquired.

[0103] Step S302: down-sample the first image to obtain a first sampled image, and down-sample the second image to obtain a second sampled image.

[0104] Step S303: splice the first sampled image and the second sampled image along the color channel dimension to obtain a spliced ​​image.

[0105] It should be noted that the explanation of steps S301 to S303 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0106] Step S304: Using the feature extraction network in the image color transfer model, perform feature extraction on the spliced ​​image in the spatial dimension to obtain a target feature map.

[0107] There is no restriction on the model structure of the image color transfer model. For example, the image color transfer model may be a generative model, or may be a model such as HDRNet, adaptive-3DLUT, etc.

[0108] The image color transfer model comprises a feature extraction network and a prediction network, wherein the feature extraction network is configured to extract useful features from the stitched image, and the prediction network is configured to predict global mapping parameters based on the extracted useful features.

[0109] In the embodiments of the present application, the feature extraction network can be used to extract features of the stitched image in the spatial dimension to obtain the target feature map.

[0110] In any one of the embodiments of the present application, the structure of the image color transfer model can be as shown in Figure 4 The feature extraction network can comprise a plurality of convolution layers, wherein each convolution layer is configured to perform a spatial dimension down-sampling operation and a convolution operation. In the present application, the target feature map can be obtained by the following steps A to C:

[0111] Step A: using a first convolution layer in the plurality of convolution layers to perform spatial dimension down-sampling and convolution processing on an input feature map of the first convolution layer to obtain an output feature map of the first convolution layer.

[0112] The size of the weight matrix W and the bias term (or bias) of the convolution kernel of each convolution layer is not limited.

[0113] The input feature map of the first convolution layer is generated based on the stitched image. For example, the stitched image can be directly used as the input feature map of the first convolution layer, or useful features can be preliminarily extracted from the stitched image and used as the input feature map of the first convolution layer.

[0114] Step B: using an i-th convolution layer in the plurality of convolution layers to perform spatial dimension down-sampling and convolution processing on an input feature map of the i-th convolution layer to obtain an output feature map of the i-th convolution layer.

[0115] Wherein, i is a positive integer greater than 1, and the input feature map is obtained by fusing the output feature map of the (i-1)-th convolution layer and the input feature map of the (i-1)-th convolution layer. For example, the output feature map of the (i-1)-th convolution layer and the input feature map of the (i-1)-th convolution layer can be fused by using a residual connection or a skip connection.

[0116] Optionally, a ReLU activation function layer can be further connected after each convolution layer to increase the nonlinear expression ability of the image color transfer model.

[0117] Step C: using the output feature map of the last convolution layer in the plurality of convolution layers as the target feature map.

[0118] Exemplarily, taking the size of the spliced image as 6x136x136, the global mapping parameter being in the form of a 3DLUT, and the number N of patches contained in each color channel of the 3DLUT being 17 as an example, the target feature map with an output size of 51x17x17 can be output through three down-sampling convolution operations with a stride of 2, so that the target feature map can be reshaped by using a reshape operator in the prediction network to obtain a 3DLUT with a size of 3x17x17x17.

[0119] In summary, the manner of extracting the target feature map through multi-layer convolution and down-sampling operations has at least the following advantages: the structure of the multi-layer convolution layer enables the model to utilize features of different scales at the same time, thereby enhancing the representation capability of the feature map; in the i-th convolution layer, the output feature map of the (i-1)-th convolution layer is fused with the input feature map, which can realize information complementation between features of different levels, so that when the features of a certain level are disturbed, the features of other levels can still provide useful information, thereby making the model have stronger resistance to noise and interference and improving the robustness and reliability of feature extraction.

[0120] In step S305, the prediction network in the image color transfer model is used to predict the global mapping parameter according to the target feature map.

[0121] In the embodiments of the present application, the prediction network can be used to predict the global mapping parameter according to the target feature map. The global mapping parameter is used to indicate the color mapping relationship between the first image collected by the first camera and the second image collected by the second camera.

[0122] In any one of the embodiments of the present application, the global mapping parameter is in the form of a three-dimensional lookup table (3DLUT), and the prediction network in the image color transfer model can perform feature reshaping on the target feature map according to the number of patches contained in each color channel of the 3DLUT to obtain the global mapping parameter.

[0123] The number N of patches contained in each color channel of the 3DLUT can be pre-set according to the actual application scenario, and the value of N in different application scenarios can be the same or different, which is not limited in the embodiments of the present application. Exemplarily, the value of N is 17, for example.

[0124] Still taking the above example as an example, the reshape operator can be used to reshape the target feature map with a size of 51x17x17 into a 3DLUT with a size of 3x17x17x17, where 3 represents the number of color channels, and 17 represents the number of patches contained in each color channel.

[0125] In summary, the global mapping parameter output by the image color transfer model can meet the application requirements in the actual application scenario.

[0126] In step S306, the second image is subjected to color migration processing using the global mapping parameter, to obtain a target image matching the color parameter of the first image.

[0127] It should be noted that the explanation of step S306 can refer to the related description in any embodiment of the present application, and will not be repeated here.

[0128] The image processing method of the embodiments of the present application uses deep learning technology to extract useful target feature maps from the stitched image, and predicts the global mapping parameter according to the target feature map, which can improve the prediction efficiency and the accuracy of the prediction result.

[0129] The embodiments of the present application provide another image processing method, Figure 5 The flowchart of the fourth image processing method provided by the embodiments of the present application is shown in FIG. 7.

[0130] It should be noted that the image processing method can be executed alone, or can be executed in combination with any one of the embodiments or the possible implementation manners in the embodiments, or can be executed in combination with any one of the technical solutions in the related art, and the embodiments of the present application do not limit this.

[0131] As shown in FIG. 8, the image processing method can include the following steps S501 to S503: Figure 5

[0132] In step S501, in response to switching of the first camera in the electronic device to the second camera, or simultaneous shooting of the first camera and the second camera, a global mapping parameter is obtained; wherein the global mapping parameter includes a plurality of mapping parameters in the 3DLUT, and the mapping parameter includes a mapping value of a color block on each color channel.

[0133] The global mapping parameter is used to indicate the color mapping relationship between the first image collected by the first camera and the second image collected by the second camera.

[0134] ​The global mapping parameters include a plurality of mapping parameters in the 3DLUT, and the mapping parameters include mapping values of a color block on each color channel. For example, assuming that the number of color blocks contained in each color channel in the 3DLUT is N = 17, the 3DLUT includes 17 * 17 * 17 = 4913 grid points, and each grid point stores a mapping parameter. Each mapping parameter includes mapping values of the three color channels R, G and B, and the total number of parameters is about 4913 * 3 = 14739. Compared with pixel-by-pixel prediction (for example, for an image with a resolution of 3840 * 2160, the number of parameters reaches 99532800), the number of parameters of the 3DLUT is greatly reduced, and the demand for computing resources and storage resources can be greatly reduced.

[0135] It should be noted that the explanation of step S501 can refer to the related description in any embodiment of the present application, and will not be repeated here.

[0136] In step S502, for any pixel point in the second image, a plurality of mapping parameters in the 3DLUT are queried according to the original parameters of the pixel point to obtain candidate parameters with the smallest distance and / or adjacent to the original parameters.

[0137] The original parameters of each pixel point include channel values of a plurality of color channels of the pixel point, such as (R, G, B).

[0138] In the embodiments of the present application, for any pixel point in the second image, a plurality of mapping parameters in the 3DLUT can be queried according to the original parameters of the pixel point to obtain candidate parameters with the smallest distance and / or adjacent to the original parameters.

[0139] As an example, taking the number of color blocks contained in each color channel in the 3DLUT as N = 17, each color channel is divided into 17 discrete intervals, and the 3DLUT has 17 3 4913 grid points (each grid point stores an RGB' value), and the coordinates of each grid point are (r, g, b), where: For example, the grid point coordinates are (0, 0, 0), to (1, 1, 1), each grid point stores an output RGB' value, that is, LUT[r][g][b] = (R', G', B').

[0140] For any pixel point in the second image, the pixel value of the pixel point is usually an 8-bit integer (0-255) or a floating-point number (0.0-1.0). First, the original parameters (or input color) of the pixel point need to be normalized to the input range (0-1) of the 3DLUT:

[0141] Afterwards, the original parameters of the pixel point (i.e. input color) are scaled to the grid point coordinate range: in, Indicates rounding down.

[0142] It should be noted that since the original parameter (input color) may not fall completely on the grid points, the target parameter (i.e. output color) can be calculated by trilinear interpolation. Trilinear interpolation is to perform linear interpolation along the R, G, and B directions respectively. For example, assuming the input color (r norm ,g norm ,b norm ) is located between the following 8 adjacent grid points of the 3DLUT: (r0,g0,b0), (r0,g0,b1), (r0,g1,b0), (r0,g1,b1), (r1,g0,b0), (r1,g0,b1), (r1,g1,b0), (r1,g1,b1), which are recorded as candidate parameters in this application:

[0143] Similarly, g0, g1, b0, and b1 can be calculated.

[0144] Step S503: linearly interpolate the candidate parameters to obtain target parameters of a pixel point corresponding to any pixel point in the target image.

[0145] In an embodiment of the present application, a linear interpolation algorithm may be used to perform linear interpolation on the candidate parameters to obtain target parameters of the pixel points in the target image corresponding to the above-mentioned pixel points.

[0146] Still using the above example, the following trilinear interpolation formula can be used to calculate the target parameters, or output color (R′, G′, B′):

[0147] R′=lerp(lerp(lerp(R' 000 ,R1' 00 ,α),lerp(R' 010 ,R′ 110 ,α),β),lerp(lerp(R' 001 ,R′ 101 ,α),lerp(R' 011 ,R′ 111 ,α),β),γ);

[0148] G′=lerp(lerp(lerp(G' 000 ,G′ 100 ,α),lerp(G' 010 ,G′ 110, α), β), lerp(lerp(G 001 , G′ 101 , α), lerp(G 011 , G′ 111 , α), β), γ) ;

[0149] B′ = lerp(lerp(lerp(B 000 , B′ 100 , α), lerp(B 010 , B′ 110 , α), β), lerp(lerp(B 001 , B′ 101 , α), lerp(B 011 , B′ 111 , α), β), γ) ;

[0150] wherein, (R direction interpolation weight), (G direction interpolation weight), (B direction interpolation weight), lerp(x, y, t) = x x (1-t) + y x t (linear interpolation function), R′ ijk , G′ ijk , B′ ijk are output color values (denoted as mapping values in the present application) of 8 adjacent grid points.

[0151] Optionally, the output color (R′, G′, B′) can also be restored to the original range (such as 0-255) by using a reverse normalization algorithm: R out = R′ x 255, G out = G′ x 255, B out = B′ x 255, that is, the final target parameters are (R out , G out , B out ).

[0152] The image processing method of the embodiments of the present application adopts a linear interpolation algorithm to perform linear interpolation on the candidate parameters, which can ensure that the target parameters (output color) are smoothly transitioned between the color blocks of the 3DLUT, thereby avoiding color mutation or discontinuity.

[0153] The above are various embodiments corresponding to the application method (i.e., the image processing method) of the image color migration model, and the present application further proposes a training method of an image color migration model.

[0154] Figure 6 FIG. 1 is a flowchart of a first training method of an image color migration model provided by the embodiments of the present application.

[0155] AsFigure 6 As shown in the figure, the training method of the image color migration model can include the following steps S601 to S604:

[0156] Step S601, obtaining an image data pair; wherein the image data pair includes a target image and a source image to be color migrated.

[0157] Wherein, the source image refers to the image to be color migrated, that is, the image to be adjusted in color parameter, that is, the color parameter (including but not limited to color, brightness, etc.) of the target image is taken as the standard, and the color migration processing is performed on the source image, so that the color parameter of the source image after color migration is consistent with the color parameter of the target image.

[0158] In the embodiments of the present application, the acquisition method of the image data pair is not limited, for example, the source image and the target image in the image data pair can be obtained from the training set or the test set. Or, the source image and the target image in the image data pair can be collected through different cameras in an electronic device equipped with multiple cameras. For example, in the case that the first camera in the electronic device is switched to the second camera, the image collected by the first camera before the camera is switched can be taken as the target image, and the image collected by the second camera after the camera is switched can be taken as the source image. For another example, in the case that the first camera and the second camera in the electronic device are shooting at the same time, the image collected by one of the cameras can be taken as the source image, and the image collected by the other camera can be taken as the target image. Or, the source image and the target image in the image data pair can be generated based on a certain sample image. Or, the source image and the target image in the image data pair can be artificially synthesized, etc. The embodiments of the present application do not limit this.

[0159] Step S602, processing the image data pair by using the image color migration model to obtain a global mapping parameter; wherein the global mapping parameter is used to indicate the color mapping relationship between the source image and the target image.

[0160] Wherein, the color parameter includes but is not limited to color, brightness, etc.

[0161] In the embodiments of the present application, the image data pair can be processed by using the image color migration model to obtain the global mapping parameter; wherein the global mapping parameter is used to indicate the color mapping relationship between the source image and the target image.

[0162] For example, the target image and the source image in the image data pair can be spliced to obtain a spliced image, and the spliced image can be input into the image color migration model for processing to obtain the global mapping parameter. The implementation principle is similar to step S101, which will not be repeated here.

[0163] In step S603, the source image is subjected to color migration processing based on the global mapping parameter, to obtain a migration image.

[0164] In the embodiments of the present application, the source image can be subjected to color migration processing based on the global mapping parameter, to obtain a source image after color adjustment, which is referred to as a migration image in the present application. The implementation principle is similar to that of step S102, and thus will not be described herein.

[0165] In step S604, the image color migration model is trained based on the migration image and a labeled image associated with the image data.

[0166] The labeled image has the same picture content as the source image, and the color parameters of the labeled image match or are consistent with the color parameters of the target image.

[0167] As an example, the labeled image can be obtained by performing a color parameter transformation operation on the source image.

[0168] As another example, the labeled image can include the target image. For example, when the picture content or the shooting angle of the target image and the source image are consistent, the target image can be used as the labeled image.

[0169] As a possible implementation manner, the image color migration model can be trained according to the color difference between the migration image and the labeled image.

[0170] The training method of the image color migration model according to the embodiments of the present application is based on deep learning technology to predict the global mapping parameter, and based on the global mapping parameter, the source image is subjected to color migration processing to obtain a source image after color adjustment (referred to as a migration image in the present application), so that the image color migration model is trained based on the difference (such as the color difference) between the migration image and the labeled image (which has the same picture content as the source image, and has the same color parameters as the target image). The model can learn the actual color mapping relationship between the source image and the target image, so that the source image is subjected to color migration based on the color mapping relationship, which can ensure that the source image after color migration is consistent with the target image in terms of color parameters, and thus the trained image color migration model can be applied to a multi-camera switching scene or a multi-camera shooting scene, which can maintain the continuity and consistency of the image quality and improve the user experience in the shooting scene.

[0171] The embodiments of the present application provide another training method of an image color migration model, Figure 7 The flowchart of the second training method of the image color migration model provided by the embodiments of the present application is shown in FIG. 6.

[0172] It should be noted that the training method of the image color transfer model can be executed alone, or can be executed in combination with any of the embodiments or possible implementation manners in the present application, or can be executed in combination with any of the related technical solutions, and the present application embodiment does not make any limitation.

[0173] As shown in the Figure 7 training method of the image color transfer model can include the following steps S701 to S705:

[0174] Step S701, obtaining an image data pair, and respectively downsampling a source image and a target image to obtain a source sample image and a target sample image.

[0175] Among the image data pair, the target image and the source image to be color transferred are included. It should be noted that the explanation of the image data pair can refer to the related description in any embodiment of the present application, which will not be repeated here.

[0176] In the present application embodiment, the high-resolution source image can be downsampled to obtain a low-resolution source sample image, and similarly, the high-resolution target image can be downsampled to obtain a low-resolution target sample image.

[0177] Step S702, splicing the source sample image and the target sample image along the color channel dimension to obtain a spliced image.

[0178] As an example, the number of color channels of the source sample image and the target sample image is 3, that is, the three color channels of R (Red), G (Green), and B (Blue), and the source sample image and the target sample image can be spliced along the color channel dimension to obtain a spliced image.

[0179] Step S703, processing the spliced image by using the image color transfer model to obtain a global mapping parameter.

[0180] The global mapping parameter is used to indicate the color mapping relationship between the source image and the target image.

[0181] In the present application embodiment, the spliced image can be input to the image color transfer model for processing to obtain the global mapping parameter. The implementation principle is similar to that of step S204, which will not be repeated here.

[0182] Step S704, performing color transfer processing on the source image based on the global mapping parameter to obtain a transferred image.

[0183] Step S705, training the image color transfer model based on the transferred image and the labeled image associated with the image data pair.

[0184] The explanation of steps S704 to S705 can be referred to the related description in any of the embodiments of the present application, which will not be repeated here.

[0185] The training method of the image color transfer model provided in the embodiments of the present application can consider both the prediction speed and the prediction accuracy of the image color transfer model by downsampling the high-resolution source image and the target image to obtain low-resolution source sample images and target sample images, and inputting the source sample images and the target sample images to the image color transfer model after splicing along the color channel dimension.

[0186] The embodiments of the present application provide another training method of an image color transfer model, Figure 8 The flowchart of the third training method of the image color transfer model provided in the embodiments of the present application is shown in FIG. 8.

[0187] It should be noted that the training method of the image color transfer model can be executed alone, or can be executed in combination with any of the embodiments or possible implementation manners of the embodiments in the present application, or can be executed in combination with any of the technical solutions in the related art, and the embodiments of the present application do not limit this.

[0188] As shown in FIG. 8, the training method of the image color transfer model can include the following steps S801 to S805: Figure 8

[0189] In step S801, an image data pair is obtained, and the image data pair is processed by using the image color transfer model to obtain global mapping parameters.

[0190] The image data pair includes a target image and a source image to be color transferred.

[0191] The global mapping parameters are used to indicate the color mapping relationship between the source image and the target image.

[0192] In step S802, the source image is color transferred based on the global mapping parameters to obtain a transferred image.

[0193] The explanation of steps S801 to S802 can be referred to the related description in any of the embodiments of the present application, which will not be repeated here.

[0194] In step S803, a first loss value is generated according to the mapping values of adjacent color blocks on each color channel in the global mapping parameters.

[0195] The global mapping parameters include a plurality of mapping color parameters in the 3DLUT, and the mapping color parameters include the mapping value of a color block on each color channel.

[0196] ​In an embodiment of the present application, a first loss value may be generated based on the mapping values ​​of adjacent color blocks on each color channel in the 3DLUT. Exemplarily, the first loss value includes a regularization loss value.

[0197] In any embodiment of the present application, the first loss value can be calculated by following steps a to d:

[0198] Step a: Determine a monotonicity loss value based on the degree of non-monotonicity between the mapping values ​​of adjacent color blocks on each color channel.

[0199] It should be noted that in color mapping, maintaining monotonicity means that as the input color increases (or decreases), the output color should also increase (or decrease) accordingly. If monotonicity constraints are not imposed, darker input areas may be mapped to brighter output areas, or vice versa, resulting in color inversion, thereby reducing the color consistency between images obtained when switching from the first camera to the second camera or when both cameras are captured simultaneously.

[0200] In view of the above problems, in this application, a monotonicity loss value can be used to penalize such non-monotonic mapping, thereby ensuring the rationality of color mapping. For example, the sum of the non-monotonicity of the color mapping between all adjacent color blocks in the 3DLUT can be used as the monotonicity loss value, that is, the sum of the non-monotonicity between the mapping values ​​of all adjacent color blocks in the 3DLUT can be used as the monotonicity loss value.

[0201] Step b: Determine the smoothness loss value based on the difference between the mapping values ​​of adjacent color blocks on each color channel.

[0202] It should be noted that in 3DLUT, adjacent color blocks should have similar mapping relationships to ensure the continuity of color mapping. If the mapping relationship of 3DLUT varies too much between adjacent color blocks, it may cause obvious layering or mottled phenomena in the mapped image.

[0203] In view of the above problems, in this application, a smoothness loss value can be used to penalize such discontinuous mapping, thereby ensuring the smoothness of color mapping. For example, the sum of the color mapping differences between all adjacent color blocks in the 3DLUT can be used as the smoothness loss value, that is, the sum of the mapping value differences between adjacent color blocks on each color channel in the 3DLUT can be used as the smoothness loss value.

[0204] The differences include but are not limited to: Euclidean distance, gradient, etc.

[0205] Step c: determining a highlight high saturation loss value according to color differences between target regions in the migration image and corresponding regions in the source image; wherein the brightness of the target regions is higher than a set first threshold, and / or, the saturation of the target regions is higher than a set second threshold.

[0206] It should be noted that in the highlight high saturation region, a slight deviation of color mapping may cause obvious visual artifacts (artifacts) such as color shift, color spots, etc., which affect the visual effect of the image and reduce the picture quality.

[0207] In view of the above problems, in the present application, the color mapping deviation of these regions can be punished by the highlight high saturation loss value, thereby reducing the occurrence of artifacts. Illustratively, the target regions in the migration image can be identified, and the color differences between the target regions and the corresponding regions in the source image can be calculated, and the sum of these color differences can be taken as the highlight high saturation loss value.

[0208] Step d: generating a first loss value according to at least one of the monotonicity loss value, the smoothness loss value and the highlight high saturation loss value.

[0209] In the embodiments of the present application, any one of the monotonicity loss value, the smoothness loss value and the highlight high saturation loss value can be taken as the first loss value, or the first loss value can also be generated according to multiple ones of the monotonicity loss value, the smoothness loss value and the highlight high saturation loss value.

[0210] Illustratively, multiple ones of the monotonicity loss value, the smoothness loss value and the highlight high saturation loss value can be weighted and summed to obtain the first loss value.

[0211] In summary, the introduction of the monotonicity loss value can prevent the occurrence of obvious brightness color inversion, thereby ensuring the rationality of color migration; the introduction of the smoothness loss value can ensure that the mapping values in the 3DLUT learned by the model change relatively smoothly, effectively improving the picture quality layering problem; the introduction of the highlight high saturation loss value can optimize the obvious artifact (artifact) problem caused by too large color shift in the highlight region and / or the high saturation region.

[0212] Step S804: determining a second loss value according to color differences between the migration image and the labeled image.

[0213] In the embodiments of the present application, the second loss value can be calculated according to the color differences between the migration image and the labeled image, wherein the second loss value is in a positive correlation with the above-mentioned color differences, that is, the greater the color differences, the greater the second loss value.

[0214] In any one of the embodiments of the present application, the second loss value can be calculated by the following steps e to g:

[0215] Step e: generating a brightness loss value according to a brightness difference between the migration image and the labeled image.

[0216] The brightness loss value is positively correlated with the brightness difference.

[0217] As an example, the pixel value difference of the migration image and the labeled image in the same color channel can be calculated, and the brightness difference between the migration image and the labeled image can be calculated according to the pixel value difference of multiple color channels (such as R, G, B color channels), so that the loss value corresponding to the brightness component can be generated according to the brightness difference, which is referred to as the brightness loss value in the present application.

[0218] For example, the migration image is denoted as I pred , and the labeled image is denoted as I gt . The corresponding pixel values of I pred and I gt in the RGB three color channels can be extracted, and for each color channel (R, G, B), the pixel-level difference can be calculated, and for each pixel point, the pixel value difference of the three color channels of the pixel point can be added to obtain the total difference of the pixel point. The total difference of all pixel points is averaged or summed to obtain the brightness loss value between I pred and I gt .

[0219] Step f: generating a color loss value according to a color difference between the migration image and the labeled image.

[0220] The color loss value is positively correlated with the color difference.

[0221] As an example, the migration image can be converted from a first color space (such as an RGB color space) to a second color space corresponding to brightness (such as an LAB (Lightness, a, b) color space, where the LAB color space is a color space closer to human eye perception, L represents lightness (equivalent to brightness), a represents a color axis from green to red, and b represents a color axis from blue to yellow), to obtain a first converted image. Similarly, the labeled image can be converted from the first color space to the second color space to obtain a second converted image, so that the loss value corresponding to the color component can be generated according to the color difference between the first converted image and the second converted image, which is referred to as the color loss value in the present application.

[0222] Step g: determining the second loss value according to the brightness loss value and the color loss value.

[0223] As an example, the sum of the luminance loss value and the color loss value can be taken as the second loss value.

[0224] As another example, the luminance loss value and the color loss value can be weighted and summed to obtain the second loss value.

[0225] As yet another example, the mean of the luminance loss value and the color loss value can be taken as the second loss value.

[0226] Step S805, according to the first loss value and the second loss value, the image enhancement model is trained.

[0227] In the embodiments of the present application, first, the total loss value can be calculated according to the first loss value and the second loss value, which is denoted as the target loss value in the present application. The target loss value is positively correlated with the first loss value, and the target loss value is also positively correlated with the second loss value.

[0228] As an example, the sum of the first loss value and the second loss value can be taken as the target loss value.

[0229] As another example, the first loss value and the second loss value can be weighted and summed to obtain the target loss value.

[0230] As yet another example, the mean of the first loss value and the second loss value can be taken as the target loss value.

[0231] Exemplarily, the label migration image is I pred , the labeled image is I gt , and the target loss value is L, then:

[0232] L = L1(I perd ,I gt ) + L1(LAB(I perd ), LAB(I gt )) + L reg ;

[0233] Wherein, L1(I perd ,I gt ) refers to the luminance loss value, L1(LAB(I perd ), LAB(I gt )) refers to the color loss value, and L reg refers to the first loss value.

[0234] Then, the model parameters in the image color migration model can be adjusted based on the target loss value to minimize the target loss value.

[0235] It should be noted that the above only takes the termination condition of the model training as an example of target loss value minimization. In actual application, other termination conditions can also be set, such as the training duration reaching a set duration, the training round reaching a set round, and the like.

[0236] The training method of the image color transfer model provided in the embodiments of the present application can comprehensively train the image color transfer model by using multiple loss values, thereby improving the model performance and enhancing the generalization ability of the model.

[0237] The embodiments of the present application provide another training method of an image color transfer model. Figure 9 The fourth training method of an image color transfer model provided in the embodiments of the present application is shown in the flowchart.

[0238] It should be noted that the training method of the image color transfer model can be executed alone, or can be executed in combination with any embodiment or possible implementation manner in the embodiments of the present application, or can be executed in combination with any technical solution in the related art. The embodiments of the present application do not limit this.

[0239] As shown in Figure 9 The training method of the image color transfer model can include the following steps S901-S906:

[0240] Step S901, obtaining a sample image.

[0241] The method of obtaining the sample image is not limited, for example, the sample image can be obtained from an existing training set or test set, or the sample image can be collected through a camera in an electronic device, or the sample image can be artificially synthesized, and the like. The embodiments of the present application do not limit this.

[0242] Step S902, cropping a first image block and a second image block from the sample image; wherein the first image block is contained in the second image block, or the second image block is contained in the first image block.

[0243] In the embodiments of the present application, two image blocks can be randomly cropped from the sample image, one of which is completely contained in the other, to simulate the difference in the field of view (FOV) between different cameras, which are denoted as the first image block and the second image block in the present application.

[0244] As an example, the first image block is labeled as The second image block is labeled as

[0245] Step S903, performing a data augmentation operation on the first image block and the second image block to obtain the image data pair.

[0246] Among them, the image data pair includes a target image and a source image to be color migrated.

[0247] Among them, the data augmentation operation includes but is not limited to: random distortion operation, random noise operation, random blur operation, color parameter transformation operation (such as brightness color transformation operation) and the like.

[0248] In the embodiments of the present application, the data augmentation operation can be performed on the first image block and the second image block to obtain the target image and the source image in the image data pair.

[0249] In any one of the embodiments of the present application, the image data pair can be obtained by the following steps E to step H:

[0250] Step E: performing a first data augmentation operation on one of the first image block and the second image block to obtain a processed image block.

[0251] Among them, the first data augmentation operation includes at least one of the following operations: random distortion operation (for simulating lens distortion), random noise operation (for simulating noise difference between images) and random blur operation (for simulating clarity difference between images).

[0252] Step F: generating a first intermediate image and a second intermediate image according to the other one of the first image block and the second image block and the processed image block in step E.

[0253] As an example, mark the first image block as Mark the second image block as The first data augmentation operation can be performed on one of or , and the other one remains unchanged to obtain and Among them, if the first data augmentation operation is performed on , then and remain unchanged; if the first data augmentation operation is performed on , then and remain unchanged.

[0254] At this time, the first intermediate image can be , and the second intermediate image can be .

[0255] Step G: performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image; wherein the second data augmentation operation comprises a color parameter transformation operation.

[0256] As an example, the second data augmentation operation can be performed on the second intermediate image to obtain the target image, and the first intermediate image is taken as the source image.

[0257] At this time, in order to realize that the picture content of the labeled image is consistent with the picture content of the source image, and the color parameters of the labeled image match the color parameters of the target image, the same second data augmentation operation as the second intermediate image can be performed on the first intermediate image to obtain the labeled image.

[0258] As another example, the second data augmentation operation can be performed on the first intermediate image to obtain the source image, and the second intermediate image is taken as the target image.

[0259] At this time, in order to realize that the picture content of the labeled image is consistent with the picture content of the source image, and the color parameters of the labeled image match the color parameters of the target image, the first intermediate image can be directly taken as the labeled image.

[0260] Step H: generating an image data pair according to the source image and the target image, that is, the image data pair comprises the source image and the target image.

[0261] Therefore, a large amount of labeled training data can be automatically generated by using the data augmentation technology, and the data distribution close to the actual scene can be achieved without additional manual data labeling, thereby reducing the labeling cost. In addition, by performing random transformation, expansion and other operations on the training data, the image color transfer model can learn more different styles of data in the training process, thereby improving the generalization ability of the image color transfer model.

[0262] Step S904: processing the image data pair by using the image color transfer model to obtain a global mapping parameter.

[0263] The global mapping parameter is used to indicate the color mapping relationship between the source image and the target image.

[0264] Step S905: performing color transfer processing on the source image based on the global mapping parameter to obtain a transfer image.

[0265] Step S906: training the image color transfer model based on the transfer image and a labeled image associated with the image data pair.

[0266] The explanation and description of steps S904 to S906 can be referred to the related description in any embodiment of the present application, which will not be repeated here.

[0267] The training method of the image color migration model of the embodiment of the application can automatically generate training data through data enhancement or data augmentation technology, and can reduce the acquisition cost of training data.

[0268] In any one of the embodiments of the application, by constructing a deep learning model, for an input pair of images taken by different cameras, the brightness and color of one of the images are adjusted to be consistent with the other image, thereby maintaining the continuity of image quality when the camera switches, and improving the user experience in the shooting scene. It can be applied to the following fields: multi-camera switching image quality smoothness, scalable multi-camera fusion imaging, panoramic imaging, data stream switching image quality smoothness, etc.

[0269] The input of the scheme provided by the application is a pair of images taken by two cameras, denoted as source image I src and target image I dst Without loss of generality, taking the brightness and color of I dst as the standard, the output of the scheme provided by the application is an image consistent with the brightness and color of I dst , denoted as I pred . Corresponding to the camera switching process, I dst is the image before switching, and after consistent color migration, the image after switching changes from the original I src to I pred , thereby realizing the continuous transition of brightness and color and realizing image quality smoothness.

[0270] As an example, the acquisition process of I pred may be as shown in Figure 10 , mainly including: respectively inputting down-sampled I src and I dst to the image color migration model, predicting global mapping parameters (i.e. 3DLUT mapping parameters) by the model, and then applying the 3DLUT mapping parameters to I src , i.e. outputting the color migrated image, i.e. I pred .

[0271] The following will introduce the three core parts of the scheme provided by the application: training data, image color migration model and loss function.

[0272] First part: training data.

[0273] The scheme provided in the application can construct image data pairs as close to the actual scene as possible based on a public data set through a series of data augmentation operations, wherein the labeled image (Ground Truth) can be constructed through brightness color transformation. Assuming that a sample image I is obtained from a public data set, the training data is obtained by performing the following operations:

[0274] Data augmentation 1: randomly crop two image blocks on the sample image, wherein one image block is completely contained in the other image block, so as to simulate the FOV difference between different cameras. This step can obtain

[0275] Data augmentation 2: randomly distort or , randomly add noise to simulate the noise difference between images, and randomly blur to simulate the clarity difference between images. This series of steps can achieve the purpose of being closer to the actually captured image data. After these steps, can be obtained and

[0276] Data augmentation 3: perform the same brightness color transformation on and , to obtain and . In this way, the input image data pair can be constructed. The Ground Truth is . This step can also only perform brightness color transformation on . In this way, the input image data pair can be constructed. The Ground Truth is

[0277] Second part: image color migration model.

[0278] Since the output of the scheme provided in the application is an image, some generative models can be used in model design. However, such models are unstable in training, and the quality of the generated results is difficult to guarantee. The scheme provided in the application can refer to HDRNet, adaptive-3DLUT and other models, and the model directly predicts global mapping coefficients.

[0279] In order to improve the performance of the model, the model structure is mainly optimized from the following aspects:

[0280] (1) The application predicts a global 3DLUT, and the number of color blocks N of each color channel is 17. The total amount of mapping parameters is 3*17*17*17=14739. The scale of the mapping parameters is obviously reduced.

[0281] (2) Since the changes in FOV between images are taken into account, the present application requires a large receptive field, even a full-image receptive field. If a high-resolution large image is directly used to calculate the 3DLUT, the performance of the model is poor. Therefore, the present application can downsample the high-resolution large image to a low-resolution small image, and then input the small image into the model, so as to significantly improve the performance of the model while ensuring the effect.

[0282] (3) The model usually predicts the 3DLUT mapping parameters in the channel dimension. However, too many channels will cause a performance bottleneck. To solve this problem, the present application can predict the 3DLUT mapping parameters in the spatial dimension, which can greatly improve the performance of the model. For example, the size of the model input data is 1x6x136x136 (where 1 means that the batch size is 1, and two images are concatenated into 6 channels along the channel dimension), after three down-sampling convolution operations with stride = 2, the output size is 1x51x17x17 feature map, and then the reshape operator is used to obtain 1x3x17x17x17 3DLUT.

[0283] Third part: loss function.

[0284] The scheme provided by the present application can use an end-to-end manner to directly perform supervised learning on the final output image (I gt is the automatically generated labeled image, and I pred is the predicted image). The loss function includes three terms, and the L1 loss function is used to calculate the brightness difference of the three color channels of RGB. In order to provide color supervision information, the scheme provided by the present application converts the image from the RGB domain to the LAB domain and calculates the L1 loss of the color component. In addition, the scheme provided by the present application also uses a regularization loss function L reg to constrain the intermediate result 3DLUT, so as to improve the image quality after color transfer.

[0285] The regularization loss function L reg mainly includes the following three terms: a monotonicity loss function, which mainly prevents obvious brightness and color inversion; a smoothness loss function, which ensures that the learned 3DLUT brightness and color change is relatively smooth, and these two loss functions can effectively improve the problem of quality stratification; and a highlight and high saturation loss function, which can optimize the obvious artifact caused by excessive brightness and color offset in the high-light and high-saturation region.

[0286] Compared with the prior art, the scheme provided by the present application has at least the following advantages:

[0287] 1. Good independence and flexibility: The solution provided in this application can be implemented through a hardware 3DLUT module (such as ISP3DLUT), or it can be developed and tested independently of the hardware module in the form of a soft algorithm;

[0288] 2. High versatility: The solution provided in this application processes two images in real time, so it can be migrated to any terminal and any sensor, and has wide applicability;

[0289] 3. Strong generalization: The solution proposed in this application uses deep learning technology. By performing random transformations and amplification on the training data, the model can learn more diverse data during training, thereby improving the model's generalization ability.

[0290] 4. Strong robustness: The solution proposed in this application adopts an end-to-end learning approach, learning an abstract representation of data through multi-layer nonlinear transformations, which can better adapt to different data distributions and changes;

[0291] 5. Easy training without additional labeling costs: The technical solution provided in this application can automatically generate large-scale labeled training data through data augmentation, and the data distribution is as close as possible to the actual scene;

[0292] 6. High Deployability: The solution provided in this application is based on 3DLUT implementation, and the total number of mapping parameters is relatively small. In addition, under the premise of ensuring the effect, the model structure is optimized through small graph calculation, spatial dimension prediction parameters, etc., so that the model has good performance and low power consumption, which is conducive to actual deployment;

[0293] 7. Better mapping effect: The solution provided in this application proposes a loss function for 3DLUT, which can effectively improve image quality problems such as image stratification and artifacts, and the image quality after mapping is better.

[0294] In order to implement the above embodiment, the embodiment of the present application further proposes an image processing device.

[0295] Figure 11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0296] like Figure 11 As shown, the image processing apparatus 1100 may include: an acquisition module 1110 and a migration module 1120 .

[0297] The acquisition module 1110 is configured to acquire global mapping parameters in response to a switch from a first camera to a second camera, or simultaneous shooting by the first camera and the second camera in the electronic device; wherein the global mapping parameters are used to indicate a color mapping relationship between a first image captured by the first camera and a second image captured by the second camera;

[0298] migrate the second image to obtain a target image matching the color parameters of the first image.

[0299] Further, in an implementation form of the embodiment, the obtaining module 1110 is configured to: perform down-sampling processing on the first image to obtain a first sample image, and perform down-sampling processing on the second image to obtain a second sample image; splice the first sample image and the second sample image along a color channel dimension to obtain a spliced image; and predict the global mapping parameter according to the spliced image.

[0300] In an implementation form of the embodiment, the obtaining module 1110 is configured to: perform spatial dimension feature extraction on the spliced image by using a feature extraction network in the image color migration model to obtain a target feature map; and predict the global mapping parameter according to the target feature map by using a prediction network in the image color migration model.

[0301] In an implementation form of the embodiment, the obtaining module 1110 is configured to: perform spatial dimension down-sampling and convolution processing on an input feature map of a first convolution layer in the multi-layer convolution layer to obtain an output feature map of the first convolution layer, wherein the input feature map of the first convolution layer is generated according to the spliced image; perform spatial dimension down-sampling and convolution processing on an input feature map of an i-th convolution layer in the multi-layer convolution layer to obtain an output feature map of the i-th convolution layer, wherein i is a positive integer greater than 1, the input feature map is obtained by fusing the output feature map of the (i-1)-th convolution layer and the input feature map, and the output feature map of the last convolution layer in the multi-layer convolution layer is taken as the target feature map.

[0302] In an implementation form of the embodiment, the global mapping parameter is in a form of a three-dimensional lookup table (3DLUT), and the obtaining module 1110 is configured to: perform feature remodeling on the target feature map according to a number of color blocks contained in each color channel in the 3DLUT by using the prediction network to obtain the global mapping parameter.

[0303] In an implementation form of the embodiment, the global mapping parameter includes a plurality of mapping parameters in the 3DLUT, and each mapping parameter includes a mapping value of a color block on each color channel; and the migrating module 1120 is configured to: for any pixel point in the second image, query the plurality of mapping parameters in the 3DLUT according to original parameters of the any pixel point to obtain candidate parameters with minimum and / or adjacent distances from the original parameters, wherein the original parameters include channel values of a plurality of color channels of the any pixel point; and perform linear interpolation on the candidate parameters to obtain target parameters of a pixel point corresponding to the any pixel point in the target image.

[0304] It should be noted that the foregoing explanation of the image processing method embodiment is also applicable to the image processing device of the embodiment, which will not be described here.

[0305] In the image processing device of the embodiment, in the case of switching from the first camera to the second camera or simultaneously shooting with both cameras, the global mapping parameters are predicted according to the first image collected by the first camera and the second image collected by the second camera, wherein all the pixel points in the entire second image share the set of global mapping parameters. For high-resolution images, the parameter amount and the calculation demand can be greatly reduced, so that the processing process is more efficient, and the color migration speed of the image is significantly improved. With the reduction of the parameter amount, not only the demand for calculation resources is reduced, but also the requirement for storage space is reduced, which is particularly important for applications in mobile devices or other resource-constrained environments, and helps to prolong the battery life and improve the overall performance of the device. At the same time, the global mapping parameters provide a unified color mapping method, ensuring that the images obtained when switching from the first camera to the second camera or simultaneously shooting with both cameras have higher color consistency, avoiding the discontinuity problem that may occur in pixel-by-pixel prediction, thereby improving the consistency and stability of the image quality after color migration.

[0306] To implement the above embodiment, the embodiment of the present application also proposes a training device of an image color migration model.

[0307] Figure 12 A structural schematic diagram of the training device of the image color migration model provided by the embodiment of the present application.

[0308] As Figure 12 shown, the training device 1200 of the image color migration model can include an acquisition module 1210, a processing module 1220, a migration module 1230, and a training module 1240.

[0309] The acquisition module 1210 is configured to acquire an image data pair, wherein the image data pair includes a target image and a source image to be color migrated.

[0310] The processing module 1220 is configured to process the image data pair by using the image color migration model to obtain global mapping parameters, wherein the global mapping parameters are used to indicate the color mapping relationship between the source image and the target image.

[0311] The migration module 1230 is configured to perform color migration processing on the source image based on the global mapping parameters to obtain a migration image.

[0312] The training module 1240 is configured to train the image color migration model based on the migration image and a labeled image associated with the image data pair.

[0313] Further, in an implementation form of the embodiment of the application, the processing module 1220 is configured to: perform down-sampling processing on the source image and the target image respectively to obtain a source sampling image and a target sampling image; splice the source sampling image and the target sampling image along a color channel dimension to obtain a spliced image; and perform processing on the spliced image by using the image color migration model to obtain the global mapping parameter.

[0314] In an implementation form of the embodiment of the application, the global mapping parameter includes a plurality of mapping parameters in the 3DLUT, and the mapping parameter includes a mapping value of a color block on each color channel. The training module 1240 is configured to: generate a first loss value according to mapping values of adjacent color blocks on each color channel in the global mapping parameter; determine a second loss value according to a color difference between the migration image and the labeled image; and train the image enhancement model according to the first loss value and the second loss value.

[0315] In an implementation form of the embodiment of the application, the training module 1240 is configured to: determine a monotonicity loss value according to a degree of non-monotonicity between mapping values of adjacent color blocks on each color channel; determine a smoothness loss value according to a difference between mapping values of adjacent color blocks on each color channel; determine a highlight and high saturation loss value according to a color difference between a target region in the migration image and a corresponding region in the source image, wherein the brightness of the target region is higher than a first threshold value, and / or the saturation of the target region is higher than a second threshold value; and generate the first loss value according to at least one of the monotonicity loss value, the smoothness loss value and the highlight and high saturation loss value.

[0316] In an implementation form of the embodiment of the application, the training module 1240 is configured to: generate a brightness loss value according to a brightness difference between the migration image and the labeled image; generate a color loss value according to a color difference between the migration image and the labeled image; and determine the second loss value according to the brightness loss value and the color loss value.

[0317] In an implementation form of the embodiment of the application, the training module 1240 is configured to: convert the migration image from a first color space to a second color space corresponding to brightness to obtain a first converted image; convert the labeled image from the first color space to the second color space to obtain a second converted image; and generate the color loss value according to a color difference between the first converted image and the second converted image.

[0318] In an implementation form of the embodiment of the application, the obtaining module 1210 is configured to: obtain a sample image; crop a first image block and a second image block from the sample image; wherein the first image block is contained in the second image block, or the second image block is contained in the first image block; and perform a data augmentation operation on the first image block and the second image block to obtain an image data pair.

[0319] In an implementation form of the embodiment of the application, the obtaining module 1210 is configured to: perform a first data augmentation operation on one of the first image block and the second image block to obtain a processed image block; wherein the first data augmentation operation comprises at least one of the following operations: a random distortion operation, a random noise adding operation, and a random blur operation; generate the first intermediate image and the second intermediate image according to the other of the first image block and the second image block and the processed image block; perform a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image; wherein the second data augmentation operation comprises a color parameter transformation operation; and generate the image data pair according to the source image and the target image.

[0320] In an implementation form of the embodiment of the application, the obtaining module 1210 is configured to: take the first intermediate image as the source image; perform the second data augmentation operation on the second intermediate image to obtain the target image; and perform the second data augmentation operation on the first intermediate image to obtain the labeled image.

[0321] In an implementation form of the embodiment of the application, the obtaining module 1210 is configured to: take the second intermediate image as the target image; perform the second data augmentation operation on the first intermediate image to obtain the source image; and take the first intermediate image as the labeled image.

[0322] It should be noted that the foregoing explanation of the training method of the image color transfer model also applies to the training device of the image color transfer model of this embodiment, which will not be described here.

[0323] In the training device of the image color transfer model of the embodiment of the application, the global mapping parameter is predicted based on the deep learning technology, and the color transfer processing is performed on the source image based on the global mapping parameter to obtain the source image after color adjustment (referred to as a transfer image in the application), so that the image color transfer model is trained based on the difference (such as the color difference) between the transfer image and the labeled image (which is consistent with the picture content of the source image and consistent with the color parameters of the target image), which can make the model learn the actual color mapping relationship between the source image and the target image, so that the color transfer can be performed on the source image based on the color mapping relationship, which can ensure that the source image after color transfer is consistent with the target image in color parameters, and then the trained image color transfer model is applied to the multi-camera switching scene or the multi-camera shooting scene, which can maintain the continuity and consistency of the image quality and improve the user experience in the shooting scene.

[0324] To achieve the above-mentioned embodiments, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training method of the image color transfer model or the image processing method according to any one of the preceding method embodiments.

[0325] Figure 13 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 13. For example, the electronic device 1300 can be a vehicle, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0326] Reference Figure 13 The electronic device 1300 can include one or more of the following components: a processing component 1302, a memory 1304, a power component 1306, a multimedia component 1308, an audio component 1310, an input / output (I / O) interface 1312, a sensor component 1314, and a communication component 1316.

[0327] The processing component 1302 usually controls overall operations of the electronic device 1300, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 1302 can include one or more processors 1320 to execute instructions to complete all or part of steps of the methods described above. Further, the processing component 1302 can include one or more modules to facilitate the interaction between the processing component 1302 and other components. For example, the processing component 1302 can include a multimedia module to facilitate the interaction between the multimedia component 1308 and the processing component 1302.

[0328] The memory 1304 is configured to store various types of data to support operations of the electronic device 1300. Examples of the data include instructions for any application or method operating on the electronic device 1300, contact data, phonebook data, messages, pictures, videos, etc. The memory 1304 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0329] The power component 1306 provides power to various components of the electronic device 1300. The power component 1306 can include a power management system, one or more power sources, and other components associated with generating, managing and distributing power for the electronic device 1300.

[0330] The multimedia component 1308 includes a screen to provide an output interface between the electronic device 1300 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 1308 includes a front camera and / or a rear camera. When the electronic device 1300 is in an operating mode, such as a capturing mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0331] The audio component 1310 is configured to output and / or input an audio signal. For example, the audio component 1310 includes a microphone (MIC) to receive an external audio signal when the electronic device 1300 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 1304 or transmitted via the communication component 1316. In some embodiments, the audio component 1310 further includes a speaker to output an audio signal.

[0332] The I / O interface 1312 provides an interface between the processing component 1302 and a peripheral interface module, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0333] The sensor component 1314 includes one or more sensors for providing status assessments for various aspects of the electronic device 1300. For example, the sensor component 1314 can detect an open / closed position of the electronic device 1300, relative positioning of components of the electronic device 1300, such as a display and a keypad of the electronic device 1300, a change in position of the electronic device 1300 or a component of the electronic device 1300, presence or absence of user contact with the electronic device 1300, orientation or acceleration / deceleration / g-force and temperature changes of the electronic device 1300. The sensor component 1314 can include an optical sensor for detecting ambient light, a proximity sensor for detecting nearby objects without any physical touch, a CMOS or CCD image sensor for use in imaging applications, and / or a gyroscope sensor, a magnetometer sensor, a pressure sensor, or a temperature sensor in some embodiments.

[0334] The communication component 1316 is configured to facilitate wired or wireless communication between the electronic device 1300 and other devices. The electronic device 1300 can access a wireless network based on a communication standard, such as WiFi, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 1316 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0335] In an example embodiment, the electronic device 1300 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements to perform the above-described methods.

[0336] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 1304 including instructions, is also provided. The instructions can be executed by the processor 1320 of the electronic device 1300 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0337] To achieve the above-mentioned embodiments, the present application further provides a chip, comprising: the chip comprises an interface circuit and a processing circuit coupled with each other, the interface circuit is configured to input or output signals, and the processing circuit is configured to execute the training method of the image color transfer model or the image processing method provided in any one of the preceding embodiments.

[0338] Figure 14 is a structural schematic diagram of a chip provided in an embodiment of the present application. Referring to Figure 14 the structural schematic diagram of the chip 1400 shown in FIG. 14, but the present application is not limited thereto.

[0339] The chip 1400 comprises a processing circuit 1401, which is configured to execute any one of the training methods of the image color transfer model or the image processing methods described above.

[0340] In some embodiments, the chip 1400 further comprises one or more interface circuits 1402. Optionally, the interface circuit 1402 is connected with the memory 1403, and the interface circuit 1402 can be configured to receive signals from the memory 1403 or other devices, and the interface circuit 1402 can be configured to send signals to the memory 1403 or other devices. For example, the interface circuit 1402 can read instructions stored in the memory 1403 and send the instructions to the processing circuit 1401.

[0341] In some embodiments, the interface circuit 1402 performs at least one of the communication steps such as sending and / or receiving in the above-mentioned methods, and the processing circuit 1401 performs other steps.

[0342] In some embodiments, the terms such as interface circuit, interface, transceiver pin, and transceiver can be replaced with each other.

[0343] In some embodiments, the chip 1400 further comprises one or more memories 1403 for storing instructions. Optionally, all or part of the memory 1403 can be outside the chip 1400.

[0344] To achieve the above-mentioned embodiments, the present application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the training method of the image color transfer model or the image processing method described in any one of the preceding method embodiments.

[0345] To achieve the above-mentioned embodiments, the present application further provides a computer program product, which stores a computer program, and the computer program is executed by a processor to implement the training method of the image color transfer model or the image processing method described in any one of the preceding method embodiments.

[0346] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Moreover, the usage of the terms "first", "second" or "third" does not limit the quantity or order of the specific features, structures, materials or characteristics, but rather the term "first", "second" or "third" can be used to distinguish different features, structures, materials or characteristics, which can be combined in any suitable manner. Furthermore, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise.

[0347] Furthermore, the terms "first", "second", or the like, merely denote different instances of a similar feature, structure, material or characteristic, without necessarily implying any relative importance or any particular order. Thus, a feature defined with "first" or "second" can implicitly or explicitly include at least one of the features. The meaning of "a", "an" and "the" includes plural references unless the context clearly dictates otherwise.

[0348] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments which can be managed as one or more modules, segments, or portions of code which include one or more steps for implementing specific logic functions or steps, and the terms in the description are used for causing or carrying out or upgrading of an action between other hardware under their control. The description of processes and methods of operations should be considered as merely illustrative of the principles of the application, and not in limitation of its scope.

[0349] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer readable medium can specifically include the following, which are non-exhaustive list: electrical connection (electrical device having one or more wires), portable computer diskette (magnetic device), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber device, and portable compact disc read-only memory (CDROM). In addition, the computer readable medium can even be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, for example, by optically scanning the paper or other suitable medium, then electronically converted into a form that can be edited, compiled, or interpreted, or otherwise processed in electronic form into an executable form suitable for use in the instruction execution system, apparatus or device.

[0350] It should be understood that parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies known in the art or their combination can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0351] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0352] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0353] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that: include: In response to a first camera in the electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously, obtaining a global mapping parameter; wherein the global mapping parameter is used to indicate a color mapping relationship between a first image captured by the first camera and a second image captured by the second camera; The global mapping parameters are used to perform color migration processing on the second image to obtain a target image that matches the color parameters of the first image.

2. The method according to claim 1, characterized in that The obtaining of global mapping parameters includes: downsampling the first image to obtain a first sampled image, and downsampling the second image to obtain a second sampled image; splicing the first sampled image and the second sampled image along a color channel dimension to obtain a spliced ​​image; The global mapping parameters are predicted based on the stitched image.

3. The method according to claim 2, characterized in that The predicting the global mapping parameter according to the spliced ​​image includes: Using a feature extraction network in an image color transfer model to extract features of the spliced ​​image in a spatial dimension, to obtain a target feature map; The prediction network in the image color transfer model is used to predict the global mapping parameters according to the target feature map.

4. The method according to claim 3, characterized in that The feature extraction network includes multiple convolutional layers. The feature extraction network in the image color migration model is used to extract features of the spliced ​​image in the spatial dimension to obtain a target feature map, including: Using the first convolutional layer in the multi-layer convolutional layer, downsampling and convolution processing are performed on the input feature map of the first convolutional layer in the spatial dimension to obtain the output feature map of the first convolutional layer; wherein the input feature map of the first convolutional layer is generated according to the spliced ​​image; Using the i-th convolutional layer in the multi-layer convolutional layer, downsampling and convolution processing are performed on the input feature map of the i-th convolutional layer in the spatial dimension to obtain the output feature map of the i-th convolutional layer; wherein i is a positive integer greater than 1, and the input feature map is obtained by fusing the output feature map of the i-1-th convolutional layer and the input feature map; The output feature map of the last convolutional layer in the multi-layer convolutional layers is used as the target feature map.

5. The method according to claim 3, characterized in that The global mapping parameters are in the form of a three-dimensional lookup table 3DLUT, The predicting of the global mapping parameters using the prediction network in the image color migration model according to the target feature map includes: The prediction network is used to reshape the target feature map according to the number of color blocks contained in each color channel in the 3DLUT to obtain the global mapping parameters.

6. The method according to any one of claims 1 to 5, characterized in that The global mapping parameters include a plurality of mapping parameters in the 3DLUT, and the mapping parameters include a mapping value of a color block on each color channel; The step of performing color migration processing on the second image using the global mapping parameters to obtain a target image matching the color parameters of the first image includes: For any pixel in the second image, query multiple mapping parameters in the 3DLUT based on original parameters of the pixel to obtain candidate parameters that have the shortest distance to and / or are adjacent to the original parameters; wherein the original parameters include channel values ​​of multiple color channels of the pixel; Linear interpolation is performed on the candidate parameters to obtain target parameters of the pixel points corresponding to the any pixel point in the target image.

7. A training method for an image color transfer model, characterized in that: include: Acquire an image data pair; wherein the image data pair includes a target image and a source image to be color migrated; Processing the image data pair using an image color migration model to obtain global mapping parameters; wherein the global mapping parameters are used to indicate a color mapping relationship between the source image and the target image; Based on the global mapping parameters, performing color migration processing on the source image to obtain a migrated image; The image color transfer model is trained based on the transfer image and the annotated image associated with the image data pair.

8. The method according to claim 7, characterized in that The adopting the image color migration model to process the image data pair to obtain global mapping parameters includes: Downsampling the source image and the target image to obtain a source sampling image and a target sampling image respectively; Splicing the source sampled image and the target sampled image along the color channel dimension to obtain a spliced ​​image; The image color migration model is used to process the stitched image to obtain the global mapping parameters.

9. The method according to claim 7, characterized in that The global mapping parameters include multiple mapping parameters in the 3DLUT, and the mapping parameters include a mapping value of a color block on each color channel. The training of the image enhancement model based on the migration image and the annotated image associated with the image data pair includes: generating a first loss value according to mapping values ​​of adjacent color blocks on each color channel in the global mapping parameter; determining a second loss value based on a color difference between the migrated image and the annotated image; The image enhancement model is trained according to the first loss value and the second loss value.

10. The method according to claim 9, characterized in that Generating a first loss value according to mapping values ​​of adjacent color blocks on each color channel in the global mapping parameter includes: Determining a monotonicity loss value according to a degree of non-monotonicity between mapping values ​​of adjacent color blocks on each color channel; Determining a smoothness loss value according to a difference between mapping values ​​of adjacent color blocks on each color channel; Determining a highlight-high saturation loss value based on a color difference between a target area in the migration image and a corresponding area in the source image; wherein the brightness of the target area is higher than a first threshold, and / or the saturation of the target area is higher than a second threshold; The first loss value is generated according to at least one of the monotonicity loss value, the smoothness loss value, and the highlight-high saturation loss value.

11. The method according to claim 9, characterized in that The determining a second loss value according to a color difference between the migrated image and the annotated image includes: generating a brightness loss value according to a brightness difference between the migrated image and the annotated image; generating a color loss value according to a color difference between the migrated image and the annotated image; The second loss value is determined according to the brightness loss value and the color loss value.

12. The method according to claim 11, characterized in that Generating a color loss value according to a color difference between the migrated image and the annotated image includes: Converting the migration image from the first color space to a second color space corresponding to brightness to obtain a first converted image; Converting the annotated image from the first color space to the second color space to obtain a second converted image; A color loss value is generated according to a color difference between the first converted image and the second converted image.

13. The method according to any one of claims 7 to 12, characterized in that The acquiring of the image data pair comprises: Get a sample image; Cropping the sample image to obtain a first image block and a second image block; wherein the first image block is contained in the second image block, or the second image block is contained in the first image block; A data augmentation operation is performed on the first image block and the second image block to obtain the image data pair.

14. The method according to claim 13, characterized in that The performing a data augmentation operation on the first image block and the second image block to obtain the image data pair includes: Performing a first data augmentation operation on one of the first image block and the second image block to obtain a processed image block; wherein the first data augmentation operation includes at least one of the following operations: a random distortion operation, a random noise addition operation, and a random blurring operation; generating a first intermediate image and a second intermediate image according to the other of the first image block and the second image block and the processed image block; performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image; wherein the second data augmentation operation includes a color parameter transformation operation; The image data pair is generated according to the source image and the target image.

15. The method according to claim 14, characterized in that The performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image includes: Using the first intermediate image as the source image; performing a second data augmentation operation on the second intermediate image to obtain the target image; The annotated image is obtained by following the steps below: The second data augmentation operation is performed on the first intermediate image to obtain the labeled image.

16. The method according to claim 14, characterized in that The performing a second data augmentation operation on the first intermediate image or the second intermediate image to obtain the source image and the target image includes: Using the second intermediate image as the target image; performing a second data augmentation operation on the first intermediate image to obtain the source image; The annotated image is obtained by following the steps below: The first intermediate image is used as the labeled image.

17. An image processing device, characterized in that: include: an acquisition module, configured to acquire global mapping parameters in response to a switch from a first camera to a second camera in the electronic device, or a simultaneous capture by the first camera and the second camera; wherein the global mapping parameters are used to indicate a color mapping relationship between a first image captured by the first camera and a second image captured by the second camera; A migration module is used to perform color migration processing on the second image using the global mapping parameters to obtain a target image that matches the color parameters of the first image.

18. The device according to claim 17, characterized in that The acquisition module is used to: downsampling the first image to obtain a first sampled image, and downsampling the second image to obtain a second sampled image; splicing the first sampled image and the second sampled image along a color channel dimension to obtain a spliced ​​image; The global mapping parameters are predicted based on the stitched image.

19. The device according to claim 18, characterized in that The acquisition module is used to: Using a feature extraction network in an image color transfer model to extract features of the spliced ​​image in a spatial dimension, to obtain a target feature map; The prediction network in the image color transfer model is used to predict the global mapping parameters according to the target feature map.

20. The device according to claim 19, characterized in that The feature extraction network includes multiple convolutional layers, and the acquisition module is used to: Using the first convolutional layer in the multi-layer convolutional layer, downsampling and convolution processing are performed on the input feature map of the first convolutional layer in the spatial dimension to obtain the output feature map of the first convolutional layer; wherein the input feature map of the first convolutional layer is generated according to the spliced ​​image; Using the i-th convolutional layer in the multi-layer convolutional layer, downsampling and convolution processing are performed on the input feature map of the i-th convolutional layer in the spatial dimension to obtain the output feature map of the i-th convolutional layer; wherein i is a positive integer greater than 1, and the input feature map is obtained by fusing the output feature map of the i-1-th convolutional layer and the input feature map; The output feature map of the last convolutional layer in the multi-layer convolutional layers is used as the target feature map.

21. The device according to claim 19, characterized in that The global mapping parameters are in the form of a three-dimensional lookup table 3DLUT, The acquisition module is used to: The prediction network is used to reshape the target feature map according to the number of color blocks contained in each color channel in the 3DLUT to obtain the global mapping parameters.

22. A training device for an image color transfer model, characterized in that: include: An acquisition module, configured to acquire an image data pair, wherein the image data pair includes a target image and a source image to be color migrated; a processing module, configured to process the image data pair using an image color migration model to obtain global mapping parameters; wherein the global mapping parameters are used to indicate a color mapping relationship between the source image and the target image; A migration module, configured to perform color migration processing on the source image based on the global mapping parameters to obtain a migrated image; A training module is used to train the image color migration model based on the migration image and the labeled image associated with the image data pair.

23. The device according to claim 22, characterized in that The processing module is used to: Downsampling the source image and the target image to obtain a source sampling image and a target sampling image respectively; Splicing the source sampled image and the target sampled image along the color channel dimension to obtain a spliced ​​image; The image color migration model is used to process the stitched image to obtain the global mapping parameters.

24. The device according to claim 22, characterized in that The global mapping parameters include multiple mapping parameters in the 3DLUT, and the mapping parameters include a mapping value of a color block on each color channel. The training module is used to: generating a first loss value according to mapping values ​​of adjacent color blocks on each color channel in the global mapping parameter; determining a second loss value based on a color difference between the migrated image and the annotated image; The image enhancement model is trained according to the first loss value and the second loss value.

25. The device according to claim 24, characterized in that The training module is used to: Determining a monotonicity loss value according to a degree of non-monotonicity between mapping values ​​of adjacent color blocks on each color channel; Determining a smoothness loss value according to a difference between mapping values ​​of adjacent color blocks on each color channel; Determining a highlight-high saturation loss value based on a color difference between a target area in the migration image and a corresponding area in the source image; wherein the brightness of the target area is higher than a first threshold, and / or the saturation of the target area is higher than a second threshold; The first loss value is generated according to at least one of the monotonicity loss value, the smoothness loss value, and the highlight-high saturation loss value.

26. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented, or the method according to any one of claims 7 to 16 is implemented.

27. A chip, characterized in that: The chip includes an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output signals, and the processing circuit is used to implement the method according to any one of claims 1 to 6, or the method according to any one of claims 7 to 16.

28. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented, or the method according to any one of claims 7 to 16 is implemented.

29. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6, or implements the method according to any one of claims 7 to 16.

Images