Image processing method and device, model training method, device and storage medium

Abstract

The present disclosure relates to an image processing method and device, a model training method, device and storage medium. The image processing method comprises: obtaining an image to be processed; inputting the image to be processed into a de-colorization processing model to obtain a target image output by the de-colorization processing model; wherein the de-colorization processing model is obtained by training a training data set, and the training data set includes multiple groups of sample data, each group of sample data includes a color-free image and a color-difference image corresponding to the color-free image. By adopting the method disclosed in the present disclosure, the color difference in the image can be quickly removed to obtain a clean color-free target image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, a model training method, device and storage medium. Background Art

[0002] As we all know, the chromatic aberration of the camera lens will cause the edge of the object in the captured image to appear a color that does not belong to the object. We call the color that appears at the edge of the object chromatic aberration color fringing (i.e., an optical chromatic aberration phenomenon). For example, when using digital imaging devices such as digital cameras and surveillance cameras to shoot high dynamic range (HDR) scene images under backlight and large aperture conditions, the edges of the high-contrast areas in the obtained images are prone to purple fringing, blue fringing, red fringing, green fringing, and other chromatic aberration color fringing. The reason for the formation of chromatic aberration color fringing can be considered that the camera lens can only accurately focus on one wavelength of light, but cannot accurately focus on other wavelengths of light.

[0003] In the related art, the chromatic aberration and color fringing phenomenon can be weakened by improving the material of the optical lens or performing a coating process on the optical lens, but this will greatly increase the hardware cost of the camera, and the suppression effect of chromatic aberration and color fringing is poor. In recent years, the method of removing chromatic aberration and color fringing in images by using image processing algorithms has attracted more and more attention due to its low cost and good effect. Specifically, the method of removing chromatic aberration and color fringing in images by using image processing algorithms can be divided into three steps: the first step is to perform edge detection of high-contrast areas (object edge detection) on the image. Since chromatic aberration and color fringing usually appear at the edge of high-contrast areas, the location where chromatic aberration and color fringing may appear can be determined by detecting the edge of high-contrast areas. The second step is to perform color difference judgment on the pixel points at the edge of the object, that is, after finding the edge of the high-contrast area, it is necessary to perform color difference judgment on the color of the edge to exclude non-chromatic aberration edges. The third step is to perform color difference suppression or color restoration processing on the pixel points with color difference at the edge of the object to restore the original color of the edge.

[0004] However, each step in the above-mentioned method of removing chromatic aberration and color fringing in an image using an image processing algorithm may result in misjudgment, thereby affecting the final effect of removing chromatic aberration and color fringing. For example, when an object edge is detected incorrectly, the object edge that originally had chromatic aberration and color fringing may not be determined as an object edge, resulting in the chromatic aberration suppression and color restoration processing of the chromatic aberration and color fringing, resulting in a poor effect of removing chromatic aberration and color fringing. For another example, when chromatic aberration is suppressed for chromatic aberration and color fringing is performed, color loss may be caused, and when color restoration is performed for chromatic aberration and color fringing, new erroneous colors may be introduced to form new chromatic aberration and color fringing, which may also result in a poor effect of removing chromatic aberration and color fringing. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method and device, a model training method, device and storage medium, so as to provide a method different from the related art to remove chromatic aberration and color fringing in images and improve the effect of removing chromatic aberration and color fringing in images.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0007] Get the image to be processed;

[0008] Inputting the image to be processed into a de-chromatization processing model to obtain a target image output by the de-chromatization processing model;

[0009] The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, and each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image.

[0010] Optionally, the image with color difference corresponding to the image without color difference is obtained by:

[0011] Separating the brightness signal and the chrominance signal of the image without color difference to obtain a brightness image corresponding to the brightness signal of the image without color difference, and obtaining a chrominance image corresponding to the chrominance signal of the image without color difference;

[0012] Determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image to obtain a new chromaticity image;

[0013] The brightness image and the new chrominance image are synthesized to obtain the image with color difference corresponding to the image without color difference.

[0014] Optionally, the chromatic aberration removal model is obtained by the following training method:

[0015] According to the training data set and the preset loss function, the deep convolutional neural network model is trained to obtain the de-colorization processing model, and the preset loss function is:

[0016] g = loss(y,y_) = c × |y-y_|;

[0017] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0018] Optionally, the chromatic aberration removal processing model includes a preprocessing module, a gaze mechanism module, and a synthesis module; the step of inputting the image to be processed into the chromatic aberration removal processing model to obtain a target image output by the chromatic aberration removal processing model includes:

[0019] Input the image to be processed into the preprocessing module, the preprocessing module separates the brightness signal and the chrominance signal in the image to be processed, obtains a brightness image to be processed corresponding to the brightness signal of the image to be processed, and obtains a chrominance image to be processed corresponding to the chrominance signal of the image to be processed, and inputs the brightness image to be processed and the chrominance image to be processed into the gaze mechanism module;

[0020] The gaze mechanism module calculates the color difference correction value according to the object edge feature information in the brightness image to be processed and the chromaticity image to be processed, obtains the correction information of the chromaticity image to be processed, and inputs the chromaticity image to be processed, the brightness image to be processed, and the correction information of the chromaticity image to be processed into the synthesis module;

[0021] The synthesis module corrects the chromaticity image to be processed according to the correction information of the chromaticity image to be processed to obtain a corrected chromaticity image to be processed, synthesizes the corrected chromaticity image to be processed and the luminance image to be processed into the target image, and outputs the target image.

[0022] Optionally, the gaze mechanism module includes a chromaticity information convolution layer, a chromaticity information activation layer, a chromaticity information multiplication layer and a chromaticity information merging processing layer connected in series in sequence;

[0023] The chromaticity information convolution layer is used to perform convolution processing on the input data of the gaze mechanism module to obtain a first chromaticity image feature;

[0024] The chromaticity information activation layer is used to activate the first chromaticity image feature to obtain a second chromaticity image feature;

[0025] The chromaticity information multiplication layer is used to perform a co-occurrence multiplication process on the input data of the gaze mechanism module and the second chromaticity image feature to obtain the third chromaticity image feature;

[0026] The chromaticity information merging processing layer is used to merge the third chromaticity image feature with the object edge feature information in the brightness image to be processed to obtain the chromaticity image correction information to be processed.

[0027] Optionally, the gaze mechanism module is a plurality of gaze mechanism modules connected in series, the input data of the first gaze mechanism module among the plurality of gaze mechanism modules is the chromaticity image to be processed, the input data of the non-first gaze mechanism module among the plurality of gaze mechanism modules is the output data of the previous gaze mechanism module, and the chromaticity image correction information to be processed is the output data of the last gaze mechanism module among the plurality of gaze mechanism modules.

[0028] Optionally, before acquiring the image to be processed, the method includes:

[0029] In response to an input operation of a user, acquiring an image input by the user;

[0030] In the case that the image format of the image input by the user is in RGB format, the image input by the user is format converted to obtain the image to be processed in YUV format.

[0031] Optionally, the image format of the target image is the YUV format, and after obtaining the target image output by the de-colorization processing model, the method includes:

[0032] Convert the target image to RGB format.

[0033] The target image in RGB format is fed back to the user's terminal.

[0034] According to a second aspect of an embodiment of the present disclosure, a method for training a chromatic aberration removal model is provided, the method comprising:

[0035] Acquire one or more images without color difference, and construct corresponding images with color difference according to the images without color difference;

[0036] Generate a training data set based on one or more of the images without color difference and the corresponding images with color difference;

[0037] The deep convolutional neural network model is trained according to the training data set to obtain the chromatic aberration removal processing model.

[0038] Optionally, constructing a corresponding image with chromatic aberration according to the image without chromatic aberration includes:

[0039] Separating the brightness signal and the chrominance signal of the image without color difference to obtain a brightness image corresponding to the brightness signal of the image without color difference, and obtaining a chrominance image corresponding to the chrominance signal of the image without color difference;

[0040] Determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image to obtain a new chromaticity image;

[0041] The brightness image and the new chrominance image are synthesized to obtain the image with color difference corresponding to the image without color difference.

[0042] Optionally, the training of a deep convolutional neural network model according to the training data set to obtain the chromatic aberration removal processing model includes:

[0043] According to the training data set and the preset loss function, the deep convolutional neural network model is trained to obtain the de-colorization processing model, and the preset loss function is:

[0044] g = loss(y,y_) = c × |y-y_|;

[0045] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0046] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0047] An acquisition module, configured to acquire an image to be processed;

[0048] An input module is configured to input the image to be processed into a chromatic aberration processing model to obtain a target image output by the chromatic aberration processing model;

[0049] The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, and each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image.

[0050] Optionally, the image with color difference corresponding to the image without color difference is obtained by:

[0051] Separating the brightness signal and the chrominance signal of the image without color difference to obtain a brightness image corresponding to the brightness signal of the image without color difference, and obtaining a chrominance image corresponding to the chrominance signal of the image without color difference;

[0052] Determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image to obtain a new chromaticity image;

[0053] The brightness image and the new chrominance image are synthesized to obtain the image with color difference corresponding to the image without color difference.

[0054] Optionally, the device further comprises a training module configured to obtain the chromatic aberration removal processing model through training in the following manner:

[0055] According to the training data set and the preset loss function, the deep convolutional neural network model is trained to obtain the de-colorization processing model, and the preset loss function is:

[0056] g = loss(y,y_) = c × |y-y_|;

[0057] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0058] Optionally, the de-colorization processing model includes a pre-processing module, a gaze mechanism module, and a synthesis module; and the input module is used for:

[0059] The image to be processed is input into the preprocessing module, and the preprocessing module separates the brightness signal and the chrominance signal in the image to be processed to obtain a brightness image to be processed corresponding to the brightness signal of the image to be processed, and obtains a chrominance image to be processed corresponding to the chrominance signal of the image to be processed, and inputs the brightness image to be processed and the chrominance image to be processed into the gaze mechanism module; the gaze mechanism module calculates the color difference correction value according to the object edge feature information in the brightness image to be processed and the chrominance image to be processed to obtain the correction information of the chrominance image to be processed, and inputs the chrominance image to be processed, the brightness image to be processed and the correction information of the chrominance image to be processed into the synthesis module; the synthesis module corrects the chrominance image to be processed according to the correction information of the chrominance image to be processed to obtain a corrected chrominance image to be processed, and synthesizes the corrected chrominance image to be processed with the brightness image to be processed to obtain the target image, and outputs the target image.

[0060] Optionally, the gaze mechanism module includes a chromaticity information convolution layer, a chromaticity information activation layer, a chromaticity information multiplication layer and a chromaticity information merging processing layer connected in series in sequence;

[0061] The chromaticity information convolution layer is used to perform convolution processing on the input data of the gaze mechanism module to obtain a first chromaticity image feature;

[0062] The chromaticity information activation layer is used to activate the first chromaticity image feature to obtain a second chromaticity image feature;

[0063] The chromaticity information multiplication layer is used to perform a co-occurrence multiplication process on the input data of the gaze mechanism module and the second chromaticity image feature to obtain the third chromaticity image feature;

[0064] The chromaticity information merging processing layer is used to merge the third chromaticity image feature with the object edge feature information in the brightness image to be processed to obtain the chromaticity image correction information to be processed.

[0065] Optionally, the gaze mechanism module is a plurality of gaze mechanism modules connected in series, the input data of the first gaze mechanism module among the plurality of gaze mechanism modules is the chromaticity image to be processed, the input data of the non-first gaze mechanism module among the plurality of gaze mechanism modules is the output data of the previous gaze mechanism module, and the chromaticity image correction information to be processed is the output data of the last gaze mechanism module among the plurality of gaze mechanism modules.

[0066] Optionally, the device further comprises:

[0067] A response module, configured to obtain an image input by a user in response to an input operation of the user before obtaining the image to be processed;

[0068] The first conversion module is configured to, when the image format of the image input by the user is in RGB format, perform format conversion on the image input by the user to obtain the image to be processed in YUV format.

[0069] Optionally, the image format of the target image is the YUV format, and the device further includes:

[0070] A second conversion module is configured to, after obtaining the target image output by the dechromatization processing model, perform format conversion on the target image to obtain the target image in the RGB format;

[0071] The feedback module is configured to feed back the target image in RGB format to the user's terminal.

[0072] According to a fourth aspect of an embodiment of the present disclosure, a training device for a chromatic aberration removal model is provided, the device comprising:

[0073] A construction module, used to obtain a color difference-free image and construct a corresponding color difference image according to the color difference-free image;

[0074] An execution module, configured to use the image without color difference and the image with color difference as a training data set;

[0075] The training module is used to train the de-chromatization processing model according to the training data set.

[0076] According to a fifth aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0077] processor;

[0078] a memory for storing processor-executable instructions;

[0079] Wherein, the processor is configured to:

[0080] Get the image to be processed;

[0081] Inputting the image to be processed into a de-chromatization processing model to obtain a target image output by the de-chromatization processing model;

[0082] The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, and each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image.

[0083] According to a sixth aspect of an embodiment of the present disclosure, a training device for a chromatic aberration removal model is provided, the device comprising:

[0084] An acquisition module is configured to acquire one or more images without color difference and construct corresponding images with color difference according to the images without color difference;

[0085] A generating module, configured to generate a training data set according to one or more of the images without color difference and the corresponding images with color difference;

[0086] The training module is configured to train the deep convolutional neural network model according to the training data set to obtain the de-colorization processing model.

[0087] Optionally, the acquisition module includes:

[0088] A separation submodule is configured to separate the brightness signal and the chrominance signal of the color difference-free image to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and to obtain a chrominance image corresponding to the chrominance signal of the color difference-free image;

[0089] an adding submodule, configured to determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image, so as to obtain a new chromaticity image;

[0090] The synthesis submodule is configured to synthesize the brightness image and the new chrominance image to obtain the color difference image corresponding to the color difference-free image.

[0091] Optionally, the training module is configured to:

[0092] According to the training data set and the preset loss function, the deep convolutional neural network model is trained to obtain the de-colorization processing model, and the preset loss function is:

[0093] g = loss(y,y_) = c × |y-y_|;

[0094] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0095] According to a seventh aspect of an embodiment of the present disclosure, a training device for a chromatic aberration removal model is provided, comprising:

[0096] processor;

[0097] a memory for storing processor-executable instructions;

[0098] Wherein, the processor is configured to:

[0099] Acquire one or more images without color difference, and construct corresponding images with color difference according to the images without color difference;

[0100] Generate a training data set based on one or more of the images without color difference and the corresponding images with color difference;

[0101] The deep convolutional neural network model is trained according to the training data set to obtain the chromatic aberration removal processing model.

[0102] According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of any one of the methods provided in the first aspect or the second aspect of the present disclosure are implemented.

[0103] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0104] By obtaining an image to be processed and inputting the image to be processed into a de-colorization processing model, a target image output by the de-colorization processing model is obtained. Compared with the method in the related art, this method of using the model processing can improve the speed and accuracy of de-colorization processing of the image.

[0105] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0107] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0108] Figure 2 The figure is a flowchart of a method for constructing a training data set according to an exemplary embodiment.

[0109] Figure 3 The figure is a diagram showing a de-colorization processing model architecture according to an exemplary embodiment.

[0110] Figure 4 The figure is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0111] Figure 5 It is a block diagram of a device according to an exemplary embodiment.

[0112] Figure 6 is a block diagram of another device according to an exemplary embodiment. DETAILED DESCRIPTION

[0113] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0114] In order to make it easier for ordinary technicians in the field to understand the technical solution of the present disclosure, the relevant professional terms involved in the present disclosure are first briefly explained below.

[0115] YUV is a type of image format used in applications such as videos, pictures, and cameras. It is actually the name of the color space shared by all "YUV" pixel formats. The YUV format is different from the RGB format (red, green, and blue). YUV is represented by a "brightness" component called Y (equivalent to grayscale) and two "chrominance" components, called U, R-Y (blue projection) and V, B-Y (red projection). This color representation method is the so-called YUV color space representation. The importance of using the YUV color space is that its brightness signal Y and chrominance signals U and V are separated. If there is only a Y signal component but no U and V signal components, then the image represented in this way is a black and white grayscale image.

[0116] The reason why chromatic aberration and color fringing appear in images is that most light sources in nature are a mixture of light of different wavelengths, and the refractive index of the camera optical lens for light of different wavelengths is different. Therefore, when designing an optical lens, the green light is often chosen to be correctly focused, which causes the light of other colors to be out of focus. Since multiple colors of light are not correctly focused, chromatic aberration and color fringing will occur when using an optical lens to form an image. Among the chromatic aberration and color fringing, chromatic aberration and color fringing are the most common. One possible situation is that when taking photos with a digital camera, due to the large contrast between the edges of the photographed object, at the junction of the high light and low light edges, a purple-red edge that does not exist may appear because the blue and red channels cannot be completely and accurately focused.

[0117] Before describing the embodiments of the technical solution of the present invention in detail, it should be stated that the user-related information used in the embodiments of the present invention, such as the image to be processed obtained from the user side, and the training data obtained from the user side for training the de-chromatization processing model, are all collected / used with the authorization or consent of the user.

[0118] The technical solution of the present disclosure is described in detail below with reference to embodiments.

[0119] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method can be used in a terminal, can also be used in a camera, such as in an algorithm for post-processing an image obtained after the camera forms an image, and can also be used in a server. The image processing method can include the following steps.

[0120] In step S11, an image to be processed is obtained.

[0121] Wherein, there are chromatic aberration and color fringing at the edges of objects in the image to be processed.

[0122] It is worth noting that in the present disclosure, other terms for the edge of an object are object contours and high-contrast area edges. High contrast refers to the contrast range of a black-and-white image, which is described by a technology (English tone) on a grayscale (scale) with equal intervals from black to white. The edge of an object can be understood as the junction of the highlight and low-light edges.

[0123] In a scenario where the image processing method of the present disclosure is applied to a user terminal or a server, in an achievable implementation manner, before acquiring the image to be processed, the image processing method may further include the following steps:

[0124] In response to a user's input operation, an image input by the user is obtained; when the image format of the image input by the user is in RGB format, the image input by the user is format converted to obtain the image to be processed in YUV format; when the image format of the image input by the user is in YUV format, the image input by the user is directly used as the image to be processed.

[0125] Those skilled in the art should understand that the image formats RGB and YUV can be converted to each other.

[0126] In this way, it is possible to meet the needs of users and remove color difference and color edge processing on images uploaded or selected by users (the images may be images obtained by users from any channel).

[0127] In step S12, the image to be processed is input into a chromatic aberration processing model to obtain a target image output by the chromatic aberration processing model.

[0128] The image formats of the to-be-processed image and the target image are in YUV format.

[0129] The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, and each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image.

[0130] The color-free image may be a clean image without color-difference color edges taken by a high-quality optical lens. Alternatively, the color-free image may be a clean image without color-difference color edges collected from the Internet. The present disclosure does not specifically limit the method for obtaining the color-free image.

[0131] Optionally, the training data set includes images with color difference and images without color difference corresponding to the images with color difference, and the images with color difference corresponding to the images without color difference are obtained by:

[0132] The brightness signal and the chrominance signal of the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and a chrominance image corresponding to the chrominance signal of the color difference-free image is obtained; an object edge image is determined according to the brightness image, and a color difference edge of a preset width is added to the object edge in the chrominance image based on the object edge position information represented by the object edge image to obtain a new chrominance image; the brightness image and the new chrominance image are synthesized to obtain the color difference image corresponding to the color difference-free image.

[0133] For example, the method of constructing a training data set according to the achromatic image is as follows: Figure 2 As shown, the following steps are included:

[0134] In step S21, the brightness signal and the chrominance signal in the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and to obtain a chrominance image corresponding to the chrominance signal of the color difference-free image.

[0135] Specifically, the brightness signal Y and the chrominance signal UV in the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal Y of the color difference-free image, and a chrominance image corresponding to the chrominance signal UV of the color difference-free image. The brightness image corresponding to the brightness signal Y of the color difference-free image is a black and white grayscale image.

[0136] In step S22, object edge detection is performed on the brightness image to obtain an object edge image corresponding to the brightness image.

[0137] In a possible implementation, an edge detection algorithm similar to the Canny edge detection algorithm may be used to perform edge detection on the brightness image of the achromatic image to obtain an object edge image corresponding to the achromatic image. The object edge image may be a binary image.

[0138] In step S23, the object edge position information in the chromaticity image is determined according to the positions of the pixel points representing the object edge in the object edge image.

[0139] Specifically, after the object edge image is obtained in step S22, the object edge position information in the chromaticity image may be determined according to the positions of the pixel points representing the object edge in the object edge image.

[0140] In step S24, according to the object edge position information, a color difference color edge of a preset width is added to the object edge in the chromaticity image to obtain a new chromaticity image.

[0141] For example, assuming that the preset width is 3 pixels, according to the object edge position information, a color difference edge with a width of 3 pixels is added to the object edge in the chromaticity image in the direction outside the object to obtain a new chromaticity image. The added color difference edge does not cover the pixel point where the object edge is located. For another example, assuming that the preset width is 2 pixels, the width of the object edge in the chromaticity image is 1 pixel (or 2 or 3 pixels). A color difference edge with a width of 2 pixels of the preset color is added to the target side of the 1 pixel in the chromaticity image to obtain a new chromaticity image. The target side is the outside of the object.

[0142] In step S25, the brightness image and the new chrominance image are synthesized to obtain a color difference image, wherein the image format of the color difference image and the image without color difference is the YUV format.

[0143] The brightness image (ie, Y component information) and the new chrominance image (ie, UV component information) are synthesized to obtain a color difference image in a YUV format.

[0144] In step S26, the image with color difference and the image without color difference are used as the training data set.

[0145] By adopting this method of constructing a training data set based on a color-free image, a color-difference image with color-difference color fringes can be reversely constructed based on a color-free image without color-difference color fringes. Compared with the method of constructing a color-difference image without color-difference color fringes based on a color-difference image with color-difference color fringes, the method of constructing a training data set disclosed in the present invention is simpler and easier. The reason is that by adopting any color-difference color-edge removal method in the related art to remove color-difference color fringes from a color-difference image with color-difference color fringes, a completely clean color-difference image without color-difference color fringes cannot be quickly obtained.

[0146] After constructing the training data set, when training the chromatic aberration removal model according to the training data set, the chromatic aberration images and the clean chromatic aberration-free images without chromatic aberration color edges reversely constructed by the above method of the present disclosure can be used to train a chromatic aberration removal model with better robustness and better effect.

[0147] It is worth noting here that when constructing a color-difference image in a training data set, the present disclosure does not specifically limit the color of the color-difference color edge added to the image, and the color of the color edge can be purple, red, blue, yellow, green, etc.

[0148] Since images in YUV format are mainly used to optimize the transmission of color video signals, making them backward compatible with old black-and-white televisions. Compared with RGB video signal transmission, its biggest advantage is that it only takes up very little bandwidth (RGB requires three independent video signals to be transmitted simultaneously). Therefore, without considering the bandwidth, the image generally required by users is an image in RGB format. Therefore, in an achievable implementation, after obtaining the target image output by the de-colorization processing model, if the image format of the target image is the YUV format, then the image processing method may also include the following steps:

[0149] The target image is converted into a format to obtain a target image in the RGB format; and the target image in the RGB format is fed back to the user's terminal, so as to meet the user's demand for using the target image in the RGB format.

[0150] By adopting the above method disclosed in the present invention, an image to be processed is obtained, in which there are chromatic aberration and color fringing at the edges of objects in the image to be processed, and the image to be processed is input into a de-chromatic aberration processing model, so as to obtain a target image output by the de-chromatic aberration processing model. Since the image format of the image to be processed is a YUV format, when the de-chromatic aberration processing model performs de-chromatic aberration and color fringing processing on the image to be processed, the chromatic aberration correction can be performed only on the UV chromaticity signal at the position of the chromatic aberration and color fringing of the image to be processed, without changing the Y brightness signal of the entire image to be processed. Therefore, the method disclosed in the present invention that uses the de-chromatic aberration processing model to perform de-chromatic aberration and color fringing processing on the image to be processed can avoid the loss of brightness information in the image to be processed, and obtain a target image with better effect. Therefore, this method disclosed in the present invention improves the effect of removing chromatic aberration and color fringing in the image.

[0151] Optionally, the chromatic aberration removal model is a deep convolutional neural network model.

[0152] In the present application, the architecture of the chromatic aberration removal processing model is preferably a deep convolutional neural network model architecture in a deep learning model. Of course, it can also be other deep learning model architectures, and the present disclosure does not make any specific limitations on this.

[0153] It is worth explaining that deep learning refers to machine learning based on artificial neural networks. Different from traditional machine learning, deep learning models need to be trained with a large number of manually annotated samples in order to train a deep learning model with higher accuracy. A deep learning model usually includes a multi-layer artificial neural network. By increasing the depth and breadth of the artificial neural network, the fitting ability of the artificial neural network can be improved, and a deep learning model with better robustness can be obtained. The essence of a deep learning model is to learn a mapping relationship f(x)=y. For example, if x is an input handwritten digital picture, then y is one of the output numbers 0 to 9. In the application scenario of the present disclosure, the input x of the decolorization processing model refers to the image to be processed with color difference and color edges, and the output y refers to the target image after removing the color difference and color edges.

[0154] It should be noted that the process of training a deep learning model is usually as follows: first prepare a training data set, which is usually in the form of paired (x, y) data sets. Where x is the input data and y is the corresponding output data. After preparing the training data set, x is input into the deep learning model for forward propagation to obtain y_ output by the deep learning model. Further, (y, y_) is input into the error function loss function, and after the convergence degree of the error judgment model (the smaller the loss value, the better) is calculated, the gradient vector is determined by back propagation (reverse derivation). Furthermore, each model parameter of the deep learning model is adjusted by the gradient vector. Then, a new y_ is calculated based on the x input into the deep learning model, until the set number of iterations is reached or the average value of the loss error no longer decreases, and a trained deep learning model is obtained. The training principle of the de-colorization processing model in the present disclosure is as shown above.

[0155] For example, the step of obtaining the de-colorization processing model by training the training data set includes: training a deep convolutional neural network model according to the chromatic aberration image, the chromatic aberration-free image, and a preset loss function in the training data set to obtain the de-colorization processing model, wherein the preset loss function is:

[0156] g = loss(y,y_) = c × |y-y_|;

[0157] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0158] Since chromatic aberration color edges always appear at the edge of high-contrast areas, in the technical solution disclosed in the present invention, in order to improve the training effect of the de-chromatic aberration processing model, the preset loss function loss function is designed as an adaptive weight loss function. The adaptive weight information corresponds to the parameter c in the above-mentioned preset loss function, and c represents the probability value set of each pixel in the object edge image as an object edge pixel. It is easy to understand that the larger the probability value of the pixel point, the greater the possibility of representing the pixel point as an object edge pixel point. Applying this adaptive weight information c to the preset loss function can make the de-chromatic aberration processing model pay more attention to the edge of the high-contrast area with a large weight, that is, pay more attention to the chromatic aberration color edge, and pay less attention to the non-high-contrast area with a small weight, that is, pay less attention to the non-chromatic aberration color edge. When the de-chromatic aberration processing model pays more attention to the edge of the high-contrast area with a large weight, that is, pays more attention to the chromatic aberration color edge, and pays less attention to the non-high-contrast area with a small weight, that is, pays less attention to the non-chromatic aberration color edge, the data calculation amount of the de-chromatic aberration processing model can be effectively reduced, and the accuracy of color difference recognition can be improved.

[0159] It is easy to understand that in an image, the proportion of pixels with chromatic aberration and color fringing to all pixels is usually very small, generally around 5%. By applying the adaptive weight information c to the preset loss function, the purpose of correcting only the pixel area with chromatic aberration and color fringing is achieved, avoiding a large number of useless calculations (i.e. avoiding calculating whether the pixels in the non-chromatic aberration and color fringing area have chromatic aberration), and improving the processing speed of the chromatic aberration processing model for chromatic aberration and color fringing.

[0160] Therefore, the chromatic aberration removal processing model trained by the preset loss function disclosed in the present invention can quickly remove chromatic aberration and color edge processing on the image to be processed to obtain the target image due to the low data calculation amount of the chromatic aberration removal processing model. This enables the image processing method disclosed in the present invention to meet the real-time requirements of users.

[0161] Optionally, the architecture of the chromatic aberration removal model in the present disclosure is as follows: Figure 3 As shown, the de-chromatization processing model includes a pre-processing module, a gaze mechanism module, and a synthesis module; the step of inputting the image to be processed into the de-chromatization processing model to obtain a target image output by the de-chromatization processing model includes:

[0162] Input the image to be processed into the preprocessing module, the preprocessing module separates the brightness signal and the chrominance signal in the image to be processed, obtains a brightness image to be processed corresponding to the brightness signal of the image to be processed, and obtains a chrominance image to be processed corresponding to the chrominance signal of the image to be processed, and inputs the brightness image to be processed and the chrominance image to be processed into the gaze mechanism module;

[0163] The gaze mechanism module calculates the color difference correction value according to the object edge feature information in the brightness image to be processed and the chromaticity image to be processed, obtains the correction information of the chromaticity image to be processed, and inputs the chromaticity image to be processed, the brightness image to be processed, and the correction information of the chromaticity image to be processed into the synthesis module;

[0164] The synthesis module corrects the chromaticity image to be processed according to the correction information of the chromaticity image to be processed to obtain a corrected chromaticity image to be processed, synthesizes the corrected chromaticity image to be processed and the luminance image to be processed into the target image, and outputs the target image.

[0165] Correspondingly, the preprocessing module is used to separate the brightness signal and the chrominance signal in the image to be processed, to obtain a brightness image to be processed corresponding to the brightness signal of the image to be processed, and to obtain a chrominance image to be processed corresponding to the chrominance signal of the image to be processed;

[0166] The gaze mechanism module is used to calculate the color difference correction value according to the object edge feature information in the brightness image to be processed and the chromaticity image to be processed, so as to obtain the correction information of the chromaticity image to be processed;

[0167] The synthesis module is used to correct the chromaticity image to be processed according to the correction information of the chromaticity image to be processed to obtain the corrected chromaticity image to be processed, and synthesize the corrected chromaticity image to be processed and the luminance image to be processed into the target image.

[0168] Among them, the gaze mechanism module includes a chromaticity information convolution layer conv, a chromaticity information activation layer sigmoid, a chromaticity information multiplication layer and a chromaticity information merging processing layer elt-prod connected in series in sequence; the chromaticity information convolution layer is used to perform convolution processing on the input data of the gaze mechanism module to obtain a first chromaticity image feature; the chromaticity information activation layer is used to perform an activation operation on the first chromaticity image feature to obtain a second chromaticity image feature; the chromaticity information multiplication layer is used to perform co-localized multiplication processing on the input data of the gaze mechanism module and the second chromaticity image feature based on a cross-layer skip connection to obtain the third chromaticity image feature; the chromaticity information merging processing layer concat is used to merge the third chromaticity image feature with the object edge feature information in the luminance image to be processed to obtain the chromaticity image correction information to be processed.

[0169] Optionally, the gaze mechanism module is a plurality of gaze mechanism modules connected in series, the input data of the first gaze mechanism module among the plurality of gaze mechanism modules is the chromaticity image to be processed, the input data of the non-first gaze mechanism module among the plurality of gaze mechanism modules is the output data of the previous gaze mechanism module, and the chromaticity image correction information to be processed is the output data of the last gaze mechanism module among the plurality of gaze mechanism modules.

[0170] In which, when the de-colorization processing model includes multiple gaze mechanism modules connected in series and / or in parallel, the input data of the first gaze mechanism module among the multiple gaze mechanism modules is the chromaticity image to be processed, the input data of the non-first gaze mechanism module among the multiple gaze mechanism modules is the output data of the previous gaze mechanism module, the output data of the previous gaze mechanism module can be called the intermediate layer chromaticity image correction information, and the output data of the last gaze mechanism module is the chromaticity image correction information to be processed.

[0171] It is worth noting that since the output data of the gaze mechanism module is the correction information of the chromaticity image to be processed, the size of the correction value of each pixel in the correction information of the chromaticity image to be processed can be used as the color difference value of each pixel (the color difference value is used to measure the size of the color difference). If the absolute value of the correction value corresponding to a certain pixel is less than a certain value, it can be considered that there is no color difference in the pixel, and the synthesis module can omit the operation of correcting the pixel. If the absolute value of the correction value of a certain pixel is greater than a certain value, it can be considered that there is color difference in the pixel, and the synthesis module performs the operation of correcting the pixel.

[0172] By adopting the chromatic aberration removal model architecture disclosed in the present invention, the chromatic aberration removal model is trained according to a preset loss function, so that the chromatic aberration removal model learns a method similar to the attention mechanism to achieve a high degree of attention to chromatic aberration and color edges, thereby enhancing the chromatic aberration removal model's performance in removing chromatic aberration and color edges of images.

[0173] The present disclosure also provides a method for training a chromatic aberration removal model, the method comprising:

[0174] Acquire one or more color difference-free images, and construct corresponding color difference images according to the color difference-free images; generate a training data set according to one or more color difference-free images and the corresponding color difference images; train a deep convolutional neural network model according to the training data set to obtain the de-colorization processing model.

[0175] In a possible implementation manner, constructing a corresponding image with color difference according to the image without color difference specifically includes the following implementation steps:

[0176] The brightness signal and the chrominance signal of the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and a chrominance image corresponding to the chrominance signal of the color difference-free image is obtained; an object edge image is determined according to the brightness image, and a color difference edge of a preset width is added to the object edge in the chrominance image based on the object edge position information represented by the object edge image to obtain a new chrominance image; the brightness image and the new chrominance image are synthesized to obtain the color difference image corresponding to the color difference-free image.

[0177] For any image without color difference, the corresponding image with color difference can be obtained through the above image construction method.

[0178] The color of the color difference color edge is at least one of red, green, blue, yellow, purple and the like.

[0179] In a possible implementation manner, the training of a deep convolutional neural network model according to the training data set to obtain the chromatic aberration removal processing model may specifically include the following steps:

[0180] According to the training data set and the preset loss function, the deep convolutional neural network model is trained to obtain the de-colorization processing model, and the preset loss function is:

[0181] g = loss(y,y_) = c × |y-y_|;

[0182] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0183] Regarding the embodiment of the training method of the above-mentioned de-chromatization processing model, the specific execution method of each step has been described in detail in the aforementioned image processing method embodiment related to the method, and will not be elaborated here.

[0184] Figure 4 is a block diagram of an image processing device according to an exemplary embodiment. Figure 4 The device 100 includes an acquisition module 121 and an input module 122.

[0185] The acquisition module 121 is configured to acquire an image to be processed.

[0186] The input module 122 is configured to input the image to be processed into the de-chromatization processing model to obtain a target image output by the de-chromatization processing model.

[0187] The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, and each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image.

[0188] By adopting this device, by obtaining an image to be processed and inputting the image to be processed into a de-colorization processing model, a target image output by the de-colorization processing model is obtained. Compared with the method in the related art, this method of using the model processing can improve the speed and accuracy of de-colorization processing of images.

[0189] Optionally, the image with color difference corresponding to the image without color difference is obtained by:

[0190] Separating the brightness signal and the chrominance signal of the image without color difference to obtain a brightness image corresponding to the brightness signal of the image without color difference, and obtaining a chrominance image corresponding to the chrominance signal of the image without color difference;

[0191] Determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image to obtain a new chromaticity image;

[0192] The brightness image and the new chrominance image are synthesized to obtain the image with color difference corresponding to the image without color difference.

[0193] Optionally, the device further includes a training module configured to train a deep convolutional neural network model according to the training data set and a preset loss function to obtain the de-colorization processing model, and the preset loss function is:

[0194] g = loss(y,y_) = c × |y-y_|;

[0195] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0196] Optionally, the de-colorization processing model includes a pre-processing module, a gaze mechanism module, and a synthesis module; and the input module is used for:

[0197] The image to be processed is input into the preprocessing module, and the preprocessing module separates the brightness signal and the chrominance signal in the image to be processed to obtain a brightness image to be processed corresponding to the brightness signal of the image to be processed, and obtains a chrominance image to be processed corresponding to the chrominance signal of the image to be processed, and inputs the brightness image to be processed and the chrominance image to be processed into the gaze mechanism module; the gaze mechanism module calculates the color difference correction value according to the object edge feature information in the brightness image to be processed and the chrominance image to be processed to obtain the correction information of the chrominance image to be processed, and inputs the chrominance image to be processed, the brightness image to be processed and the correction information of the chrominance image to be processed into the synthesis module; the synthesis module corrects the chrominance image to be processed according to the correction information of the chrominance image to be processed to obtain a corrected chrominance image to be processed, and synthesizes the corrected chrominance image to be processed with the brightness image to be processed to obtain the target image, and outputs the target image.

[0198] Optionally, the gaze mechanism module includes a chromaticity information convolution layer, a chromaticity information activation layer, a chromaticity information multiplication layer and a chromaticity information merging processing layer connected in series in sequence;

[0199] The chromaticity information convolution layer is used to perform convolution processing on the input data of the gaze mechanism module to obtain a first chromaticity image feature;

[0200] The chromaticity information activation layer is used to activate the first chromaticity image feature to obtain a second chromaticity image feature;

[0201] The chromaticity information multiplication layer is used to perform a co-occurrence multiplication process on the input data of the gaze mechanism module and the second chromaticity image feature to obtain the third chromaticity image feature;

[0202] The chromaticity information merging processing layer is used to merge the third chromaticity image feature with the object edge feature information in the brightness image to be processed to obtain the chromaticity image correction information to be processed.

[0203] Optionally, the gaze mechanism module is a plurality of gaze mechanism modules connected in series, the input data of the first gaze mechanism module among the plurality of gaze mechanism modules is the chromaticity image to be processed, the input data of the non-first gaze mechanism module among the plurality of gaze mechanism modules is the output data of the previous gaze mechanism module, and the chromaticity image correction information to be processed is the output data of the last gaze mechanism module among the plurality of gaze mechanism modules.

[0204] Optionally, the device further comprises:

[0205] A response module, configured to obtain an image input by a user in response to an input operation of the user before obtaining the image to be processed;

[0206] The first conversion module is configured to, when the image format of the image input by the user is in RGB format, perform format conversion on the image input by the user to obtain the image to be processed in YUV format.

[0207] Optionally, the image format of the target image is the YUV format, and the device further includes:

[0208] A second conversion module is configured to, after obtaining the target image output by the de-chromatization processing model, perform format conversion on the target image to obtain the target image in the RGB format;

[0209] The feedback module is configured to feed back the target image in RGB format to the user's terminal.

[0210] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0211] Based on the same inventive concept, the present disclosure provides a training device for a chromatic aberration removal model, the device comprising:

[0212] An acquisition module is configured to acquire one or more images without color difference and construct corresponding images with color difference according to the images without color difference;

[0213] A generating module, configured to generate a training data set according to one or more of the images without color difference and the corresponding images with color difference;

[0214] The training module is configured to train the deep convolutional neural network model according to the training data set to obtain the de-colorization processing model.

[0215] Optionally, the acquisition module includes:

[0216] A separation submodule is configured to separate the brightness signal and the chrominance signal of the color difference-free image to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and to obtain a chrominance image corresponding to the chrominance signal of the color difference-free image;

[0217] an adding submodule, configured to determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image, so as to obtain a new chromaticity image;

[0218] The synthesis submodule is configured to synthesize the brightness image and the new chrominance image to obtain the color difference image corresponding to the color difference-free image.

[0219] Optionally, the training module is configured to:

[0220] According to the training data set and the preset loss function, the deep convolutional neural network model is trained to obtain the de-colorization processing model, and the preset loss function is:

[0221] g = loss(y,y_) = c × |y-y_|;

[0222] Among them, g represents the loss information between the first target image y_ generated by the dechromatization processing model according to the chromatic aberration image and the color-free image y corresponding to the chromatic aberration image during the training process of the dechromatization processing model, and c represents the probability value set of each pixel point in the object edge image being an object edge pixel point.

[0223] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the image processing method or the de-colorization processing model training method provided by the present disclosure.

[0224] Figure 5 8 is a block diagram of an apparatus 800 for image processing according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0225] Reference Figure 5 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0226] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned image processing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0227] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0228] The power component 806 provides power to the various components of the device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0229] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0230] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0231] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0232] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0233] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0234] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned image processing method.

[0235] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to complete the above-mentioned image processing method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0236] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned image processing method when executed by the programmable device.

[0237] Figure 6 1 is a block diagram of a device 1900 for image processing according to an exemplary embodiment. For example, the device 1900 may be provided as a server. Figure 6 , the device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the steps of the above-mentioned image processing method.

[0238] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output (I / O) interface 1958. The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0239] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0240] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into a de-chromatization processing model to obtain a target image output by the de-chromatization processing model; The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image; The image with color difference corresponding to the image without color difference is obtained by: Separating the brightness signal and the chrominance signal of the image without color difference to obtain a brightness image corresponding to the brightness signal of the image without color difference, and obtaining a chrominance image corresponding to the chrominance signal of the image without color difference; Determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image to obtain a new chromaticity image; The brightness image and the new chrominance image are synthesized to obtain the image with color difference corresponding to the image without color difference.

2. The method according to claim 1, characterized in that The de-chromatization processing model is trained in the following way: According to the training data set and the preset loss function, the deep convolutional neural network model is trained to obtain the de-colorization processing model, and the preset loss function is: ； Among them, the Characterizes that during the training process of the de-chromatization processing model, the first target image generated by the de-chromatization processing model according to the chromatization image The image without color difference corresponding to the image with color difference The loss information between A probability value set characterizing that each pixel in the object edge image is an object edge pixel.

3. The method according to claim 1 or 2, characterized in that: The chromatic aberration removal processing model includes a preprocessing module, a gaze mechanism module, and a synthesis module; the step of inputting the image to be processed into the chromatic aberration removal processing model to obtain a target image output by the chromatic aberration removal processing model includes: Input the image to be processed into the preprocessing module, the preprocessing module separates the brightness signal and the chrominance signal in the image to be processed, obtains a brightness image to be processed corresponding to the brightness signal of the image to be processed, and obtains a chrominance image to be processed corresponding to the chrominance signal of the image to be processed, and inputs the brightness image to be processed and the chrominance image to be processed into the gaze mechanism module; The gaze mechanism module calculates the color difference correction value according to the object edge feature information in the brightness image to be processed and the chromaticity image to be processed, obtains the correction information of the chromaticity image to be processed, and inputs the chromaticity image to be processed, the brightness image to be processed, and the correction information of the chromaticity image to be processed into the synthesis module; The synthesis module corrects the chromaticity image to be processed according to the correction information of the chromaticity image to be processed to obtain a corrected chromaticity image to be processed, synthesizes the corrected chromaticity image to be processed and the luminance image to be processed into the target image, and outputs the target image.

4. The method according to claim 3, characterized in that The gaze mechanism module includes a chromaticity information convolution layer, a chromaticity information activation layer, a chromaticity information multiplication layer and a chromaticity information merging processing layer connected in series in sequence; The chromaticity information convolution layer is used to perform convolution processing on the input data input to the gaze mechanism module to obtain a first chromaticity image feature; The chromaticity information activation layer is used to activate the first chromaticity image feature to obtain a second chromaticity image feature; The chromaticity information multiplication layer is used to perform a co-localized multiplication process on the input data of the gaze mechanism module and the second chromaticity image feature to obtain a third chromaticity image feature; The chromaticity information merging processing layer is used to merge the third chromaticity image feature with the object edge feature information in the brightness image to be processed to obtain the chromaticity image correction information to be processed.

5. The method according to claim 4, characterized in that The gaze mechanism module is a plurality of gaze mechanism modules connected in series, the input data of the first gaze mechanism module among the plurality of gaze mechanism modules is the chromaticity image to be processed, the input data of the non-first gaze mechanism module among the plurality of gaze mechanism modules is the output data of the previous gaze mechanism module, and the chromaticity image correction information to be processed is the output data of the last gaze mechanism module among the plurality of gaze mechanism modules.

6. The method according to claim 1, characterized in that Before obtaining the image to be processed, the method includes: In response to an input operation of a user, acquiring an image input by the user; In the case that the image format of the image input by the user is in RGB format, the image input by the user is format converted to obtain the image to be processed in YUV format.

7. The method according to claim 6, characterized in that The image format of the target image is the YUV format. After obtaining the target image output by the de-colorization processing model, the method includes: Convert the target image to RGB format. The target image in RGB format is fed back to the user's terminal.

8. A method for training a chromatic aberration removal model, characterized in that: The method comprises: Acquire one or more images without color difference, and construct corresponding images with color difference according to the images without color difference; Generate a training data set based on one or more of the images without color difference and the corresponding images with color difference; Training a deep convolutional neural network model according to the training data set to obtain the chromatic aberration removal processing model; The step of constructing a corresponding image with color difference according to the image without color difference comprises: The brightness signal and the chrominance signal of the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and a chrominance image corresponding to the chrominance signal of the color difference-free image is obtained; an object edge image is determined according to the brightness image, and a color difference edge of a preset width is added to the object edge in the chrominance image based on the object edge position information represented by the object edge image to obtain a new chrominance image; the brightness image and the new chrominance image are synthesized to obtain the color difference image corresponding to the color difference-free image.

9. An image processing device, characterized in that: The device comprises: An acquisition module, configured to acquire an image to be processed; An input module is configured to input the image to be processed into a chromatic aberration processing model to obtain a target image output by the chromatic aberration processing model; The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image; The image with color difference corresponding to the image without color difference is obtained by: The brightness signal and the chrominance signal of the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and a chrominance image corresponding to the chrominance signal of the color difference-free image is obtained; an object edge image is determined according to the brightness image, and a color difference edge of a preset width is added to the object edge in the chrominance image based on the object edge position information represented by the object edge image to obtain a new chrominance image; the brightness image and the new chrominance image are synthesized to obtain the color difference image corresponding to the color difference-free image.

10. An image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Get the image to be processed; Inputting the image to be processed into a de-chromatization processing model to obtain a target image output by the de-chromatization processing model; The de-colorization processing model is obtained by training a training data set, wherein the training data set includes multiple groups of sample data, each group of sample data includes a color-difference-free image and a color-difference image corresponding to the color-difference-free image; The image with color difference corresponding to the image without color difference is obtained by: The brightness signal and the chrominance signal of the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and a chrominance image corresponding to the chrominance signal of the color difference-free image is obtained; an object edge image is determined according to the brightness image, and a color difference edge of a preset width is added to the object edge in the chrominance image based on the object edge position information represented by the object edge image to obtain a new chrominance image; the brightness image and the new chrominance image are synthesized to obtain the color difference image corresponding to the color difference-free image.

11. A training device for a chromatic aberration removal model, characterized in that: The device comprises: An acquisition module is configured to acquire one or more images without color difference and construct corresponding images with color difference according to the images without color difference; A generating module, configured to generate a training data set according to one or more of the images without color difference and the corresponding images with color difference; A training module is configured to train a deep convolutional neural network model according to the training data set to obtain the chromatic aberration removal processing model; Wherein, the acquisition module includes: A separation submodule is configured to separate the brightness signal and the chrominance signal of the color difference-free image to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and to obtain a chrominance image corresponding to the chrominance signal of the color difference-free image; an adding submodule, configured to determine an object edge image according to the brightness image, and add a color difference color edge of a preset width to the object edge in the chromaticity image based on the object edge position information represented by the object edge image, so as to obtain a new chromaticity image; The synthesis submodule is configured to synthesize the brightness image and the new chrominance image to obtain the color difference image corresponding to the color difference-free image.

12. A training device for a chromatic aberration removal model, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Acquire one or more images without color difference, and construct corresponding images with color difference according to the images without color difference; Generate a training data set based on one or more of the images without color difference and the corresponding images with color difference; Training a deep convolutional neural network model according to the training data set to obtain the chromatic aberration removal processing model; The step of constructing a corresponding image with color difference according to the image without color difference comprises: The brightness signal and the chrominance signal of the color difference-free image are separated to obtain a brightness image corresponding to the brightness signal of the color difference-free image, and a chrominance image corresponding to the chrominance signal of the color difference-free image is obtained; an object edge image is determined according to the brightness image, and a color difference edge of a preset width is added to the object edge in the chrominance image based on the object edge position information represented by the object edge image to obtain a new chrominance image; the brightness image and the new chrominance image are synthesized to obtain the color difference image corresponding to the color difference-free image.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images