Image processing method, system, device and storage medium

Abstract

The application provides an image processing method, system, device and storage medium, the method comprises the following steps: obtaining an original Raw image and an RGB image processed by imaging from an imaging device; inputting the original Raw image and the RGB image into a first processing model to obtain a color drawing image output by the first processing model; inputting the color drawing image and a numerical image indicating halo suppression information into a second processing model to obtain an image after halo suppression output by the second processing model. By adopting the end-to-end image processing method based on the processing model, color drawing and halo suppression processing can be automatically and timely performed on the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an image processing method, system, device and storage medium. BACKGROUND

[0002] In daily life such as a traffic light scene (which can not be limited to this scene, and other related scenes are also applicable), due to the effects of environmental light (intensity of sunlight, vehicle lights, street lights, etc.) and mobile device imaging algorithms, the color of the lamp beads of the traffic light may be biased, such as yellowish red, whitish red, whitish yellow, etc., which may easily cause misjudgment and missed judgment in traffic violation judgment. In addition, due to the effects of fog or night reflection, colored halos may appear around the lamp beads of the traffic light, the imaging quality of the image may be greatly reduced, and the subsequent intelligent algorithms such as image enhancement and image denoising may be affected.

[0003] To solve the problem of traffic signal lamp coloration, a traditional solution is to obtain an RGB image after imaging by a mobile device, find the position of a lamp card on the RGB image by a lamp card detection algorithm, then separate the colors of R, G and B channels in the lamp bead area of the lamp card, and determine the color of the lamp bead by the ratio of R channel / G channel and B channel / G channel. Alternatively, the position of the lamp bead is found by a threshold segmentation method, and then the color of the lamp bead is manually determined according to the type of the traffic signal lamp and the conversion rules of red light, street light and yellow light. These methods usually have strong prior information and are strongly related to the type of the lamp card. In addition, if the RGB image is overexposed, the accuracy of the method will be greatly reduced, and the robustness is low.

[0004] To solve the problem of traffic signal lamp coloration, a traditional solution is to obtain an RGB image after imaging by a mobile device, find the position of a lamp card on the RGB image by a lamp card detection algorithm, then separate the colors of R, G and B channels in the lamp bead area of the lamp card, and determine the color of the lamp bead by the ratio of R channel / G channel and B channel / G channel. Alternatively, the position of the lamp bead is found by a threshold segmentation method, and then the color of the lamp bead is manually determined according to the type of the traffic signal lamp and the conversion rules of red light, street light and yellow light. These methods usually have strong prior information and are strongly related to the type of the lamp card. In addition, if the RGB image is overexposed, the accuracy of the method will be greatly reduced, and the robustness is low. SUMMARY

[0005] To solve the problem of traffic signal lamp coloration, a traditional solution is to obtain an RGB image after imaging by a mobile device, find the position of a lamp card on the RGB image by a lamp card detection algorithm, then separate the colors of R, G and B channels in the lamp bead area of the lamp card, and determine the color of the lamp bead by the ratio of R channel / G channel and B channel / G channel. Alternatively, the position of the lamp bead is found by a threshold segmentation method, and then the color of the lamp bead is manually determined according to the type of the traffic signal lamp and the conversion rules of red light, street light and yellow light. These methods usually have strong prior information and are strongly related to the type of the lamp card. In addition, if the RGB image is overexposed, the accuracy of the method will be greatly reduced, and the robustness is low.

[0006] The embodiment of the present application provides a kind of image processing method, comprising the following steps:

[0007] Raw image and RGB image after imaging processing are obtained from imaging device;

[0008] The original Raw image and the RGB image are input into the first processing model, and the color drawing image output by the first processing model is obtained;

[0009] The color drawing image and the numerical value graph indicating the light halo suppression information are input into the second processing model, and the image after light halo suppression output by the second processing model is obtained.

[0010] In some embodiments, before the color drawing image and the numerical value graph indicating the light halo suppression information are input into the second processing model, the following steps are further included:

[0011] The specified light halo suppression information is obtained, and the light halo suppression information includes light halo suppression range and / or light halo suppression level;

[0012] The numerical value graph indicating the light halo suppression information with the same resolution as the color drawing image is generated based on the light halo suppression information, and the numerical value graph includes the numerical value graph indicating the light halo suppression range and / or the numerical value graph indicating the light halo suppression level.

[0013] In some embodiments, the numerical value graph indicating the light halo suppression information with the same resolution as the color drawing image is generated based on the light halo suppression information, including the following steps:

[0014] The numerical value of the light halo suppression range is calculated based on the specified light halo suppression range and the preset minimum value and maximum value of the light halo suppression range, and the numerical value graph indicating the light halo suppression range with the same resolution as the color drawing image is generated based on the numerical value of the light halo suppression range; And / or,

[0015] The numerical value of the light halo suppression level is calculated based on the specified light halo suppression level and the preset minimum value and maximum value of the light halo suppression level, and the numerical value graph indicating the light halo suppression level with the same resolution as the color drawing image is generated based on the numerical value of the light halo suppression level.

[0016] In some embodiments, the specified light halo suppression information further includes the specified light halo suppression position, and the numerical value graph further includes the numerical value graph indicating the light halo suppression position.

[0017] In some embodiments, the numerical value graph indicating the light halo suppression information with the same resolution as the color drawing image is generated based on the light halo suppression information, including the following steps:

[0018] Based on the specified suppression halo position, a numerical image indicating the suppression halo position is generated with the same resolution as the stroke image, wherein the pixel value corresponding to the suppression halo position is a first value, the pixel value corresponding to the non-suppression halo position is a second value, and the first value is different from the second value.

[0019] In some embodiments, the method further comprises the following steps:

[0020] A plurality of sample original Raw images and corresponding sample RGB images are collected to form a plurality of sample image pairs.

[0021] Based on the sample RGB image in each sample image pair, a stroke processing is performed to obtain a target stroke image.

[0022] The sample image pair is taken as the input of the first processing model, and the target stroke image is taken as the fitting target of the first processing model, and the first processing model is trained.

[0023] In some embodiments, the method further comprises the following steps:

[0024] A plurality of sample stroke images are obtained.

[0025] A plurality of Gaussian images indicating different suppression halo information and having the same resolution as the sample stroke image are generated.

[0026] Each sample stroke image is multiplied pixel by pixel with the corresponding Gaussian image to obtain a corresponding target suppression halo image.

[0027] Each sample stroke image is combined with each numerical image indicating different suppression halo information to form the input of the second processing model, and the target suppression halo image is taken as the fitting target of the second processing model, and the second processing model is trained.

[0028] In some embodiments, the second processing model comprises a first convolutional layer, a second convolutional layer, a residual block, a third convolutional layer and a fourth convolutional layer connected in series, and after the stroke image and the numerical image indicating the suppression halo information are input into the second processing model, the numerical image indicating the suppression halo information is added to the input of the second convolutional layer and the input of the residual block.

[0029] In the image processing method of the present application, based on the original Raw image and the RGB image, the corresponding stroke image is obtained through the first processing model, which can improve the accuracy of the stroke, and the stroke image and the numerical image indicating the suppression halo information are input into the second processing model, which can improve the accuracy of the suppression halo. Figure OneThe corresponding image after the light halo is suppressed is obtained through the second processing model, automatic processing of color drawing and light halo suppression is realized, the overall imaging effect of the image is improved, the process is simplified, and the real-time requirement can be met.

[0030] The embodiment of the present application also provides an image processing system applied to the image processing method, and the system comprises:

[0031] An image acquisition module is configured to acquire an original Raw image and an RGB image processed by imaging from an imaging device.

[0032] A first processing module is configured to input the original Raw image and the RGB image into a first processing model to obtain a color drawing image output by the first processing model.

[0033] A second processing module is configured to input the color drawing image and a numerical image indicating light halo suppression information into a second processing model to obtain an image after the light halo is suppressed output by the second processing model.

[0034] The embodiment of the present application can improve the accuracy of color drawing by acquiring the original Raw image and the RGB image through the image acquisition module and the first processing module, obtaining the corresponding color drawing image through the first processing model, and inputting the color drawing image and the numerical image indicating the light halo suppression information into the second processing module. Figure One The corresponding image after the light halo is suppressed is obtained through the second processing model, automatic processing of color drawing and light halo suppression is realized, the overall imaging effect of the image is improved, the process is simplified, and the real-time requirement can be met.

[0035] The embodiment of the present application also provides an image processing device, which comprises:

[0036] A processor;

[0037] A memory, wherein executable instructions of the processor are stored in the memory.

[0038] The processor is configured to execute the steps of the image processing method by executing the executable instructions.

[0039] The processor executes the image processing method by executing the executable instructions when the image processing device is used, and thus the beneficial effects of the image processing method can be obtained.

[0040] The embodiment of the present application also provides a computer readable storage medium for storing a program, and the program is executed by the processor to realize the steps of the image processing method.

[0041] By using the computer readable storage medium provided by the present application, the program stored therein is executed to realize the steps of the image processing method, so that the beneficial effects of the image processing method can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0042] Other features, objects, and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments thereof as read with the accompanying drawings.

[0043] Figure 1 is a flow chart of an image processing method according to an embodiment of the present application;

[0044] Figure 2 is a flow chart of raw image preprocessing according to an embodiment of the present application;

[0045] Figure 3 is a flow chart of RGB image preprocessing according to an embodiment of the present application;

[0046] Figure 4 is a network structure diagram of an auto-encoder according to an embodiment of the present application;

[0047] Figure 5 is a network structure diagram of a residual block according to an embodiment of the present application;

[0048] Figure 6 is a network structure diagram of a second processing model according to an embodiment of the present application;

[0049] Figure 7 is a schematic diagram of an image processing system according to an embodiment of the present application;

[0050] Figure 8 is a structural schematic diagram of an image processing device according to an embodiment of the present application;

[0051] Figure 9 is a structural schematic diagram of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those skilled in the art. Like reference numerals refer to like elements throughout the figures, and descriptions of the same elements will not be repeated.

[0053] As shown in FIG. 1, in an embodiment, the present application provides an image processing method, comprising the following steps: Figure 1

[0054] ​S100: Obtain an original Raw image and an RGB image after imaging processing from an imaging device;

[0055] In this embodiment, the imaging device may, for example, be a camera with an imaging processing algorithm, where the imaging processing algorithm refers to an algorithm for processing the original Raw image to obtain the RGB image. The imaging device may use a CMOS (Complementary Metal Oxide Semiconductor) sensor, which collects data as the original Raw image, and after internal imaging processing, outputs the RGB image.

[0056] Due to nonlinear operations or brightening operations in digital image processing, the RGB image may appear overexposed, color cast, etc. The most original Raw image can retain the most abundant and original information of the image. Therefore, the Raw image can provide the most original image information, and the RGB image can provide reconstruction information from the Raw image to the RGB image. Combining the two can accurately depict the image while restoring the content of the image without difference, and has strong robustness, and can adapt to traffic signal lamp color description in different scenes and image target area color description in other application scenarios.

[0057] S200: Input the original Raw image and the RGB image into a first processing model to obtain a color description image output by the first processing model.

[0058] In this embodiment, the first processing model may be a first autoencoder model pre-trained, and the first autoencoder model is used to automatically color the image. The image obtained by color description is also an RGB image. In order to distinguish from the RGB image collected in step S100, the color description image is used in the following description. In other alternative embodiments, the first processing model may also be other types of machine learning models, such as a convolutional neural network.

[0059] S300: Input the color description image and a numerical image indicating the suppression of light halo information into a second processing model to obtain an image after suppressing light halo output by the second processing model.

[0060] In this embodiment, the second processing model is a pre-trained second autoencoder model, and the second autoencoder model is used to automatically perform the light halo suppression processing on the image, so as to realize the automation of light halo suppression, and the effect of light halo suppression can be adjusted by the numerical graph indicating the light halo suppression information. In the numerical graph indicating the light halo suppression information, the area indicating the light halo information and the area not indicating the light halo information are indicated by different numerical values. The light halo information indicated by the numerical graph can be obtained by reading the shape and numerical value of the area indicating the light halo information.

[0061] In this embodiment, the second processing model is a pre-trained second autoencoder model, and the second autoencoder model is used to automatically perform the light halo suppression processing on the image, so as to realize the automation of light halo suppression, and the effect of light halo suppression can be adjusted by the numerical graph indicating the light halo suppression information. In the numerical graph indicating the light halo suppression information, the area indicating the light halo information and the area not indicating the light halo information are indicated by different numerical values. The light halo information indicated by the numerical graph can be obtained by reading the shape and numerical value of the area indicating the light halo information. Figure One In this embodiment, the second processing model is a pre-trained second autoencoder model, and the second autoencoder model is used to automatically perform the light halo suppression processing on the image, so as to realize the automation of light halo suppression, and the effect of light halo suppression can be adjusted by the numerical graph indicating the light halo suppression information. In the numerical graph indicating the light halo suppression information, the area indicating the light halo information and the area not indicating the light halo information are indicated by different numerical values. The light halo information indicated by the numerical graph can be obtained by reading the shape and numerical value of the area indicating the light halo information.

[0062] Therefore, the present application solves the problems of poor robustness and low color accuracy of the traditional traffic signal lamp due to image overexposure, solves the problem that the traditional traffic signal lamp cannot automatically realize color drawing and needs to realize color drawing according to specific rules, and solves the problem that the CPU occupancy rate of the traditional traffic signal lamp is high and real-time processing is difficult. The image processing method does not need to depend on the rules of the traffic signal lamp, can more flexibly realize the color drawing and light halo suppression of the traffic light, can accelerate the neural network operation unit of the mobile terminal device, reduce the running performance of the CPU, and realize the real-time requirement. In addition, the image processing method of the present application can not only be applied to the image processing of the traffic signal lamp, but also be applied to the image processing of other fields that need to draw color and suppress light halo, such as the processing of device indicator light images and the processing of vehicle light images.

[0063] Generally, the output of the imaging processing algorithm is 8 bits, and the numerical range is between 0 and 255. The CMOS sensor data is generally 12 bits or 16 bits, and the numerical range is between 0 and 4095 or 0 and 65535. In addition, due to the limitation of storage space, the Raw image and the RGB image can be distributed and normalized to [0-1] through a simple preprocessing operation, and finally they are used together as the input of the first processing model. In step S200, the original Raw image and the RGB image are input into the first processing model, including the following steps:

[0064] The original Raw image is converted to the sRGB domain and normalized to obtain a first preprocessed image;

[0065] normalizing the RGB image to obtain a second pre-processed image;

[0066] inputting the first pre-processed image and the second pre-processed image into the first processing model.

[0067] In this embodiment, the image processing method further comprises training the first processing model by the following steps:

[0068] (1) collecting a plurality of sample original Raw images and corresponding sample RGB images to form a plurality of sample image pairs.

[0069] Specifically, the Raw images and corresponding RGB images collected by different imaging devices such as cameras under different illuminations, different scenes and different exposure times are paired.

[0070] (2) performing color tracing on the sample RGB image in each sample image pair to obtain a target color tracing image, the number of target color tracing images corresponding to the number of sample image pairs;

[0071] Specifically, the light bead region of the traffic signal lamp is selected for the RGB image, and the color tracing is performed on the light bead region according to the prior information and rules using the PS technology. For example, if the region is judged as red according to the prior information and rules, the corresponding region is assigned a value of (255, 0, 0) using the PS technology, if the region is judged as green, the corresponding region is assigned a value of (0, 255, 0) using the PS technology, and if the region is judged as yellow, the corresponding region is assigned a value of (255, 255, 0) using the PS technology. The processed image is used as the target color tracing image to be fitted by the first processing model. Similarly, when the image processing method is applied to other target region recognition scenarios, the colors can be classified, and the color tracing value corresponding to each color classification can be set. When a region is recognized as a certain color classification, the corresponding region is assigned a value of the corresponding color tracing value using the PS technology.

[0072] Before the sample image pairs are input into the first processing model, the original Raw images and the RGB images in the sample image pairs need to be pre-processed. In addition to the above-mentioned conversion of the Raw image to the sRGB domain and the image normalization, other pre-processing methods can be further included, such as Figure 2A specific preprocessing procedure of the original Raw image is shown, that is, the preprocessing of the original Raw image can include Bayer format unification, Pach Raw processing, expansion to the sRGB domain through nearest neighbor interpolation, normalization processing, horizontal or vertical flipping (sample data augmentation such as horizontal, vertical flipping or random flipping of the image), and finally outputting an sRGB image. In this embodiment, by expanding to the sRGB domain through nearest neighbor interpolation, the numerical range between the input data can be reduced, and the model is more easily converged. Figure 3 A specific preprocessing procedure of the RGB image is shown, which can include normalization processing, horizontal or vertical flipping and the like. Other preprocessing methods can also be selected and used as needed, such as black level correction or other data augmentation processing. In this embodiment, by performing black level correction, normalization and Bayer format rearrangement of the original sensor image, i.e., the original raw image, the training difficulty of the model is reduced, and the convergence of the model is accelerated.

[0073] (3) inputting the sample image pair into the first processing model, and inputting the target stroke image corresponding to the sample image pair into the first processing model as a fitting target, training the first processing model, and obtaining optimal network parameters of the first processing model for realizing image stroke in real-time image processing.

[0074] In this embodiment, before the step S300 of inputting the stroke image and the numerical map indicating the light halo suppression information into the second processing model, the following steps are further included:

[0075] obtaining specified light halo suppression information, the light halo suppression information including a light halo suppression range and / or a light halo suppression level;

[0076] generating a numerical map indicating the light halo suppression information with the same resolution as the stroke image based on the light halo suppression information, the numerical map including a numerical map indicating the light halo suppression range and / or a numerical map indicating the light halo suppression level.

[0077] Therefore, in this embodiment, the light halo range and the light halo level that need to be suppressed can be specified through the numerical map indicating the light halo range and / or the numerical map indicating the light halo level. Here, the light halo range refers to the number of pixels of the light halo range, and the light halo level refers to the degree of suppression of the light halo, for example, the range of supportable light halo levels is set to 0-5, the greater the specified light halo level, the stronger the degree of suppression of the light halo, and the smaller the specified light halo level, the weaker the degree of suppression of the light halo.

[0078] In this embodiment, generating a numerical map indicating the light halo suppression information with the same resolution as the stroke image based on the light halo suppression information includes the following steps:

[0079] based on the specified halo suppression range and the preset minimum and maximum values of the halo suppression range, calculating a numerical value of the halo suppression range, and generating a numerical value map indicating the numerical value of the halo suppression range with the same resolution as the stroke image based on the numerical value of the halo suppression range; and / or,

[0080] based on the specified halo suppression level and the preset minimum and maximum values of the halo suppression level, calculating a numerical value of the halo suppression level, and generating a numerical value map indicating the numerical value of the halo suppression level with the same resolution as the stroke image based on the numerical value of the halo suppression level.

[0081] Specifically, the numerical value of the halo suppression range can be calculated using the following formula:

[0082] y1=(x1-a1) / (b1-a1)

[0083] wherein x1 is the specified halo suppression range, in units of pixel number, a1 is the minimum value of the halo suppression range, in units of pixel number, b1 is the maximum value of the halo suppression range, in units of pixel number, and y1 is the output halo suppression range data. For example, if the specified halo suppression range is 50 pixels and the supported halo range is 0-250 pixels, the normalized numerical value is 0.2, and then a numerical value map with the same resolution as the input stroke image is generated based on this numerical value, with all numerical values being 0.2.

[0084] The generation method of the numerical value map indicating the halo suppression level is similar to that of the numerical value map indicating the halo suppression range. That is, the following formula can be used to calculate:

[0085] y2=(x2-a2) / (b2-a2)

[0086] wherein x2 is the specified halo suppression level, in units of pixel number, a2 is the minimum value of the halo suppression level, in units of pixel number, b2 is the maximum value of the halo suppression level, in units of pixel number, and y2 is the output halo suppression level data.

[0087] The generation method of the numerical value map indicating the halo suppression level is illustrated as follows. If the specified halo suppression range level is 2 and the supported level range is 0-5, the normalized numerical value is 0.4. Therefore, in this embodiment, by generating a numerical value map indicating the halo suppression information and normalizing it, the accuracy of the model can be improved.

[0088] In other embodiments, the numerical value of the specified halo suppression range and the numerical value of the specified halo suppression level can also be obtained in other ways. For example, the ratio of the specified halo suppression range to the maximum value of the halo suppression range is directly calculated, and this ratio is taken as the numerical value of the corresponding halo suppression range. The ratio of the specified halo suppression level to the maximum value of the halo suppression level is directly calculated, and this ratio is taken as the numerical value of the corresponding halo suppression level. In another embodiment, a first mapping table between the specified halo suppression range and the numerical value of the halo suppression range can also be preset, and the first mapping table is searched according to the specified halo suppression range to obtain the numerical value of the corresponding halo suppression range. Similarly, a second mapping table between the specified halo suppression level and the numerical value of the halo suppression level can also be preset, and the second mapping table is searched according to the specified halo suppression level to obtain the numerical value of the corresponding halo suppression level.

[0089] In this embodiment, if the halo suppression position is not limited, halo suppression will be performed in other non-specified areas, such as street lamps, car lights, etc. To solve this technical problem, the specified halo suppression information can also include a specified halo suppression position, and the numerical value map also includes a numerical value map indicating the halo suppression position, in which only the non-zero numerical value is in the specified range, and the rest is 0. Halo suppression can be specified only in a specific area.

[0090] In this embodiment, generating a numerical value map indicating the halo suppression information with the same resolution as the color drawing image based on the halo suppression information includes the following steps:

[0091] Based on the specified halo suppression position, a numerical value map indicating the halo suppression position with the same resolution as the color drawing image is generated, wherein the pixel value corresponding to the halo suppression position is a first value, and the pixel value corresponding to the non-halo suppression position is a second value, and the first value is different from the second value.

[0092] For example, the first value is a non-zero value, and the second value is 0. First, a full 0 numerical value map is created, and the image resolution size is consistent with the color drawing image. Then, the coordinates of the lantern board area are obtained according to the lantern board detection algorithm, and the coordinates of the lantern board area are assigned to the corresponding region of the full 0 numerical value map according to the coordinates, and the normalized numerical value of the halo suppression level is obtained.

[0093] In this embodiment, the image processing method further includes training the second processing model by using the following steps:

[0094] Obtaining a plurality of sample color drawing images;

[0095] A plurality of Gaussian maps indicating different suppression halo information and having the same resolution as the sample color map image are generated, that is, a training data set is made according to different suppression halo ranges and different suppression halo levels, each sample color map image can form a sample image pair with a Gaussian map indicating a suppression halo range, a Gaussian map indicating different suppression halo levels and a Gaussian map indicating a suppression halo position;

[0096] The sample color map image is multiplied pixel by pixel with the corresponding Gaussian map to obtain a target suppression halo image corresponding to each sample image pair;

[0097] Each sample color map image is combined with each numerical map indicating different suppression halo information as input of the second processing model, and the corresponding target suppression halo image is taken as the fitting target of the second processing model, and the second processing model is trained.

[0098] In this embodiment, the first processing model and the second processing model can be a network structure of any image pixel-level task, and here a UResnet network structure is selected as an example for illustration. The specific structure of the UResnet network structure is as follows: Figure 4 Therefore, it includes four convolutional layers (convolution 1, convolution 2, convolution 3, convolution 4), two pooling layers (pooling 1, pooling 2), a residual block and two up-sampling layers (up-sampling 1, up-sampling 2). Figure 5 The specific structure of the residual block is shown, wherein the residual block includes two convolutional layers (convolution 5, convolution 6) and a channel merging layer. Using the UResnet network structure as an image color map and suppression halo autoencoder, the details of the image can be better preserved by removing the cross-layer connection.

[0099] In addition, after the color map image and the numerical map indicating the suppression halo information are input into the second processing model in the step S300, the numerical map indicating the suppression halo information is further added to the input of the second convolutional layer and the input of the residual block. In this embodiment, as shown in Figure 6 In addition to inputting the numerical map indicating the suppression halo level, the numerical map indicating the suppression halo range and the numerical map indicating the suppression halo position into the first convolutional layer (convolution 1) together with the color map image, the numerical map indicating the suppression halo level, the numerical map indicating the suppression halo range and the numerical map indicating the suppression halo position are also input layer by layer into the second convolutional layer (convolution 2) and the residual block, which can strengthen the supervision of the numerical map on the encoding part in the autoencoder, so that the numerical map can better act on the image and improve the effect of suppression halo.

[0100] In this embodiment, the loss function adopted by the autoencoder is composed of a pixel loss function and a high-level semantic loss function, and the expression is as follows:

[0101]

[0102] Wherein, alpha1, alpha2 are weights of pixel loss function, high-level semantic loss function respectively; Y represents output image pixel value of autoencoder; Represent the pixel value of the target image; L pixel Represent the pixel loss function, constrain the image at the pixel level; L vgg Represent the high-level semantic loss function, constrain the image in color and details, etc.

[0103] L pixel The expression is as follows:

[0104]

[0105] L vgg The expression is as follows:

[0106]

[0107] Wherein, Represent the feature vector extracted by the network, generally the feature vector extracted by the network structure of VGG16, through the constraint of the high-level semantic loss function, more real output image can be obtained, details and color texture are reserved.

[0108] As Figure 7 Indicated, the embodiment of the application further provides an image processing system applied to the image processing method, the system comprises:

[0109] Image acquisition module M100 is used for obtaining original Raw image and RGB image after image processing from imaging device;

[0110] First processing module M200 is used for inputting the original Raw image and the RGB image into the first processing model, and obtaining the color drawing image output by the first processing model;

[0111] Second processing module M300 is used for inputting the color drawing image and the numerical value graph indicating the light halo suppression information into the second processing model, and obtaining the image after light halo suppression output by the second processing model.

[0112] The embodiment of the application can improve the accuracy of color drawing by obtaining the corresponding color drawing image through the first processing model based on the original Raw image and the RGB image through the image acquisition module M100 and the first processing module M200, and the numerical value indicating the light halo suppression information is input into the second processing module M300 to obtain the image after light halo suppression output by the second processing model. Figure OneThe corresponding image after light halo suppression is obtained through the second processing model, automatic processing of color drawing and light halo suppression is realized, the overall imaging effect of the image is improved, the process is simplified, and the real-time requirement can be met.

[0113] The embodiment of the present application also provides an image processing device, comprising a processor, a memory, wherein executable instructions of the processor are stored in the memory, and the processor is configured to execute the steps of the image processing method by executing the executable instructions.

[0114] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software, which can be collectively referred to as "circuitry", "module" or "system".

[0115] The electronic device 600 according to this embodiment of the present application will be described below with reference to Figure 8 Figure 8 The electronic device 600 shown is merely an example, and should not impose any limitation on the functions and use range of the embodiment of the present application.

[0116] As shown in Figure 8 , the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0117] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present application described in the electronic prescription flow processing method part of the present specification. For example, the processing unit 610 can execute the steps as shown in Figure 1

[0118] The storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 6201 and / or a cache memory unit 6202, and can further include a read-only memory (ROM) 6203.

[0119] ​​The storage unit 620 can also include a program / utility 6204 having a set of programs / modules 6205, each of which performs one or more of the processes or functions as described herein. The programs / utility 6204 can include, but is not limited to, one or more of: an operating system, one or more applications, other program modules, and program data.

[0120] The bus 630 can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures, and the like.

[0121] The electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices such as printers, scanners, etc.; and / or various types of networks. In general, use of the term device herein is intended to encompass both devices and networks. Communications to, from, and / or within the electronic device 600 can be achieved by an input / output (I / O) interface 650, and can be achieved using any suitable connection (wire and / or wireless), and any suitable protocol, now existing or being developed in the future. Further, the electronic device 600 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, by way of a network adapter 660. The network adapter 660 can be any of a plurality of different types of adapters to enable the electronic device 600 to communicate across networks and / or via a wireless link. It should be appreciated that, while not shown explicitly, other hardware and / or software components could be used in conjunction with the electronic device 600. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0122] By employing the image processing device provided by the present application, the processor executes the executable instructions to execute the image processing method, thereby achieving the beneficial effects of the image processing method.

[0123] The embodiments of the present application also provide a computer readable storage medium for storing a program, which, when executed by a processor, implements the steps of the image processing method. In some possible implementation manners, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing an end device to perform the steps described in the above electronic prescription flow processing method part of the specification according to various exemplary embodiments of the present application when the program product is run on the end device.

[0124] Reference Figure 9As shown, a program product 800 for implementing the above-described method according to an embodiment of the present application is described, which can take the form of a portable compact disc read-only memory (CD-ROM) and includes a program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in the present document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0125] The program product can take any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] The computer readable storage medium can include a data signal transported, for example, in a baseband or as a carrier wave traveling along a co-axial cable or Ethernet. The data signal can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, and the like, or any suitable combination of the above.

[0127] The program code can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider.

[0128] The beneficial effects of the image processing method can be achieved by using the computer readable storage medium provided by the present application, wherein the program stored therein implements the steps of the image processing method when executed.

[0129] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application should not be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the protection scope of the present application.

Claims

1. An image processing method, characterized in that, Includes the following steps: Acquire raw RAW images and processed RGB images from the imaging device; The original Raw image and the RGB image are input into the first processing model to obtain the color image output by the first processing model; The first processing model is used to obtain a corresponding colorimetric image based on the original Raw image and the RGB image, wherein the original Raw image provides original image information and the RGB image provides reconstruction information from the Raw image to the RGB image; The color image and the numerical map indicating halo suppression information are input into the second processing model to obtain the halo-suppressed image output by the second processing model; The training process of the first processing model includes: Multiple original Raw images of samples and their corresponding RGB images are collected to form multiple sample image pairs; Based on the color processing of each pair of sample images in the sample RGB image, the target color image is obtained. The colors are classified and a color value corresponding to each color classification is set. When a region in the sample RGB image is identified as a color classification, the corresponding region is assigned the corresponding color value. The sample image pairs are used as input to the first processing model, and the target color image is used as the fitting target of the first processing model to train the first processing model.

2. The image processing method according to claim 1, characterized in that, Before inputting the colorimetric image and the numerical map indicating suppressed halo information into the second processing model, the following steps are also included: Obtain specified halo suppression information, which includes halo suppression range and / or halo suppression level; Based on the halo suppression information, a numerical map indicating halo suppression information with the same resolution as the colorimetric image is generated. The numerical map includes a numerical map indicating the range of halo suppression and / or a numerical map indicating the level of halo suppression.

3. The image processing method according to claim 2, characterized in that, Generating a numerical map indicating the halo suppression information with the same resolution as the colorimetric image based on the halo suppression information includes the following steps: Based on a specified halo suppression range and preset minimum and maximum values ​​for the halo suppression range, calculate a numerical value for the halo suppression range; and generate a numerical map indicating the halo suppression range with the same resolution as the colorimetric image based on the numerical value of the halo suppression range; and / or, Based on the specified halo suppression level and the preset minimum and maximum values ​​of the halo suppression level, the value of the halo suppression level is calculated, and a numerical map indicating the halo suppression level with the same resolution as the colorimetric image is generated based on the value of the halo suppression level.

4. The image processing method according to claim 2, characterized in that, The specified halo suppression information also includes the specified halo suppression location, and the numerical graph also includes a numerical graph indicating the halo suppression location.

5. The image processing method according to claim 4, characterized in that, Generating a numerical map indicating the halo suppression information with the same resolution as the colorimetric image based on the halo suppression information includes the following steps: Based on the specified halo suppression position, a numerical map indicating the halo suppression position with the same resolution as the color image is generated, wherein the pixel value corresponding to the halo suppression position is a first value, and the pixel value corresponding to the non-halo suppression position is a second value, and the first value is different from the second value.

6. The image processing method according to claim 1, characterized in that, It also includes the following steps: Obtain multiple sample color images; Generate multiple Gaussian plots with different halo suppression information and the same resolution as the sample colorimetric image; Each of the sample color images is multiplied pixel by pixel with the corresponding Gaussian image to obtain the corresponding target suppressed halo image; Each of the sample color images is combined with a corresponding numerical image indicating different halo suppression information as input to the second processing model, and the corresponding target halo suppression image is used as the fitting target of the second processing model to train the second processing model.

7. The image processing method according to claim 6, characterized in that, The second processing model includes a first convolutional layer, a second convolutional layer, a residual block, a third convolutional layer, and a fourth convolutional layer connected in series. After inputting the color image and the numerical map indicating halo suppression information into the second processing model, it also includes adding the numerical map indicating halo suppression information to the input of the second convolutional layer and the input of the residual block.

8. An image processing system, characterized in that, The system, applied to the image processing method according to any one of claims 1 to 7, comprises: The image acquisition module is used to acquire raw RAW images and processed RGB images from the imaging device. The first processing module is used to input the original Raw image and the RGB image into the first processing model to obtain the color image output by the first processing model; The second processing module is used to input the color image and the numerical map indicating halo suppression information into the second processing model to obtain the halo-suppressed image output by the second processing model.

9. An image processing device, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the image processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the image processing method according to any one of claims 1 to 7.

Images