Image processing method, image processing system, electronic device and storage medium

Abstract

An image processing method, an image processing system, an electronic device and a storage medium, the method comprising: splitting the acquired original image to obtain the first channel original data, the second channel original data and the third channel original data; performing super-resolution processing on the first channel original data to obtain the first channel magnified data; performing regional segmentation on the original image, and based on the obtained regional information and classification information, performing correction processing on the second channel original data and the third channel original data to obtain the second channel corrected data and the third channel corrected data; performing amplification processing on the second channel corrected data and the third channel corrected data, and merging them with the first channel magnified data to obtain an enhanced image. The above scheme can be used to adaptively enhance the image, has strong universality and adjustability, and can realize industrial-grade image enhancement applications while reducing the amount of data calculation.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of data processing technology, and in particular, to an image processing method, an image processing system, an electronic device, and a storage medium. Background Art

[0002] In daily life and production activities, images are very important information transmission media. However, in the process of collecting and transmitting images, there are adverse factors that affect imaging, such as ambient lighting, noise interference, hardware lack of support, excessive compression, etc., which lead to poor data quality and make it impossible to read and recognize the useful information in the image. Based on this, image enhancement technology was born. Image enhancement technology highlights the data of the interesting part of the image and weakens or removes the data of the uninteresting part of the image, thereby highlighting the useful information in the image and obtaining an image that is more suitable for analysis and processing.

[0003] At present, traditional image enhancement techniques use frequency domain methods and spatial domain methods. In addition, as deep learning (DL) research has made significant progress, those skilled in the art apply deep learning to image enhancement methods in the hope of improving performance.

[0004] However, current image enhancement technologies are designed based on algorithm architectures based on single image enhancement requirements. They are highly targeted, have simple enhancement effects, have application limitations, and cannot meet the increasingly complex and changing image enhancement needs. Summary of the invention

[0005] In view of this, the embodiments of this specification provide an image processing method, an image processing system, an electronic device and a storage medium, which can adaptively enhance various images, have strong universality and adjustability, and can realize industrial-grade image enhancement applications while reducing the amount of data calculation.

[0006] The embodiment of this specification provides an image processing method, including:

[0007] Get the original image;

[0008] Splitting the original image to obtain first channel original data, second channel original data and third channel original data;

[0009] Performing super-resolution processing on the original data of the first channel to obtain amplified data of the first channel;

[0010] Performing region segmentation on the original image to obtain region information and classification information corresponding to the region information;

[0011] Based on the region information and the classification information, the second channel original data and the third channel original data are corrected to obtain second channel corrected data and third channel corrected data;

[0012] Amplifying the second channel corrected data and the third channel corrected data to obtain second channel amplified data and third channel amplified data;

[0013] The first channel amplified data, the second channel amplified data and the third channel amplified data are combined to obtain an enhanced image.

[0014] Furthermore, the performing region segmentation on the original image to obtain region information and classification information corresponding to the region information includes:

[0015] The original image is input into a preset first deep neural network model to perform region segmentation prediction on the original image to obtain a prediction label set, wherein the prediction label set includes: the region information and classification information corresponding to the region information.

[0016] Further, the correction processing of the second channel original data and the third channel original data includes any one of the following:

[0017] Performing contrast adjustment on the second channel data and the third channel original data;

[0018] Performing color adjustment on the second channel original data and the third channel original data;

[0019] According to a preset adjustment sequence, contrast adjustment and color adjustment are performed on the second channel original data and the third channel original data.

[0020] Further, based on the region information and the classification information, the second channel original data and the third channel original data are corrected to obtain second channel corrected data and third channel corrected data, including:

[0021] The second channel original data, the third channel original data, the region information and the classification information are input into a preset second deep neural network model to correct the second channel original data and the third channel original data to obtain the second channel corrected data and the third channel corrected data.

[0022] Further, the amplifying the second channel corrected data and the third channel corrected data to obtain the second channel amplified data and the third channel amplified data includes:

[0023] The size of the second channel corrected data is enlarged to be consistent with the size of the first channel enlarged data, and the size of the third channel corrected data is enlarged to be consistent with the size of the first channel enlarged data, so as to obtain the second channel enlarged data and the third channel enlarged data.

[0024] Further, the splitting of the original image to obtain the first channel original data, the second channel original data and the third channel original data includes any one of the following:

[0025] Based on the frequency band type and the channel type, the original image is split into high-frequency band original data of the brightness channel, low-frequency band original data of the brightness channel, and the third channel original data, wherein the high-frequency band original data of the brightness channel is used as the first channel original data, and the low-frequency band original data of the brightness channel is used as the second channel original data;

[0026] Based on the frequency band type and the channel type, the original image is split into brightness channel high frequency band original data, brightness channel original data and the third channel original data, wherein the brightness channel high frequency band original data is used as the first channel original data, and the brightness channel original data is used as the second channel original data.

[0027] Furthermore, before merging the first channel amplified data, the second channel amplified data and the third channel amplified data, the method further includes:

[0028] A first correction process is performed on the first channel amplified data, where the first correction process includes at least one of a gain correction, an offset correction, and a curve mapping correction.

[0029] Further, merging the first channel amplified data, the second channel amplified data and the third channel amplified data includes:

[0030] Combining the first channel amplified data and the second channel amplified data to obtain fourth channel amplified data;

[0031] The fourth channel amplified data and the third channel amplified data are combined to obtain the enhanced image.

[0032] Furthermore, the image processing method further includes:

[0033] A second correction process is performed on the enhanced image, where the second correction process includes at least one of transient correction, multi-band correction, and artifact correction.

[0034] Furthermore, before splitting the original image, the method further includes:

[0035] It is determined whether the original image adopts a preset image format, and when it is determined that the original image does not adopt the preset image format, the image format of the original image is converted into the preset image format.

[0036] Furthermore, the obtaining of the original image includes:

[0037] A video frame is obtained from a video stream as the original image.

[0038] The embodiment of this specification also provides an image processing system, including:

[0039] An image acquisition unit, adapted to acquire an original image;

[0040] An image splitting unit, adapted to split the original image to obtain first channel original data, second channel original data and third channel original data;

[0041] A first processing unit, adapted to perform super-resolution processing on the first channel original data to obtain first channel amplified data;

[0042] A region segmentation unit, adapted to perform region segmentation on the original image to obtain region information and classification information corresponding to the region information;

[0043] A second processing unit is adapted to perform correction processing on the second channel original data and the third channel original data according to the region information and the classification information to obtain second channel corrected data and third channel corrected data;

[0044] A third processing unit is adapted to amplify the second channel corrected data and the third channel corrected data to obtain second channel amplified data and third channel amplified data;

[0045] The merging unit is adapted to merge the first channel amplified data, the second channel amplified data and the third channel amplified data to obtain an enhanced image.

[0046] Optionally, the region segmentation unit includes a first deep neural network model, which is suitable for performing region segmentation prediction on the original image and outputting a prediction label set, wherein the prediction label set includes: the region information and classification information corresponding to the region information.

[0047] Optionally, the second processing unit includes a second deep neural network model, and the second deep neural network model is suitable for performing correction processing on the second channel original data and the third channel original data.

[0048] Optionally, the second deep neural network model includes a first sub-model, adapted to perform at least one of contrast adjustment and color adjustment on the second channel raw data, and perform at least one of contrast adjustment and color adjustment on the third channel raw data;

[0049] Alternatively, the second deep neural network model includes a second sub-model and a third sub-model, and the second sub-model and the third sub-model are connected in a preset order, wherein:

[0050] The second sub-model is adapted to perform contrast adjustment on data input therein, and the input data of the second sub-model includes: at least one of the second channel original data, the third channel original data and the data output by the third sub-model;

[0051] The third sub-model is suitable for performing color adjustment on data input therein, and the input data of the third sub-model includes: at least one of the second channel original data, the third channel original data and the data output by the second sub-model.

[0052] An embodiment of the present specification also provides an electronic device for implementing image processing, the electronic device comprising a memory and a processor, the memory storing computer instructions that can be executed on the processor, wherein the processor executes the steps of the method described in any of the above embodiments when executing the computer instructions.

[0053] An embodiment of the present specification also provides a computer-readable storage medium on which computer instructions are stored, wherein the computer instructions, when executed, execute the steps of the method described in any of the above embodiments.

[0054] By using the image processing method provided in the embodiment of this specification, by splitting the original image, the first channel original data, the second channel original data and the third channel original data can be obtained, and the first channel original data can be super-resolution processed to obtain the first channel enlarged data with enhanced details; based on the regional information and classification information obtained by segmenting the original image region, the second channel original data and the third channel original data can be selectively corrected to obtain the second channel corrected data and the third channel corrected data, and then, the first channel enlarged data, the second channel enlarged data and the third channel enlarged data are merged to obtain an enhanced image. As can be seen from the above, the algorithm architecture provided in this specification does not completely rely on deep learning, but combines the deep neural network as an optional solution with the traditional processing method to reduce the complexity and data volume of the algorithm architecture, realize lightweight operations, and improve processing efficiency. In addition, regional segmentation can increase the flexibility of the correction process, so that various images can be adaptively enhanced. Therefore, the image processing method provided in this specification has strong universality and adjustability, and effectively guarantees the effect of image enhancement while reducing the amount of data calculation, and can achieve industrial-level applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the embodiments of this specification or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is a flow chart of an image processing method in an embodiment of this specification;

[0057] Figure 2 is a schematic diagram of a splitting process of an original image in an embodiment of this specification;

[0058] Figure 3 is a schematic diagram of another original image splitting process in an embodiment of this specification;

[0059] Figure 4 is a schematic diagram of a super-resolution processing process in an embodiment of this specification;

[0060] Figure 5 is a schematic diagram of an amplification process in an embodiment of this specification;

[0061] Figure 6 is a schematic diagram of a correction process in an embodiment of this specification;

[0062] Figure 7 is a schematic diagram of another correction processing process in an embodiment of this specification;

[0063] Figure 8 is a flowchart of an image processing method in an embodiment of this specification;

[0064] Fig. 9 It is a structural block diagram of an image processing system in an embodiment of this specification. DETAILED DESCRIPTION

[0065] As described in the background technology, both traditional image enhancement technology and deep learning-based image enhancement technology are designed based on a single image enhancement requirement, which is highly targeted, has a simple enhancement effect, and has application limitations. As image enhancement requirements become more and more complex and changeable, the algorithm architecture is also becoming more and more complex, the amount of data is gradually increasing, and as image enhancement requirements change or increase, it takes a lot of time and manpower to adjust the architecture. Therefore, the current image enhancement technology still has great limitations.

[0066] In order to solve the technical problems existing in the existing image enhancement technology, the embodiment of this specification provides an image processing solution. After obtaining the original image, the original image is split into multiple channel raw data, and based on the regional information and classification information of the original image, the raw data of some channels are adaptively enhanced, thereby strengthening the universality and adjustability of the algorithm architecture, and effectively enhancing useful information by correcting the data, thereby obtaining an image with better enhancement effect.

[0067] In order to enable those skilled in the art to more clearly understand and implement the concepts, implementation schemes and advantages of the embodiments of this specification, the following is a detailed description through specific application scenarios with reference to the accompanying drawings.

[0068] Reference Figure 1 The flowchart of an image processing method in the embodiment of this specification is shown in the figure. In the embodiment of this specification, the method may include the following steps:

[0069] S11, obtaining the original image.

[0070] In a specific implementation, the original image can be obtained from an image captured by an image capture device, from a video stream captured by a video capture device, or from an image file or video file stored in a readable storage medium.

[0071] The original image can be obtained from the video stream by intercepting a video frame or taking a screenshot.

[0072] In addition, the image acquisition device may be a camera, a mobile phone, an aerial camera, etc., and the video acquisition device may be a video camera, a mobile phone, etc. It is understood that this specification does not specifically limit the source, acquisition method, and acquisition path of the original image.

[0073] S12, splitting the original image to obtain first channel original data, second channel original data and third channel original data.

[0074] In a specific implementation, the original image may be split according to a preset splitting method to obtain first channel original data, second channel original data, and third channel original data.

[0075] In an achievable example, the original image can be split into channels according to a preset image channel splitting method. The image channel splitting method can be set according to the channel type actually required. For example, the original image can be split into R (red) channel raw data, G (green) channel raw data, and B (blue) channel raw data. For another example, the original image can be split into Y (brightness) channel raw data, U (first chroma) channel raw data, and V (second chroma) channel raw data. For another example, the original image can be split into brightness channel raw data and chroma channel raw data, wherein the chroma channel raw data may include: U (first chroma) channel raw data and V (second chroma) channel raw data.

[0076] In another feasible example, the original image can be split into frequency bands according to a preset image frequency band splitting method. The image frequency band splitting method can be set according to the frequency band type actually required. For example, the original image can be high-pass filtered to obtain the original data of the high-frequency band of the image; the original image can be low-pass filtered to obtain the original data of the low-frequency band of the image. The frequency band obtained by filtering is related to the filtering strength, and the filtering strength can be set according to the actual usage scenario.

[0077] In practical applications, according to the needs of specific scenarios, the above methods can be selected or combined to split the original image, and then the split data can be set as one of the first channel original data, the second channel original data and the third channel original data according to a preset configuration method.

[0078] For example, Figure 2The figure shows a schematic diagram of the splitting process of an original image. By performing channel splitting on the original image, the original data Y of the luminance channel and the original data UV of the chrominance channel can be obtained. After the original data Y of the luminance channel is low-pass filtered, the original data LFY of the low-frequency band of the luminance channel can be obtained. The original data LFY of the low-frequency band of the luminance channel is subtracted from the original data Y of the luminance channel, that is, the original data LFY of the low-frequency band of the luminance channel is removed from the original data Y of the luminance channel, and the original data HFY of the high-frequency band of the luminance channel can be obtained. According to the preset configuration mode, the original data HFY of the high-frequency band of the luminance channel can be set as the original data of the first channel, and the original data LFY of the low-frequency band of the luminance channel can be set as the original data of the second channel, that is, the original data of the first channel and the original data of the second channel are two components of the original data of the luminance channel; and the original data UV of the chrominance channel can be set as the original data of the third channel.

[0079] For example, Figure 3 As shown, it is a schematic diagram of another splitting process of the original image. By performing channel splitting on the original image, the original data Y of the luminance channel and the original data UV of the chrominance channel can be obtained. After the original data Y of the luminance channel is subjected to high-pass filtering, the original data HFY of the high-frequency band of the luminance channel can be obtained. According to the preset configuration mode, the original data HFY of the high-frequency band of the luminance channel can be set as the original data of the first channel, and the original data Y of the luminance channel can be set as the original data of the second channel, that is, the original data of the first channel is a component of the original data of the luminance channel; and the original data UV of the chrominance channel can be set as the original data of the third channel.

[0080] It is understandable that the above examples are only for illustrative purposes. In actual applications, the splitting method and channel setting of the original image can be set according to actual needs, and this manual does not limit this.

[0081] It should be noted that in some applications, the channel original data obtained by splitting may be called differently, for example, the brightness channel original data may be called the brightness channel original image, etc. This specification does not limit the name of the splitting result.

[0082] S13, performing super-resolution processing on the first channel original data to obtain first channel amplified data.

[0083] In specific implementations, due to environmental lighting, noise interference, hardware lack of support, excessive compression, etc., the raw data of each channel may be insufficiently collected or collected incorrectly, resulting in blurred images. Super-resolution (SR) processing of the raw data of the first channel can construct more pixels of the corresponding channel, while enlarging the size of the raw data of the first channel, enriching the detail information of the raw data of the first channel, thereby improving the resolution of the raw data of the first channel and ensuring the imaging quality of the raw data of the first channel.

[0084] Furthermore, a model constructed based on a deep neural network (DNN) (i.e., a DNN super-resolution model) can be used to perform super-resolution processing on the original data of the first channel to improve the resolution of the original data of the first channel. Figure 4 As shown in FIG. 1 , it is a schematic diagram of a super-resolution processing process. After the DNN super-resolution model performs super-resolution processing on the first channel original data P1 of size a*b, the size of the first channel original data is enlarged to (a*N)*(b*N), and the resolution of the first channel original data P1 is improved to obtain the first channel enlarged data P2. Among them, the enlargement factor N is determined by the parameters of the DNN super-resolution model.

[0085] S14, performing region segmentation on the original image to obtain region information and classification information corresponding to the region information.

[0086] In a specific implementation, the original image is segmented into regions, and different regions in the original image may be labeled, for example, skin, grass, sky, high light, low light and other regions in the original image may be labeled.

[0087] Specifically, the regions in the original image can be identified, segmented and labeled by a traditional region segmentation method or a DNN-based region segmentation method. Among them, the traditional region segmentation method can include a histogram method, a contour fitting method, a region growing method and a split-merge hybrid method, etc. The DNN-based region segmentation method can adopt various DNN region segmentation models, such as FCN (Fully Convolutional Networks) model, DeepLab (a semantic segmentation neural network) model, etc.

[0088] In an achievable example, the original image is input into a preset first deep neural network model to perform region segmentation prediction on the original image to obtain a prediction label set, wherein the first deep neural network model may include one or more DNN region segmentation models, and the prediction label set consists of prediction labels, and the prediction labels are used to characterize the predicted region information and the classification information corresponding to the region information.

[0089] S15, based on the region information and the classification information, correcting the second channel original data and the third channel original data to obtain second channel corrected data and third channel corrected data.

[0090] In a specific implementation, the type of the corresponding region in the second channel raw data can be determined according to the region information and the classification information, and the type of the corresponding region in the third channel raw data can be determined; then, according to the type of the region, the corresponding type of correction processing can be performed on the corresponding region in the second channel raw data, and the corresponding type of correction processing can be performed on the corresponding region in the third channel raw data. Thus, the correction method of the data can be adjusted according to the classification information, and the correction processing process is more flexible and changeable, so as to adapt to more different types of images.

[0091] For example, based on the region information and the classification information, it can be determined that region X in the second channel raw data is a tree, and tree-related correction processing can be performed on region X in the second channel raw data.

[0092] For another example, based on the region information and the classification information, it can be determined that region Y in the third channel raw data is a human being, and human-related correction processing can be performed on region X in the second channel raw data.

[0093] S16, amplifying the second channel corrected data and the third channel corrected data to obtain second channel amplified data and third channel amplified data.

[0094] In a specific implementation, the size of the second channel corrected data can be enlarged to be consistent with the size of the first channel amplified data, and the size of the third channel corrected data can be enlarged to be consistent with the size of the first channel amplified data, so as to obtain the second channel amplified data and the third channel amplified data.

[0095] For example, Figure 5 As shown, it is a schematic diagram of an amplification process, combined with reference Figure 4, the second channel corrected data P3 is enlarged by a preset enlargement method, and the size of the second channel corrected data P3 is enlarged by N*N times. The size of the obtained second channel enlarged data P4 is (a*N)*(b*N), which is consistent with the size of the first channel enlarged data P2.

[0096] Among them, at least one up-sampling method such as bilinear difference and bicubic difference can be used to achieve the size enlargement of the second channel correction data and the third channel correction data.

[0097] It can be understood that the magnification processing is used to enlarge the size of the data and may not involve the function of improving the data resolution. Compared with the super-resolution processing, the method used in the magnification processing is relatively simple, the required parameters are relatively few, the processing process is relatively simple, and the processing efficiency is faster.

[0098] After the amplification process, the sizes of the second channel amplified data, the third channel amplified data and the first channel amplified data are consistent, which is convenient for subsequent data merging.

[0099] S17, merging the first channel amplified data, the second channel amplified data and the third channel amplified data to obtain an enhanced image.

[0100] From the above, it can be seen that the algorithm architecture provided in this specification does not rely entirely on deep learning, but combines deep neural networks as an optional solution with traditional processing methods to reduce the complexity and data volume of the algorithm architecture, achieve lightweight operations, and improve processing efficiency. In addition, region segmentation can increase the flexibility of the correction process, thereby enabling adaptive enhancement of various images. Therefore, the image processing method provided in this specification has strong universality and adjustability, and effectively guarantees the effect of image enhancement while reducing the amount of data calculation, and can be applied at an industrial level.

[0101] It should be noted that there is no necessary order between some steps in the above embodiments. They can be executed simultaneously or in sequence without causing any contradiction, and the order can be swapped. For example, when actually executing the steps of the image processing method provided in this specification, it is only necessary to set step S14 to be executed after step S11 and before step S15. Step S14 can be executed synchronously with other steps, such as step S14 can be executed synchronously with step S12; step S14 can also be executed in a set order, such as step S14 can be executed after step S12. The embodiments of this specification do not specifically limit the order of steps.

[0102] In a specific implementation, there may be a situation where the image format of the original image does not match the preset image channel splitting method. For example, the original image is in RGB format, and the preset image channel splitting method is to split the original image into brightness channel raw data and chrominance channel raw data. In this case, the image format of the original image can be converted from RGB format to YUV format. To this end, the image format of the original image can be converted into an image format that matches the image channel splitting method, thereby ensuring that the original image can be accurately split into channels.

[0103] Furthermore, in order to determine whether image format conversion is required, before splitting the original image, the image format adopted by the original image can be judged, and when it is determined that the original image does not adopt the preset image format, the image format of the original image can be converted into the preset image format, thereby converting the image format of the original image into an image format that matches the image channel splitting method.

[0104] For example, after acquiring the original image, it is determined whether the original image is in YUV format. If so, the image splitting operation is continued; otherwise, the image format of the original image is converted into YUV format before the image splitting operation is performed.

[0105] In a specific implementation, the brightness jump of an image can better reflect the real boundary than the color jump, and the image frequency can characterize the severity of the grayscale change in the image. The high-frequency band corresponds to the part of the image that changes dramatically, that is, the edge (contour) and details of the original image. Therefore, the high-frequency band raw data of the brightness channel can be split from the original image and used as the first channel raw data for super-resolution processing, thereby effectively enhancing the details and edges in the image.

[0106] In a specific implementation, in order to reduce the amount of data and improve data processing efficiency, the second channel original data and the third channel original data can be merged first, and then the correction processing is performed, and the correction processing result includes the second channel corrected data and the third channel corrected data. Then, the correction processing result is amplified, and after the amplification processing is completed, the amplification processing result is split into the second channel amplified data and the third channel amplified data.

[0107] In a specific implementation, the correction processing may include at least one of contrast adjustment and color adjustment. Accordingly, the correction processing performed on the second channel original data and the third channel original data may include any one of the following:

[0108] 1) performing contrast adjustment on the second channel data and the third channel original data;

[0109] 2) performing color adjustment on the second channel original data and the third channel original data;

[0110] 3) According to a preset adjustment sequence, contrast adjustment and color adjustment are performed on the second channel raw data and the third channel raw data. For example, contrast adjustment is performed on the second channel raw data and the third channel raw data first, and then color adjustment is performed; for another example, color adjustment is performed on the second channel raw data and the third channel raw data first, and then contrast adjustment is performed.

[0111] In a specific implementation, the second channel original data, the third channel original data, the region information and the classification information can be input into a preset second deep neural network model (i.e., a model built based on DNN) to correct the second channel original data and the third channel original data to obtain the second channel corrected data and the third channel corrected data.

[0112] In an optional example, the second deep neural network model may include a sub-model, namely a first sub-model, by which at least one of contrast adjustment and color adjustment can be performed on the second channel raw data, and at least one of contrast adjustment and color adjustment can be performed on the third channel raw data.

[0113] For example, Figure 6 As shown, it is a schematic diagram of a correction processing process, in which the second channel original data P5 is input into the second deep neural network model, and the contrast and color of the second channel original data are adjusted by the first sub-model to obtain the second channel corrected data P6.

[0114] In another optional example, the second deep neural network model may include two sub-models, namely a second sub-model and a third sub-model, and contrast adjustment may be performed through the second sub-model, and color adjustment may be performed through the third sub-model.

[0115] According to the connection relationship between the second sub-model and the third sub-model (such as series connection or parallel connection, etc.), the input data of the second sub-model may include: at least one of the second channel original data, the third channel original data, and the data output by the third sub-model. Correspondingly, the input data of the third sub-model may include: at least one of the second channel original data, the third channel original data, and the data output by the second sub-model.

[0116] For example, Figure 7As shown, it is a schematic diagram of another correction processing process, in which the third channel original data P7 is input into the second deep neural network model, the contrast of the third channel original data is adjusted by the second sub-model, and the color of the result output by the second sub-model is adjusted by the third sub-model to obtain the third channel corrected data P8.

[0117] In specific implementations, the dynamic range of an image can be divided into standard dynamic range (SDR) and high dynamic range (HDR). Different contrast adjustment methods and color adjustment methods can be selected according to different dynamic range requirements. Among them, HDR requires more dynamic range and image details than SDR. Therefore, in order to meet the needs of HDR, deep learning can be used to improve the contrast adjustment effect and color adjustment effect. For example, a DNN-based standard dynamic range conversion to high dynamic range (SDR2HDR) model can be used to achieve contrast adjustment, and a DNN-based gamut mapping algorithm (GMA) model can be used to achieve color adjustment.

[0118] In a specific implementation, before merging the first channel amplified data, the second channel amplified data and the third channel amplified data, the method may further include: performing a first correction process on the first channel amplified data, wherein the first correction process includes at least one of gain correction, offset correction and curve mapping correction. Thus, the use requirements of users can be met.

[0119] In a specific implementation, the merging of the first channel amplified data, the second channel amplified data and the third channel amplified data may specifically include: merging the first channel amplified data and the second channel amplified data to obtain fourth channel amplified data; merging the fourth channel amplified data and the third channel amplified data to obtain the enhanced image.

[0120] In a specific implementation, the image processing method may further include: performing a second correction process on the enhanced image to obtain a corrected enhanced image.

[0121] The second correction process includes at least one of transient correction, multi-band correction and artifact correction. These processes can enhance the visibility of multi-bands, improve chromaticity transients and reduce artifacts, improve data quality, and thus improve the clarity of images or video frames.

[0122] In order to enable those skilled in the art to more clearly understand and implement the concepts, implementation schemes and advantages of the embodiments of this specification, the following is a detailed description through embodiments with reference to the accompanying drawings.

[0123] In one embodiment of the present invention, Figure 8 As shown, it is a flowchart of an image processing method. After acquiring the original image, the original image is split to obtain the first channel original data A1, the second channel original data A2 and the third channel original data A3, and the original image is segmented into regions to obtain a predicted label set A4, wherein the predicted label set A4 includes: predicted region information and classification information corresponding to the region information.

[0124] In the first branch Z1, super-resolution processing is performed on the first channel original data A1 to obtain first channel amplified data B1, and then first correction processing is performed on the first channel amplified data B1 to obtain corrected first channel amplified data C1.

[0125] In the second branch Z2, based on the area information and the classification information, the second channel original data A2 and the third channel original data A3 are corrected to obtain second channel corrected data B2 and third channel corrected data B3, and then the second channel corrected data B2 and the third channel corrected data B3 are amplified to obtain second channel amplified data C2 and third channel amplified data C3.

[0126] Then, the corrected first channel amplified data C1 and the second channel amplified data C2 are combined to obtain fourth channel amplified data C4, and the fourth channel amplified data C4 and the third channel amplified data C3 are combined to obtain an enhanced image.

[0127] In an optional example, if Figure 8 As shown, after obtaining the enhanced image, the enhanced image may be subjected to a second correction process to obtain a corrected enhanced image. The second correction process includes: at least one of transient correction, multi-band correction, and artifact correction.

[0128] It can be understood that the above describes multiple embodiment schemes provided by the embodiments of this specification, and the various optional methods introduced in each embodiment scheme can be combined and cross-referenced with each other without conflict, thereby extending a variety of possible embodiment schemes, which can all be considered as embodiment schemes disclosed and open in this specification.

[0129] This specification also provides an image processing system corresponding to the above-mentioned image processing method, which is described in detail below through specific embodiments with reference to the accompanying drawings. It should be noted that the image processing system described below can be considered as a functional module required to implement the image processing method provided in this specification; the content of the image processing system described below can be referenced to the content of the image processing method described above.

[0130] Reference Fig. 9 The structural block diagram of an image processing system in an embodiment of this specification is shown. In this embodiment of this specification, the image processing system M0 may include:

[0131] An image acquisition unit M1, adapted to acquire an original image;

[0132] An image splitting unit M2 is adapted to split the original image to obtain first channel original data, second channel original data and third channel original data;

[0133] A first processing unit M3 is adapted to perform super-resolution processing on the first channel original data to obtain first channel amplified data;

[0134] A region segmentation unit M4, adapted to perform region segmentation on the original image to obtain region information and classification information corresponding to the region information;

[0135] The second processing unit M5 is adapted to perform correction processing on the second channel original data and the third channel original data according to the region information and the classification information to obtain second channel corrected data and third channel corrected data;

[0136] A third processing unit M6 is adapted to amplify the second channel corrected data and the third channel corrected data to obtain second channel amplified data and third channel amplified data;

[0137] The merging unit M7 is adapted to merge the first channel amplified data, the second channel amplified data and the third channel amplified data to obtain an enhanced image.

[0138] From the above, it can be seen that the algorithm architecture provided in this specification does not rely entirely on deep learning, but combines deep neural networks as an optional solution with traditional processing methods to reduce the complexity and data volume of the algorithm architecture, achieve lightweight operations, and improve processing efficiency. In addition, region segmentation can increase the flexibility of the correction process, thereby enabling adaptive enhancement of various images. Therefore, the image processing method provided in this specification has strong universality and adjustability, and effectively guarantees the effect of image enhancement while reducing the amount of data calculation, and can be applied at an industrial level.

[0139] In a specific implementation, the region segmentation unit M4 may include a first deep neural network model M41, which is suitable for performing region segmentation prediction on the original image and outputting a predicted label set, wherein the predicted label set may include: the region information and classification information corresponding to the region information.

[0140] In a specific implementation, the second processing unit M5 includes a second deep neural network model M51, and the second deep neural network model M51 is suitable for correcting the second channel original data and the third channel original data.

[0141] In an achievable embodiment, in combination with reference Fig. 9 and Figure 6 The second deep neural network model M51 may include a first sub-model, suitable for performing at least one of contrast adjustment and color adjustment on the second channel raw data, and performing at least one of contrast adjustment and color adjustment on the third channel raw data.

[0142] In another possible embodiment, in combination with reference Fig. 9 and Figure 7 , the second deep neural network model M51 may include a second sub-model and a third sub-model, and the second sub-model and the third sub-model are connected in a preset order, wherein:

[0143] The second sub-model is suitable for performing contrast adjustment on data input therein, and the input data of the second sub-model includes: at least one of the second channel original data, the third channel original data and the data output by the third sub-model.

[0144] The third sub-model is suitable for performing color adjustment on data input therein, and the input data of the third sub-model includes: at least one of the second channel original data, the third channel original data and the data output by the second sub-model.

[0145] In a specific implementation, the image processing system M0 may further include:

[0146] The fourth processing unit M8 is located between the first processing unit M3 and the merging unit M7, and is suitable for performing a first correction processing on the first channel amplified data, and sending the first correction processing result as the first channel amplified data to the merging unit M7, wherein the first correction processing includes at least one of gain correction, offset correction, and curve mapping correction.

[0147] In a specific implementation, the merging unit M7 includes:

[0148] A first merging subunit M71 is adapted to merge the first channel amplified data and the second channel amplified data to obtain fourth channel amplified data;

[0149] The second merging subunit M72 is adapted to secondly merge the fourth channel amplified data and the third channel amplified data to obtain the enhanced image.

[0150] In a specific implementation, the image processing system M0 may further include:

[0151] The correction unit M9 is adapted to perform a second correction process on the enhanced image, wherein the second correction process includes at least one of transient correction, multi-band correction, and artifact correction.

[0152] It can be understood that the above describes multiple embodiment schemes provided by the embodiments of this specification, and the various optional methods introduced in each embodiment scheme can be combined and cross-referenced with each other without conflict, thereby extending a variety of possible embodiment schemes, which can all be considered as embodiment schemes disclosed and open in this specification.

[0153] The embodiment of this specification also provides an electronic device for implementing image processing, the electronic device may include a memory and a processor, the memory is suitable for storing one or more computer instructions that can be run on the processor, and the processor executes the steps of the image processing method described in any of the above embodiments when running the computer instructions. The specific steps can refer to the above embodiments and will not be repeated here.

[0154] Optionally, the processor can be implemented by a processing chip such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), etc., or can be implemented by an application specific integrated circuit (ASIC) or one or more integrated circuits configured to implement the embodiments of this specification.

[0155] Optionally, the memory may include a random access memory (Random Access Memory, RAM) memory, and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory.

[0156] In a specific implementation, the data processing device may further include an expansion interface suitable for connecting with other devices to achieve data interaction. For example, the electronic device may be connected to a display device to play the enhanced image or a video stream containing the enhanced image.

[0157] The embodiment of this specification also provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed, the steps of the image processing method described in any of the above embodiments of this specification can be executed. The computer-readable storage medium can be various suitable readable storage media such as an optical disk, a mechanical hard disk, and a solid-state hard disk. The instructions stored on the computer-readable storage medium execute the steps of the image processing method described in any of the above embodiments. For details, please refer to the above embodiments and will not be repeated here.

[0158] The computer-readable storage medium may include, for example, any suitable type of memory unit, memory device, memory item, memory medium, storage device, storage item, storage medium and / or storage unit, such as memory, removable or non-removable medium, erasable or non-erasable medium, writable or rewritable medium, digital or analog medium, hard disk, floppy disk, compact disk read-only memory (CD-ROM), compact disk recordable (CD-R), compact disk rewritable (CD-RW), optical disk, magnetic medium, magneto-optical medium, removable memory cards or disks, various types of digital versatile discs (DVDs), magnetic tapes, cassettes, etc.

[0159] Computer instructions may include any suitable type of code, e.g., source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, etc., implemented using any suitable high-level, low-level, object-oriented, visual, compiled and / or interpreted programming language.

[0160] It should be noted that the terms "first" and "second" in the embodiments of this specification are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Moreover, the terms "first", "second", etc. are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein.

[0161] Although the embodiments of this specification are disclosed as above, the embodiments of this specification are not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this specification. Therefore, the protection scope of the embodiments of this specification shall be subject to the scope defined by the claims.

Claims

1. An image processing method, characterized in that: include: Get the original image; Splitting the original image to obtain first channel original data, second channel original data and third channel original data; Performing super-resolution processing on the original data of the first channel to obtain amplified data of the first channel; Performing region segmentation on the original image to obtain region information and classification information corresponding to the region information; Based on the region information and the classification information, the second channel original data and the third channel original data are corrected to obtain second channel corrected data and third channel corrected data; Amplifying the second channel corrected data and the third channel corrected data to obtain second channel amplified data and third channel amplified data; The first channel amplified data, the second channel amplified data and the third channel amplified data are combined to obtain an enhanced image.

2. The image processing method according to claim 1, characterized in that: The performing region segmentation on the original image to obtain region information and classification information corresponding to the region information includes: The original image is input into a preset first deep neural network model to perform region segmentation prediction on the original image to obtain a prediction label set, wherein the prediction label set includes: the region information and classification information corresponding to the region information.

3. The image processing method according to claim 1, characterized in that: The correcting process of the second channel original data and the third channel original data includes any one of the following: Performing contrast adjustment on the second channel raw data and the third channel raw data; Performing color adjustment on the second channel original data and the third channel original data; According to a preset adjustment sequence, contrast adjustment and color adjustment are performed on the second channel original data and the third channel original data.

4. The image processing method according to claim 1, characterized in that: The method of performing correction processing on the second channel original data and the third channel original data based on the region information and the classification information to obtain second channel corrected data and third channel corrected data includes: The second channel original data, the third channel original data, the region information and the classification information are input into a preset second deep neural network model to correct the second channel original data and the third channel original data to obtain the second channel corrected data and the third channel corrected data.

5. The image processing method according to claim 1, characterized in that: The amplifying the second channel corrected data and the third channel corrected data to obtain the second channel amplified data and the third channel amplified data includes: The size of the second channel corrected data is enlarged to be consistent with the size of the first channel enlarged data, and the size of the third channel corrected data is enlarged to be consistent with the size of the first channel enlarged data, so as to obtain the second channel enlarged data and the third channel enlarged data.

6. The image processing method according to claim 1, characterized in that: The splitting of the original image to obtain the first channel original data, the second channel original data and the third channel original data includes any one of the following: Based on the frequency band type and the channel type, the original image is split into high-frequency band original data of the brightness channel, low-frequency band original data of the brightness channel, and the third channel original data, wherein the high-frequency band original data of the brightness channel is used as the first channel original data, and the low-frequency band original data of the brightness channel is used as the second channel original data; Based on the frequency band type and the channel type, the original image is split into brightness channel high frequency band original data, brightness channel original data and the third channel original data, wherein the brightness channel high frequency band original data is used as the first channel original data, and the brightness channel original data is used as the second channel original data.

7. The image processing method according to claim 1, characterized in that: Before merging the first channel amplified data, the second channel amplified data and the third channel amplified data, the method further includes: A first correction process is performed on the first channel amplified data, where the first correction process includes at least one of a gain correction, an offset correction, and a curve mapping correction.

8. The image processing method according to claim 1, characterized in that: The merging of the first channel amplified data, the second channel amplified data and the third channel amplified data comprises: Combining the first channel amplified data and the second channel amplified data to obtain fourth channel amplified data; The fourth channel amplified data and the third channel amplified data are combined to obtain the enhanced image.

9. The image processing method according to claim 8, characterized in that: Also includes: A second correction process is performed on the enhanced image, where the second correction process includes at least one of transient correction, multi-band correction, and artifact correction.

10. The image processing method according to claim 1, characterized in that: Before splitting the original image, the method further includes: It is determined whether the original image adopts a preset image format, and when it is determined that the original image does not adopt the preset image format, the image format of the original image is converted into the preset image format.

11. The image processing method according to claim 1, characterized in that: The obtaining of the original image comprises: A video frame is obtained from a video stream as the original image.

12. An image processing system, characterized in that: include: An image acquisition unit, adapted to acquire an original image; An image splitting unit, adapted to split the original image to obtain first channel original data, second channel original data and third channel original data; A first processing unit, adapted to perform super-resolution processing on the first channel original data to obtain first channel amplified data; A region segmentation unit, adapted to perform region segmentation on the original image to obtain region information and classification information corresponding to the region information; A second processing unit is adapted to perform correction processing on the second channel original data and the third channel original data according to the region information and the classification information to obtain second channel corrected data and third channel corrected data; A third processing unit is adapted to amplify the second channel corrected data and the third channel corrected data to obtain second channel amplified data and third channel amplified data; The merging unit is adapted to merge the first channel amplified data, the second channel amplified data and the third channel amplified data to obtain an enhanced image.

13. The image processing system according to claim 12, characterized in that: The region segmentation unit includes a first deep neural network model, which is suitable for performing region segmentation prediction on the original image and outputting a prediction label set, wherein the prediction label set includes: the region information and classification information corresponding to the region information.

14. The image processing system according to claim 12, characterized in that: The second processing unit includes a second deep neural network model, which is suitable for performing correction processing on the second channel original data and the third channel original data.

15. The image processing system according to claim 14, characterized in that: The second deep neural network model includes a first sub-model, which is suitable for performing at least one of contrast adjustment and color adjustment on the second channel raw data, and performing at least one of contrast adjustment and color adjustment on the third channel raw data; Alternatively, the second deep neural network model includes a second sub-model and a third sub-model, and the second sub-model and the third sub-model are connected in a preset order, wherein: The second sub-model is adapted to perform contrast adjustment on data input therein, and the input data of the second sub-model includes: at least one of the second channel original data, the third channel original data and the data output by the third sub-model; The third sub-model is suitable for performing color adjustment on data input therein, and the input data of the third sub-model includes: at least one of the second channel original data, the third channel original data and the data output by the second sub-model.

16. An electronic device for implementing image processing, the electronic device comprising a memory and a processor, the memory storing computer instructions that can be run on the processor, characterized in that: When the processor executes the computer instructions, the steps of the method according to any one of claims 1 to 11 are performed.

17. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed, the steps of the method according to any one of claims 1 to 11 are executed.

Images