A camera image enhancement method and system based on deep learning

Abstract

The present invention discloses a camera image enhancement method and system based on deep learning, which relates to the field of image processing technology and includes the following steps: inputting the acquired original image sequence into a deep neural network model, outputting the semantic segmentation result of the current frame image through a segmentation network, and guiding the deep neural network model to learn the target of interest in the current frame image so as to fuse the image data of the same target in different frame images; outputting the brightness of the current frame image through a brightness estimation network to adjust the camera aperture; and outputting an enhanced image corresponding to the current frame image through an image enhancement network. The camera image enhancement method and system fully retain the details of the picture.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a camera image enhancement method and system based on deep learning. Background Art

[0002] After capturing raw data, cameras using CMOS sensors typically undergo a series of image processing steps called ISPs to achieve optimal visual quality. Traditional ISP algorithms can effectively process a variety of images in well-lit scenes. However, they are less effective in low-light scenes with moving objects. The main difficulty lies in the fact that traditional ISP methods do not distinguish between objects within the image, making both exposure time selection and multi-frame data fusion ineffective. Long exposure times or direct weighted fusion of multiple frames can achieve good results for static portions of the image, but can introduce a smearing effect for moving objects. Short exposure times, while preserving details of moving objects, can result in noticeable noise in the overall image. Considerable work has been conducted using AI to improve image visual quality, but most of this work is suitable for processing static images. Directly applying these methods to video sequences, due to the presence of moving objects, makes it difficult to achieve ideal visual quality without leveraging preceding and following temporal information.

[0003] Due to the successful application of deep learning in many fields, the concept of deep learning problem-solving has become standardized: simply give a network structure, prepare training data, and train until convergence. However, when it comes to specific problems, how to design the network structure, especially the input and output data formats; how to guide the network to learn valuable information; and how to obtain more effective training data are all issues that require extensive research and technical solutions. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a camera image enhancement method and system based on deep learning, which fully retains the details of the picture.

[0005] The present invention proposes a camera image enhancement method based on deep learning, comprising the following steps:

[0006] Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image. The deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network. The output of the backbone network is connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, respectively. The backbone network inputs the original image sequence;

[0007] The semantic segmentation result of the current frame image is output through the segmentation network to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images;

[0008] The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture;

[0009] The enhanced image corresponding to the current frame image is output through the image enhancement network.

[0010] Furthermore, during the training of the deep neural network model, if the original image sequence is a combination of images of different frames in a dynamic scene, the process is as follows:

[0011] Perform feature matching between the current frame image and the historical frame image, and filter the matching results through the semantic segmentation results of the deep neural network model to remove false matches;

[0012] Extract the segmentation area of the class of interest in the original image sequence, cluster it according to the feature matching results, and the features clustered in one class represent the segmentation results of the target in different frame images;

[0013] All segmentation results clustered in one category are matched to the segmentation results of the current frame image through perspective transformation, and all these segmentation results are weighted fused to obtain the enhanced result of the target of interest in the current frame image;

[0014] Paste the image enhancement result back to the corresponding position of the current frame image, and obtain the enhanced image corresponding to the current frame image after manual screening;

[0015] The deep neural network model performs semantic segmentation on the current frame image to obtain the semantic segmentation result segment image of the current frame image;

[0016] The brightness index of the original image sequence is manually labeled. If the brightness index is greater than or equal to the set brightness threshold, the brightness index is labeled as 1. If the brightness index is less than the set brightness threshold, the brightness index is labeled as 0. The deep neural network model outputs the brightness index of the current frame image to serve as data to assist in camera aperture adjustment.

[0017] Furthermore, during the training of the deep neural network model, if the original image sequence is a combination of images of different frames in a static scene, the training process of the deep neural network model is as follows:

[0018] Install two cameras with the same hardware configuration side by side, set one camera to a short exposure time and the other to a long exposure time;

[0019] Two fixed cameras are used to capture short videos of still images of various scenes;

[0020] The short exposure time image sequence is used as the input of the neural network model, and the long exposure time image sequence is used as the enhanced image output by the neural network model;

[0021] The short exposure time image sequence is subjected to image semantic segmentation by the neural network model, and the semantic segmentation result segment image corresponding to the short exposure time image sequence is output;

[0022] The brightness index of the image sequence with short exposure time and the image sequence with long exposure time are manually labeled as the brightness index of the corresponding picture, which is used as data to assist the camera aperture adjustment.

[0023] Furthermore, the backbone network includes a splicing layer, an encoder group and a decoder group connected in sequence, the output of the decoder group is respectively connected to the input of the segmentation network, the input of the brightness estimation network and the input of the image enhancement network, and the splicing layer inputs the original image sequence.

[0024] Furthermore, the encoder group includes encoders enc1, enc2, enc3, enc4, enc5, and enc6, and the decoder group includes decoders dec1, dec2, dec3, dec4, and dec5, and enc1, enc2, enc3, enc4, enc5, enc6, dec5, dec4, dec3, dec2, and dec1 are connected in sequence;

[0025] The outputs of enc6, dec1, dec2, dec3, dec4, and dec5 are all connected to the input of the segmentation network;

[0026] The output of dec1 is connected to the input of the brightness estimation network and the input of the image enhancement network respectively.

[0027] Furthermore, the segmentation network includes a segmentation channel normalization layer, upsampling, a segmentation splicing layer concat and a segmentation two-dimensional convolution Cov2d. The output of dec1 is connected to the segmentation splicing layer concat through the segmentation channel normalization layer. The outputs of dec2, dec3, dec4 and dec5 are all connected to the segmentation splicing layer concat in turn through the segmentation channel normalization layer and upsampling. The output of the segmentation splicing layer concat is connected to the input of the segmentation two-dimensional convolution Cov2d, and the segmentation two-dimensional convolution Cov2d outputs the semantic segmentation result.

[0028] Furthermore, the brightness estimation network includes an auxiliary channel normalization layer and a conversion layer, the output of dec1 is connected to the auxiliary channel normalization layer and the conversion layer in sequence, and the conversion layer outputs the brightness of the image.

[0029] Furthermore, the image enhancement network includes an enhanced channel normalization layer and an enhanced two-dimensional convolution Cov2d, the output of dec1 is connected to the enhanced channel normalization layer and the enhanced two-dimensional convolution Cov2d in sequence, and the enhanced two-dimensional convolution Cov2d outputs an enhanced image corresponding to the image.

[0030] A camera image enhancement system based on deep learning enhances the input image based on a deep neural network model. The specific enhancement process is as follows:

[0031] Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image. The deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network. The output of the backbone network is connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, respectively. The backbone network inputs the original image sequence;

[0032] The semantic segmentation result of the current frame image is output through the segmentation network to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images;

[0033] The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture;

[0034] The enhanced image corresponding to the current frame image is output through the image enhancement network.

[0035] The advantages of the camera image enhancement method and system based on deep learning provided by the present invention are: the camera image enhancement method and system based on deep learning provided in the structure of the present invention can effectively remove noise and retain color for static background images, and the effect is no less than the traditional long exposure time ISP method. For images with moving targets, it can effectively remove the tailing effect of the target of interest and fully retain the details of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a structural flow chart of the deep neural network model of the present invention;

[0037] Figure 2 It is a structural diagram of the encoder;

[0038] Figure 3 Schematic diagram of the decoder structure;

[0039] Figure 4 This is a schematic diagram of the structure of the block;

[0040] Figure 5 This is a training flowchart for a deep neural network model in a dynamic scenario;

[0041] Figure 6 This is a training flowchart for a deep neural network model in a static scenario;

[0042] Figure 7 These are three snapshots of short video sequences in dynamic scenes.

[0043] Figure 8 for Figure 7 Schematic diagram of semantic segmentation results corresponding to the deep neural network model;

[0044] Figure 9 Schematic diagram of the superposition of the target cutouts from different frames;

[0045] Figure 10 Schematic diagrams for comparing static scenes, where (m1) is an image obtained by using the existing image processing process, and (m2) is an image obtained by the camera image enhancement method of this embodiment;

[0046] Figure 11 1 is a comparative schematic diagram of a dynamic scene, n1) is a diagram obtained by using the existing image processing process, and (n2) is a diagram obtained by the camera image enhancement method of this embodiment. DETAILED DESCRIPTION

[0047] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0048] like Figures 1 to 11 As shown, the present invention proposes a camera image enhancement method based on deep learning, comprising the following steps:

[0049] Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image. The deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network. The output of the backbone network is connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, respectively. The backbone network inputs the original image sequence;

[0050] The semantic segmentation result of the current frame image is output through the segmentation network to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images;

[0051] The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture;

[0052] The enhanced image corresponding to the current frame image is output through the image enhancement network.

[0053] During the execution of the deep neural network model, the enhanced image corresponding to the current frame is output. During the training process, the generated semantic segmentation results and image brightness are used to assist in adjusting the deep neural network model and the camera aperture.

[0054] This embodiment involves two interrelated aspects: 1) A deep learning network structure is proposed: a deep neural network model that can take the current frame and several historical frame images (input images) as input data; it can output multiple data, including: (a1) an enhanced image (enhanced image), (a2) a semantic segmentation result (segment image) used to assist the network in learning target segmentation, and (a3) data used to indicate the brightness of the image, that is, to assist aperture adjustment (brightness index). 2) A corresponding training data acquisition strategy is proposed, divided into two methods: static scenes and dynamic scenes, thereby achieving: (b1) for static background images, it effectively removes noise and preserves color, with an effect comparable to traditional long-exposure ISP methods; (b2) for images with moving targets, it effectively removes the tailing effect of the target of interest and fully preserves the image details.

[0055] A. Structure of Deep Neural Network Model

[0056] The deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network;

[0057] The backbone network includes a concatenation layer, an encoder group, and a decoder group connected in sequence. The encoder group includes encoders enc1, enc2, enc3, enc4, enc5, and enc6. The decoder group includes decoders dec1, dec2, dec3, dec4, and dec5. Enc1, enc2, enc3, enc4, enc5, enc6, dec5, dec4, dec3, dec2, and dec1 are connected in sequence. For each encoder, as Figure 2As shown, they all include a normalization layer composed of two-dimensional convolution, normalization, and activation function, as well as a split layer for splitting character strings, N blocks connected in sequence, the normalization layer, the split layer, and the block are connected in sequence and connected to the input of the concatenation layer concat, and the output of the split layer and the output of the block are separately connected to the input of the concatenation layer concat, so as to splice the features through the concatenation layer concat, and connect the spliced features to two normalization layers set in series, so as to output the encoded features through the normalization layer.

[0058] For each decoder, Figure 3 As shown in the figure, the main structure is similar to that of the encoder, but in the encoder, the concatenation layer is connected to two normalization layers set in series. In the decoder, the concatenation layer is connected to a normalization layer and an upsampling layer in sequence to output the decoded features.

[0059] In this embodiment, the segmentation network includes a segmentation channel normalization layer, upsampling, a segmentation splicing layer concat and a segmentation two-dimensional convolution Cov2d. The output of dec1 is connected to the segmentation splicing layer concat through the segmentation channel normalization layer. The outputs of dec2, dec3, dec4, and dec5 are all connected to the segmentation splicing layer concat in turn through the segmentation channel normalization layer and upsampling. The output of the segmentation splicing layer concat is connected to the input of the segmentation two-dimensional convolution Cov2d, and the segmentation two-dimensional convolution Cov2d outputs the semantic segmentation result.

[0060] In this embodiment, the brightness estimation network includes an auxiliary channel normalization layer and a conversion layer. The output of dec1 is connected to the auxiliary channel normalization layer and the conversion layer in sequence. The conversion layer outputs the brightness of the image and adjusts the aperture according to the brightness of the image.

[0061] The image enhancement network includes an enhanced channel normalization layer and an enhanced two-dimensional convolution Cov2d. The output of dec1 is connected to the enhanced channel normalization layer and the enhanced two-dimensional convolution Cov2d in sequence. The enhanced two-dimensional convolution Cov2d outputs an enhanced image corresponding to the image.

[0062] Among them, the segmentation channel normalization layer, auxiliary channel normalization layer and enhancement channel normalization layer are all two-dimensional convolution Cov2+normalization BN+activation function RELU.

[0063] B. Training of Deep Neural Network Models

[0064] A deep neural network model is used to infer the enhanced image. The model structure is as follows Figure 1 Shown:

[0065] a) The current frame and several previous frames are buffered and used as input data (input image) for the deep neural network model. Unlike the usual practice of using a single frame as network input, N frames are input (experiments show that N is optimal after weighing various factors) to allow the network to learn temporal information.

[0066] b) The output of the deep neural network model, the enhanced image, is the image enhanced by the model.

[0067] c) The output of the deep neural network model, the segment image, is the result of semantic segmentation of the current image. The main purpose of designing this output is to guide the model to learn to distinguish objects of interest in the image (primarily license plates, people, motor vehicles, non-motor vehicles, etc.), so as to better fuse image data of the same object from different frames.

[0068] d) The brightness index output by the deep neural network model is an overall estimate of the brightness of the current image. Generally, a value greater than 0.8 indicates that the image is too bright, and the upper-level logic should consider reducing the aperture. A value less than 0.2 indicates that the image is too dark, and the upper-level logic should consider increasing the aperture. If the value is between the two, the aperture can be left unchanged.

[0069] The training data acquisition process is divided into dynamic scene mode and static scene mode. Dynamic scene mode refers to the scene with moving objects of interest (mainly license plates, people, motor vehicles, non-motor vehicles, etc.). Static scene mode refers to the scene being basically still, without moving objects of interest.

[0070] The flowchart for obtaining training data in dynamic scenes is as follows Figure 5 , steps 101 to 107:

[0071] Step 101, obtaining an original image sequence;

[0072] Step 102: Match feature points of the current frame image with several previous historical frame images, and filter the matching results based on the segmentation results to remove false matches.

[0073] Step 103: Extract the segmented regions of the class of interest in the original image sequence and cluster them based on the feature matching results. The regions clustered in one class represent the segmentation results of an object in different frames.

[0074] Step 104: All segmentation results that are clustered together are matched to the segmentation results of the current frame image through perspective transformation, and then these results are weighted and fused to obtain the enhanced result of the target of interest in the current frame;

[0075] Step 105: Paste the enhancement result of the previous step back to the corresponding position of the current frame, and then manually filter it to obtain the enhanced image corresponding to the current frame image;

[0076] Step 106: The deep neural network model performs semantic segmentation on the current frame image. Sequences with poor segmentation results are discarded using a semantic segmentation algorithm-assisted annotation, manual-assisted annotation, and manual screening. The retained image is used as the semantic segmentation result segment image for the current frame image. The segmented categories should include categories of interest such as vehicles, license plates, non-motor vehicles, and human figures.

[0077] Step 107: Manually label the brightness index of the original image sequence. If the brightness index is greater than or equal to the set brightness threshold, the brightness index is marked as 1. If the brightness index is less than the set brightness threshold, the brightness index is marked as 0. The deep neural network model outputs the brightness index of the current frame image to serve as data to assist in camera aperture adjustment.

[0078] In the brightness index marking process, if the picture is too bright, it is marked as 1, and if the picture is too dark, it is marked as 0;

[0079] Because the segmentation, matching, and fusion of moving objects in steps 103 and 104 are relatively complex, the following diagrams (c1) to (c3) are provided to further illustrate:

[0080] c1) Figure 7 These are three snapshots of a short video sequence with dynamic targets, with the vehicle targets ranging from near to far.

[0081] c2) Figure 8 yes Figure 7 The corresponding semantic segmentation result image shows the vehicle target marked in red. The segmented area should be as accurate as possible. If the automatic segmentation is not effective, manual adjustment is required.

[0082] Sequences that are difficult to segment are discarded;

[0083] c3) Cut out the same target in different frames of the original image according to the segmented area, such as Figure 9 , through key point matching, the targets of different frames are appropriately transformed, matched to the size of the target in the current frame, and then weighted addition is performed to obtain the enhanced result of the target in the current frame, and then written back to the original image of the current frame.

[0084] Figure 11Figure 1 is a comparison diagram of a dynamic scene. (n1) is obtained using an existing image processing process, and (n2) is obtained using the camera image enhancement method of this embodiment. In the image (n1), the license plate is trailing and unrecognizable; in the image (n2), the license plate is clearly recognizable.

[0085] In this embodiment, the flowchart for obtaining training data in a static scene is as follows: Figure 6 , steps 201 to 206.

[0086] Step 201: Two cameras with the same hardware configuration are installed closely side by side; when collecting images, it is necessary to ensure that the images are basically completely consistent.

[0087] Step 202, setting one camera to a short exposure time and another camera to a long exposure time;

[0088] Step 203: Capture short still-image videos of various scenes using two fixed cameras;

[0089] Step 204: using the image sequence with short exposure time as input to the neural network model, and using the image sequence with long exposure time as an enhanced image output by the neural network model;

[0090] Step 205: Using a neural network model to assist in labeling and manual screening to perform semantic segmentation of the image, and obtaining a semantic segmentation result, a segment image;

[0091] Step 206 : manually label the brightness index of the image sequence with short exposure time and the image sequence with long exposure time as the corresponding brightness index, so as to serve as data for assisting the camera aperture adjustment.

[0092] Figure 10 : This is a schematic diagram of a static scene comparison. (m1) is obtained using the existing image processing process, and (m2) is obtained using the camera image enhancement method of this embodiment. By comparing the two images, the image in (m2) is purer and clearer.

[0093] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A camera image enhancement method based on deep learning, characterized in that: The steps include: Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image. The deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network. The output of the backbone network is connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, respectively. The backbone network inputs the original image sequence; The semantic segmentation result of the current frame image is output through the segmentation network to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images; The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture; Output the enhanced image corresponding to the current frame image through the image enhancement network; The backbone network includes a splicing layer, an encoder group, and a decoder group connected in sequence, the splicing layer inputs the original image sequence, and the encoder group input is connected to the output of the splicing layer; The encoder group includes encoders enc1, enc2, enc3, enc4, enc5, and enc6, and the decoder group includes decoders dec1, dec2, dec3, dec4, and dec5, and enc1, enc2, enc3, enc4, enc5, enc6, dec5, dec4, dec3, dec2, and dec1 are connected in sequence; The outputs of enc6, dec1, dec2, dec3, dec4, and dec5 are all connected to the input of the segmentation network; The output of dec1 is connected to the input of the brightness estimation network and the input of the image enhancement network respectively; The segmentation network includes a segmentation channel normalization layer, upsampling, a segmentation splicing layer concat and a segmentation two-dimensional convolution Cov2d. The output of dec1 is connected to the segmentation splicing layer concat through the segmentation channel normalization layer. The outputs of dec2, dec3, dec4 and dec5 are all connected to the segmentation splicing layer concat in turn through the segmentation channel normalization layer and upsampling. The output of the segmentation splicing layer concat is connected to the input of the segmentation two-dimensional convolution Cov2d, and the segmentation two-dimensional convolution Cov2d outputs the semantic segmentation result. The brightness estimation network includes an auxiliary channel normalization layer and a conversion layer. The output of dec1 is connected to the auxiliary channel normalization layer and the conversion layer in sequence. The conversion layer outputs the brightness of the image. The image enhancement network includes an enhanced channel normalization layer and an enhanced two-dimensional convolution Cov2d. The output of dec1 is connected to the enhanced channel normalization layer and the enhanced two-dimensional convolution Cov2d in sequence. The enhanced two-dimensional convolution Cov2d outputs an enhanced image corresponding to the image.

2. The camera image enhancement method based on deep learning according to claim 1, characterized in that During the training of a deep neural network model, if the original image sequence is a combination of images of different frames in a dynamic scene, the process is as follows: Perform feature matching between the current frame image and the historical frame image, and filter the matching results through the semantic segmentation results of the deep neural network model to remove false matches; Extract the segmentation area of the class of interest in the original image sequence, cluster it according to the feature matching results, and the features clustered in one class represent the segmentation results of the target in different frame images; All segmentation results clustered in one category are matched to the segmentation results of the current frame image through perspective transformation, and all these segmentation results are weighted fused to obtain the enhanced result of the target of interest in the current frame image; Paste the image enhancement result back to the corresponding position of the current frame image, and after manual screening, obtain the enhanced image corresponding to the current frame image; The deep neural network model performs semantic segmentation on the current frame image to obtain the semantic segmentation result segment image of the current frame image; The brightness index of the original image sequence is manually labeled. If the brightness index is greater than or equal to the set brightness threshold, the brightness index is labeled as 1. If the brightness index is less than the set brightness threshold, the brightness index is labeled as 0. The deep neural network model outputs the brightness index of the current frame image to serve as data to assist in camera aperture adjustment.

3. The camera image enhancement method based on deep learning according to claim 1, characterized in that During the training process of the deep neural network model, if the original image sequence is a combination of images of different frames in a static scene, the training process of the deep neural network model is as follows: Install two cameras with the same hardware configuration side by side, set one camera to a short exposure time and the other to a long exposure time; Two fixed cameras are used to capture short videos of still images of various scenes; The short exposure time image sequence is used as the input of the neural network model, and the long exposure time image sequence is used as the enhanced image output by the neural network model; The short exposure time image sequence is subjected to image semantic segmentation by the neural network model, and the semantic segmentation result segment image corresponding to the short exposure time image sequence is output; The brightness index of the image sequence with short exposure time and the image sequence with long exposure time are manually labeled as the brightness index of the corresponding picture, which is used as data to assist the camera aperture adjustment.

Images