Camera image enhancement method and system based on deep learning

Abstract

The invention discloses a camera image enhancement method and system based on deep learning, and relates to the technical field of image processing, and the method comprises the following steps: inputting an obtained original image sequence into a deep neural network model, and outputting a semantic segmentation result of a current frame image through a segmentation network, the target fusion module is used for guiding a deep neural network model to learn an interested target in a current frame image so as to fuse image data of the same target in different frame images; outputting the picture brightness degree of the current frame image through a brightness estimation network so as to adjust a camera aperture; outputting an enhanced image corresponding to the current frame image through an image enhancement network; according to the camera image enhancement method and system, picture details are fully reserved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a camera image enhancement method and system based on deep learning. Background Art

[0002] After the camera with CMOS Sensor as the acquisition device collects the raw data, it generally goes through a series of ISP processing processes in order to obtain images with better visual effects. The traditional ISP algorithm can already process various images of scenes with good lighting conditions very well. However, the processing effect is still not satisfactory in scenes with low illumination and moving targets in the picture. The main difficulty is that the traditional ISP method does not distinguish the targets in the picture, so both the exposure time selection and the data fusion of multiple frames are not targeted. If the exposure time is set to be longer or multiple frames are directly weighted fused, better results will be achieved for the static part of the picture, but it will cause the tailing effect of the moving target; if the exposure time is set to be shorter, although the details of the moving target can be retained, it will cause obvious noise in the overall picture. At present, there are quite a lot of works that use AI to improve the visual effects of images, but most of them are suitable for processing static images. If these methods are directly applied to video sequences, it is difficult to achieve ideal visual effects if the timing information before and after is not used due to the existence of moving targets.

[0003] Since deep learning has been successfully applied in many fields, the idea of ​​deep learning to solve problems has been standardized, that is, to give a network structure, prepare training data, and train until convergence. However, when it comes to specific problems, how to design the network structure, especially the input and output data format; how to guide the network to learn valuable information; how to obtain more effective training data; these are issues that researchers of specific technical solutions have to spend a lot of manpower and material resources to study and solve. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a camera image enhancement method and system based on deep learning, which fully retains the details of the picture.

[0005] The present invention proposes a camera image enhancement method based on deep learning, comprising the following steps:

[0006] Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image, and the deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network, wherein the output of the backbone network is respectively connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, and the backbone network inputs the original image sequence;

[0007] Output the semantic segmentation result of the current frame image through the segmentation network, which is used to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images;

[0008] The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture;

[0009] The enhanced image corresponding to the current frame image is output through the image enhancement network.

[0010] Furthermore, in the training process of the deep neural network model, if the original image sequence is a combination of images of different frames in a dynamic scene, the process is as follows:

[0011] Perform feature matching between the current frame image and the historical frame image, and filter the matching results through the semantic segmentation results of the deep neural network model to remove false matches;

[0012] Cut out the segmentation area of ​​the category of interest in the original image sequence, and cluster it according to the feature matching results. The features clustered in one category represent the segmentation results of the target in different frame images;

[0013] All the segmentation results that are clustered in one category are matched to the segmentation results of the current frame image through perspective transformation, and all the segmentation results are weighted and fused to obtain the enhanced result of the target of interest in the current frame image;

[0014] Paste the image enhancement result back to the corresponding position of the current frame image, and obtain the enhanced image corresponding to the current frame image after manual screening;

[0015] The deep neural network model performs semantic segmentation on the current frame image to obtain a semantic segmentation result segment image of the current frame image;

[0016] The brightness index of the original image sequence is manually annotated. If the brightness index is greater than or equal to the set brightness threshold, the brightness index is marked as 1. If the brightness index is less than the set brightness threshold, the brightness index is marked as 0. The deep neural network model outputs the brightness index of the current frame image as data to assist the camera aperture adjustment.

[0017] Furthermore, in the training process of the deep neural network model, if the original image sequence is a combination of images of different frames in a static scene, the training process of the deep neural network model is as follows:

[0018] Install two cameras with the same hardware configuration closely side by side, set one camera to a short exposure time and the other to a long exposure time;

[0019] Two fixed cameras are used to capture short videos of still images of various scenes;

[0020] The image sequence with short exposure time is used as the input of the neural network model, and the image sequence with long exposure time is used as the enhanced image output by the neural network model;

[0021] The short exposure time image sequence is subjected to image semantic segmentation by a neural network model, and the semantic segmentation result segment image corresponding to the short exposure time image sequence is output;

[0022] The brightness index of the image sequence with short exposure time and the image sequence with long exposure time are manually annotated as the brightness index of the corresponding picture, which can be used as data to assist the camera aperture adjustment.

[0023] Furthermore, the backbone network includes a splicing layer, an encoder group and a decoder group connected in sequence, the output of the decoder group is respectively connected to the input of the segmentation network, the input of the brightness estimation network and the input of the image enhancement network, and the splicing layer inputs the original image sequence.

[0024] Further, the encoder group includes encoders enc1, enc2, enc3, enc4, enc5, and enc6, and the decoder group includes decoders dec1, dec2, dec3, dec4, and dec5, and enc1, enc2, enc3, enc4, enc5, enc6, dec5, dec4, dec3, dec2, and dec1 are connected in sequence;

[0025] The outputs of enc6, dec1, dec2, dec3, dec4, and dec5 are all connected to the input of the segmentation network;

[0026] The output of dec1 is connected to the input of the brightness estimation network and the input of the image enhancement network respectively.

[0027] Furthermore, the segmentation network includes a segmentation channel normalization layer, upsampling, a segmentation splicing layer concat and a segmentation two-dimensional convolution Cov2d. The output of dec1 is connected to the segmentation splicing layer concat through the segmentation channel normalization layer. The outputs of dec2, dec3, dec4 and dec5 are connected to the segmentation splicing layer concat in turn through the segmentation channel normalization layer and upsampling. The output of the segmentation splicing layer concat is connected to the input of the segmentation two-dimensional convolution Cov2d, and the segmentation two-dimensional convolution Cov2d outputs the semantic segmentation result.

[0028] Furthermore, the brightness estimation network includes an auxiliary channel normalization layer and a conversion layer, the output of dec1 is connected to the auxiliary channel normalization layer and the conversion layer in sequence, and the conversion layer outputs the brightness of the image.

[0029] Furthermore, the image enhancement network includes an enhanced channel normalization layer and an enhanced two-dimensional convolution Cov2d, the output of dec1 is sequentially connected to the enhanced channel normalization layer and the enhanced two-dimensional convolution Cov2d, and the enhanced two-dimensional convolution Cov2d outputs an enhanced image corresponding to the image.

[0030] A camera image enhancement system based on deep learning enhances the input image based on a deep neural network model. The specific enhancement process is as follows:

[0031] Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image, and the deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network, wherein the output of the backbone network is respectively connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, and the backbone network inputs the original image sequence;

[0032] Output the semantic segmentation result of the current frame image through the segmentation network, which is used to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images;

[0033] The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture;

[0034] The enhanced image corresponding to the current frame image is output through the image enhancement network.

[0035] The advantages of the camera image enhancement method and system based on deep learning provided by the present invention are: the camera image enhancement method and system based on deep learning provided in the structure of the present invention can better remove noise and retain color for static background images, and the effect is no less than that of the traditional long exposure time ISP method. For images with moving targets, the tailing effect of the target of interest is better removed, and the details of the image are fully retained. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a structural flow chart of the deep neural network model of the present invention;

[0037] Figure 2 It is a structural diagram of the encoder;

[0038] Figure 3 It is a schematic diagram of the structure of the decoder;

[0039] Figure 4 It is a schematic diagram of the structure of the block Block;

[0040] Figure 5 This is a training flowchart of a deep neural network model in a dynamic scenario;

[0041] Figure 6 This is a training flowchart of a deep neural network model in a static scene;

[0042] Figure 7 These are three snapshots of short video sequences in dynamic scenes.

[0043] Figure 8 for Figure 7 Schematic diagram of semantic segmentation results corresponding to the deep neural network model;

[0044] Fig. 9 This is a schematic diagram of the superposition of the target cutouts from different frames;

[0045] Fig.10 Schematic diagrams for comparing static scenes, where (m1) is a diagram obtained by using an existing image processing process, and (m2) is a diagram obtained by using the camera image enhancement method of this embodiment;

[0046] Fig.11 1 is a schematic diagram of a comparison of dynamic scenes, n1) is a diagram obtained by using the existing image processing process, and (n2) is a diagram obtained by the camera image enhancement method of this embodiment. DETAILED DESCRIPTION

[0047] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0048] like Figures 1 to 11 As shown, a camera image enhancement method based on deep learning proposed in the present invention comprises the following steps:

[0049] Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image, and the deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network, wherein the output of the backbone network is respectively connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, and the backbone network inputs the original image sequence;

[0050] Output the semantic segmentation result of the current frame image through the segmentation network, which is used to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images;

[0051] The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture;

[0052] The enhanced image corresponding to the current frame image is output through the image enhancement network.

[0053] During the execution of the deep neural network model, the enhanced image corresponding to the current frame image is output. During the training process, the generated semantic segmentation results and picture brightness are used to assist in adjusting the deep neural network model and the aperture adjustment of the camera.

[0054] In this embodiment, two interrelated aspects are involved: 1) A deep learning network structure is proposed: a deep neural network model can take the current frame and several historical frame images (input image) as input data; it can output multiple data, including: (a1) an enhanced image (enhanced image), (a2) a semantic segmentation result (segment image) used to assist the network in learning target segmentation, (a3) ​​data used to indicate the brightness of the picture, that is, data for auxiliary aperture adjustment (brightness index). 2) A corresponding training data acquisition strategy is proposed, which is divided into two modes: static scene and dynamic scene, so as to achieve: (b1) for static background pictures, the noise is better removed and the color is retained, and the effect is no less than the traditional long exposure time ISP method, (b2) for pictures with moving targets, the tailing effect of the target of interest is better removed, and the details of the picture are fully retained.

[0055] A. Structure of Deep Neural Network Model

[0056] The deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network;

[0057] The backbone network includes a concatenation layer, an encoder group and a decoder group connected in sequence. The encoder group includes encoders enc1, enc2, enc3, enc4, enc5, and enc6. The decoder group includes decoders dec1, dec2, dec3, dec4, and dec5. enc1, enc2, enc3, enc4, enc5, enc6, dec5, dec4, dec3, dec2, and dec1 are connected in sequence. For each encoder, such as Figure 2As shown, they all include a normalization layer composed of a two-dimensional convolution, normalization, and an activation function, as well as a split layer for splitting a string, N blocks connected in sequence, the normalization layer, the split layer, and the block Block are connected in sequence and then connected to the input of the concatenation layer concat, and the output of the split layer and the output of the block Block are separately connected to the input of the concatenation layer concat, so as to concatenate the features through the concatenation layer concat, and connect the concatenated features to two normalization layers that are set in series, so as to output the encoded features through the normalization layer.

[0058] For each decoder, Figure 3 As shown in the figure, the main structure is similar to that of the encoder, but in the encoder, the concatenation layer is connected to two normalization layers set in series after the concat, and in the decoder, the concatenation layer is connected to a normalization layer and an upsampling layer in sequence to output the decoded features.

[0059] In this embodiment, the segmentation network includes a segmentation channel normalization layer, upsampling, a segmentation splicing layer concat and a segmentation two-dimensional convolution Cov2d. The output of dec1 is connected to the segmentation splicing layer concat through the segmentation channel normalization layer. The outputs of dec2, dec3, dec4, and dec5 are connected to the segmentation splicing layer concat in turn through the segmentation channel normalization layer and upsampling. The output of the segmentation splicing layer concat is connected to the input of the segmentation two-dimensional convolution Cov2d, and the segmentation two-dimensional convolution Cov2d outputs the semantic segmentation result.

[0060] In this embodiment, the brightness estimation network includes an auxiliary channel normalization layer and a conversion layer. The output of dec1 is connected to the auxiliary channel normalization layer and the conversion layer in sequence. The conversion layer outputs the brightness of the image and adjusts the aperture according to the brightness of the image.

[0061] The image enhancement network includes an enhanced channel normalization layer and an enhanced two-dimensional convolution Cov2d. The output of dec1 is connected to the enhanced channel normalization layer and the enhanced two-dimensional convolution Cov2d in sequence. The enhanced two-dimensional convolution Cov2d outputs an enhanced image corresponding to the image.

[0062] Among them, the segmentation channel normalization layer, auxiliary channel normalization layer and enhancement channel normalization layer are all two-dimensional convolution Cov2+normalization BN+activation function RELU.

[0063] B. Training of Deep Neural Network Model

[0064] A deep neural network model is used to infer the enhanced image. The model structure is as follows Figure 1 Shown:

[0065] a) The current frame and several previous frames are buffered and used as input data (input image) of the deep neural network model. Unlike the usual one-frame image as the network model input, N frames are used as input (experiments show that N is 8 to 16, which is relatively optimal after weighing various factors) for the network to learn time series information.

[0066] b) The output enhanced image of the deep neural network model is the image enhanced by the model.

[0067] c) The output segment image of the deep neural network model is the result of semantic segmentation of the current image. The main purpose of designing this output is to guide the model to learn the ability to distinguish objects of interest in the image (mainly license plates, people, motor vehicles, non-motor vehicles, etc.), so as to better fuse image data of the same object in different frames.

[0068] d) The output brightness index of the deep neural network model is an overall estimate of the brightness of the current image. Generally, if it is greater than 0.8, the image is considered too bright, and the upper-level logic should consider reducing the aperture; generally, if it is less than 0.2, the image is considered too dark, and the upper-level logic should consider increasing the aperture; if it is between the two, it is possible to consider not adjusting the aperture.

[0069] The training data acquisition process is divided into dynamic scene mode and static scene mode. The dynamic scene mode refers to the moving targets of interest in the picture (mainly license plates, people, motor vehicles, non-motor vehicles, etc.). The static scene mode refers to the picture being basically still, without moving targets of interest.

[0070] The flowchart for obtaining training data in dynamic scenes is as follows Figure 5 , step 101 to step 107:

[0071] Step 101, obtaining an original image sequence;

[0072] Step 102, matching feature points of the current frame image with several previous historical frame images, and filtering the matching results through the segmentation results to remove false matches;

[0073] Step 103, extract the segmented area of ​​the category of interest in the original image sequence, and cluster them according to the feature matching results. The segments clustered in one category represent the segmentation results of a target in different frame images;

[0074] Step 104, all the segmentation results that are grouped together are matched to the segmentation results of the current frame image through perspective transformation, and then these results are weighted and fused to obtain the enhancement result of the target of interest in the current frame;

[0075] Step 105, pasting the enhancement result of the previous step back to the corresponding position of the current frame, and then manually screening to obtain an enhanced image corresponding to the current frame image;

[0076] Step 106, the deep neural network model performs semantic segmentation on the current frame image, and discards the sequences with poor segmentation effect by using semantic segmentation algorithm-assisted annotation, manual-assisted annotation, and manual screening. The retained image is used as the semantic segmentation result segment image of the current frame image. The segmented categories should include categories of interest such as vehicles, license plates, non-motor vehicles, and human figures;

[0077] Step 107, manually labeling the brightness index of the original image sequence. If the brightness index is greater than or equal to the set brightness threshold, the brightness index is labeled as 1. If the brightness index is less than the set brightness threshold, the brightness index is labeled as 0. The deep neural network model outputs the brightness index of the current frame image as data to assist the camera aperture adjustment.

[0078] In the process of brightness index marking, if the picture is too bright, it is marked as 1, and if the picture is too dark, it is marked as 0;

[0079] Steps 103 and 104 regarding the segmentation, matching, and fusion of moving objects are relatively complex, so we will further illustrate (c1) to (c3) with diagrams:

[0080] c1) Figure 7 These are three snapshots of a short video sequence with dynamic targets, with vehicle targets from near to far;

[0081] c2) Figure 8 yes Figure 7 The corresponding semantic segmentation result image shows that the vehicle target is marked in red. The segmented area should be as accurate as possible. If the automatic segmentation effect of the program is not good, manual adjustment is required.

[0082] Sequences that are difficult to segment are discarded;

[0083] c3) Cut out the same target in different frames according to the segmented area, such as Fig. 9 , through key point matching, the targets of different frames are appropriately transformed to match the size of the target in the current frame, and then weighted addition is performed to obtain the enhanced result of the target in the current frame, and then it is written back to the original image of the current frame.

[0084] Fig.111 is a schematic diagram of a dynamic scene comparison. (n1) is a diagram obtained by using the existing image processing process, and (n2) is a diagram obtained by using the camera image enhancement method of this embodiment. In the image of (n1), the license plate is trailing and cannot be recognized; in the image of (n2), the license plate is clearly identifiable.

[0085] In this embodiment, the flowchart of obtaining training data in a static scene is as follows: Figure 6 , step 201 to step 206.

[0086] Step 201, two cameras with the same hardware configuration are installed closely side by side; when collecting, it is necessary to ensure that the images are basically completely consistent.

[0087] Step 202, setting one camera to a short exposure time and another camera to a long exposure time;

[0088] Step 203, collecting still picture short videos of various scenes through two fixed cameras;

[0089] Step 204, using the image sequence with short exposure time as input of the neural network model, and using the image sequence with long exposure time as enhanced image output by the neural network model;

[0090] Step 205, performing semantic segmentation of the image by using a neural network model to assist annotation and manual screening, and obtaining a semantic segmentation result, a segment image;

[0091] Step 206 , manually marking the brightness index of the image sequence with short exposure time and the image sequence with long exposure time as the corresponding brightness index of the picture, so as to serve as data for assisting the aperture adjustment of the camera.

[0092] Fig.10 1 is a schematic diagram of a static scene comparison, (m1) is a picture obtained by using the existing image processing process, and (m2) is a picture obtained by the camera image enhancement method of this embodiment. By comparing the two pictures, the picture of (m2) is purer and clearer.

[0093] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A camera image enhancement method based on deep learning, characterized in that: The steps include: Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image, and the deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network, wherein the output of the backbone network is respectively connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, and the backbone network inputs the original image sequence; Output the semantic segmentation result of the current frame image through the segmentation network, which is used to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images; The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture; The enhanced image corresponding to the current frame image is output through the image enhancement network.

2. The camera image enhancement method based on deep learning according to claim 1, characterized in that: In the training process of the deep neural network model, if the original image sequence is a combination of images of different frames in a dynamic scene, the process is as follows: Perform feature matching between the current frame image and the historical frame image, and filter the matching results through the semantic segmentation results of the deep neural network model to remove false matches; Cut out the segmentation area of ​​the category of interest in the original image sequence, and cluster it according to the feature matching results. The features clustered in one category represent the segmentation results of the target in different frame images; All the segmentation results that are clustered in one category are matched to the segmentation results of the current frame image through perspective transformation, and all the segmentation results are weighted and fused to obtain the enhanced result of the target of interest in the current frame image; Paste the image enhancement result back to the corresponding position of the current frame image, and obtain the enhanced image enhancedimage corresponding to the current frame image after manual screening; The deep neural network model performs semantic segmentation on the current frame image to obtain the semantic segmentation result segmentimage of the current frame image; The brightness index of the original image sequence is manually annotated. If the brightness index is greater than or equal to the set brightness threshold, the brightness index is marked as 1. If the brightness index is less than the set brightness threshold, the brightness index is marked as 0. The deep neural network model outputs the brightness index of the current frame image as data to assist the camera aperture adjustment.

3. The camera image enhancement method based on deep learning according to claim 1, characterized in that: In the training process of the deep neural network model, if the original image sequence is a combination of images of different frames in a static scene, the training process of the deep neural network model is as follows: Install two cameras with the same hardware configuration closely side by side, set one camera to a short exposure time and the other to a long exposure time; Two fixed cameras are used to capture short videos of still images of various scenes; The image sequence with short exposure time is used as the input of the neural network model, and the image sequence with long exposure time is used as the enhanced image output by the neural network model; The short exposure time image sequence is subjected to image semantic segmentation by a neural network model, and the semantic segmentation result segment image corresponding to the short exposure time image sequence is output; The brightness index of the image sequence with short exposure time and the image sequence with long exposure time are manually annotated as the brightness index of the corresponding picture, which can be used as data to assist the camera aperture adjustment.

4. The camera image enhancement method based on deep learning according to claim 1, characterized in that: The backbone network includes a splicing layer, an encoder group and a decoder group connected in sequence, the output of the decoder group is respectively connected to the input of the segmentation network, the input of the brightness estimation network and the input of the image enhancement network, and the splicing layer inputs the original image sequence.

5. The camera image enhancement method based on deep learning according to claim 4, characterized in that: The encoder group includes encoders enc1, enc2, enc3, enc4, enc5, and enc6, and the decoder group includes decoders dec1, dec2, dec3, dec4, and dec5, and enc1, enc2, enc3, enc4, enc5, enc6, dec5, dec4, dec3, dec2, and dec1 are connected in sequence; The outputs of enc6, dec1, dec2, dec3, dec4, and dec5 are all connected to the input of the segmentation network; The output of dec1 is connected to the input of the brightness estimation network and the input of the image enhancement network respectively.

6. The camera image enhancement method based on deep learning according to claim 5, characterized in that: The segmentation network includes a segmentation channel normalization layer, upsampling, a segmentation splicing layer concat and a segmentation two-dimensional convolution Cov2d. The output of dec1 is connected to the segmentation splicing layer concat through the segmentation channel normalization layer. The outputs of dec2, dec3, dec4 and dec5 are connected to the segmentation splicing layer concat in turn through the segmentation channel normalization layer and upsampling. The output of the segmentation splicing layer concat is connected to the input of the segmentation two-dimensional convolution Cov2d, and the segmentation two-dimensional convolution Cov2d outputs the semantic segmentation result.

7. The camera image enhancement method based on deep learning according to claim 5, characterized in that: The brightness estimation network includes an auxiliary channel normalization layer and a conversion layer. The output of dec1 is connected to the auxiliary channel normalization layer and the conversion layer in sequence. The conversion layer outputs the brightness of the image.

8. The camera image enhancement method based on deep learning according to claim 5, characterized in that: The image enhancement network includes an enhanced channel normalization layer and an enhanced two-dimensional convolution Cov2d. The output of dec1 is connected to the enhanced channel normalization layer and the enhanced two-dimensional convolution Cov2d in sequence. The enhanced two-dimensional convolution Cov2d outputs an enhanced image corresponding to the image.

9. A camera image enhancement system based on deep learning, characterized in that: The input image is enhanced based on the deep neural network model. The specific enhancement process is as follows: Inputting the acquired original image sequence into a deep neural network model, wherein the original image sequence includes a current frame image and a historical frame image, and the deep neural network model includes a backbone network, a segmentation network, a brightness estimation network, and an image enhancement network, wherein the output of the backbone network is respectively connected to the input of the segmentation network, the input of the brightness estimation network, and the input of the image enhancement network, and the backbone network inputs the original image sequence; Output the semantic segmentation result of the current frame image through the segmentation network, which is used to guide the deep neural network model to learn the target of interest in the current frame image, so as to fuse the image data of the same target in different frame images; The brightness estimation network outputs the brightness of the current frame image to adjust the camera aperture; The enhanced image corresponding to the current frame image is output through the image enhancement network.

Images