Low-illumination vehicle-mounted image enhancement method based on multispectral fusion and deep learning

Abstract

The invention relates to the technical field of image processing, and discloses a low-illumination vehicle-mounted image enhancement method based on multispectral fusion and deep learning. Comprising the steps that a low-illumination image is divided into subsets, an image to be enhanced is obtained through matching, spectral features of the image to be enhanced are extracted, a fused image is generated through fusion, the fused image is divided into sub-images, enhanced sub-images are generated through a deep learning enhancement model, and the enhanced sub-images are spliced into an enhanced low-illumination image. Compared with the prior art, the method has the advantages that the workload of enhancement processing of each sub-image can be reduced, the detail fineness of the sub-images during enhancement processing can be improved, and the problem of low detail enhancement precision easily occurring during integral enhancement of the fused image is avoided; according to the invention, an accurate judgment basis is provided for the vehicle-mounted image recognition system to recognize the low-illumination vehicle-mounted image, the high-precision enhancement effect of the low-illumination vehicle-mounted image is realized, and the working performance of the vehicle-mounted image recognition system is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a low-illumination vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning. Background Art

[0002] With the rapid development of autonomous driving technology, on-board image recognition is becoming increasingly important. Especially at night or in low-light scenes, low-light images taken will have a significant negative impact on the performance of the on-board image recognition system. In order to improve the recognition accuracy of low-light images, it is necessary to enhance the low-light images. Multispectral technology and deep learning technology are effectively used to improve the quality of low-light on-board images due to their strong image fusion and enhancement characteristics.

[0003] The patent application with reference publication number CN117808721A discloses a method, device, equipment and medium for low-light image enhancement based on deep learning, including obtaining an image to be enhanced, extracting frequency domain features and color features of the image to be enhanced, cross-fusing the frequency domain features and the color features to obtain fusion features, performing attention calculation on the image to be enhanced according to the fusion features to obtain a wide dynamic enhanced image, performing cross-image restoration on the wide dynamic enhanced image to obtain a low-light enhanced image of the image to be enhanced; When enhancing existing low-light vehicle images, the enhanced effect of the low-light image is achieved by taking the fused image as a whole for enhancement processing. For example, in the above-mentioned patent application, the frequency domain features and color features are cross-fused, and the image to be enhanced is enhanced as a whole according to the fused features, and finally the enhanced effect of the image to be enhanced is achieved. Since the types and quantities of information contained in the low-light vehicle images are numerous and complex, when different types of low-light images are fused into an overall image to be enhanced, and a large amount of information in the image to be enhanced is enhanced as a whole, on the one hand, the workload of the enhancement processing will be increased, and on the other hand, some detail information in the image to be enhanced will not be accurately enhanced, thereby reducing the clarity and accuracy of the details of the enhanced low-light vehicle images, and weakening the enhancement effect of the low-light vehicle images.

[0004] In view of this, the present invention proposes a low-illumination vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning, applied to an image processor, comprising: S1: Divide the captured low-light image into subsets, and match the image to be enhanced from the subset based on the enhancement matching criterion. The enhancement matching criterion is: match the visible light image, infrared image and short-wave infrared image corresponding to the maximum value of the available coefficient; S2: Extract the spectral features of the image to be enhanced, including edge features, color features, thermal radiation features and reflectivity features, and fuse the spectral features based on the deep learning fusion model to generate a fused image; S3: Based on the image segmentation criterion, the fused image is segmented into sub-images, and the enhancement coefficients of the sub-images are calculated one by one. The image segmentation criterion is that there is no overlapping area between adjacent sub-images; S4: Taking the enhancement coefficient as the enhancement order, the sub-images are enhanced in sequence through the deep learning enhancement model to generate enhanced sub-images; S5: stitching the enhanced sub-images into an enhanced low-light image, and annotating the enhanced interpretation in the enhanced low-light image, where the enhanced interpretation includes a high-attention area and a normal-attention area.

[0006] Further, the subset includes a first subset, a second subset, and a third subset; The method of dividing the subsets is: The attribute information of all low-light images taken by the vehicle-mounted image acquisition system is queried one by one, and the low-light images with the attribute information of "KJ", "HW" and "DB" are recorded as visible light images, infrared images and short-wave infrared images respectively; respectively querying the shooting time of the visible light image, infrared image and short-wave infrared image, and eliminating the visible light images, infrared images or short-wave infrared images that do not have the same shooting time; According to the sequence of shooting time, the remaining visible light images, infrared images and short-wave infrared images are numbered in ascending order to generate a first subset, a second subset and a third subset.

[0007] Furthermore, the matching method of the image to be enhanced is: Combine the visible light images, infrared images and short-wave infrared images with the same numbers in the first subset, the second subset and the third subset to generate A image units; The resolution of the visible light image and the contrast of the infrared image in A image units are queried one by one through the image management system to obtain A resolution values ​​and A contrast values; Mark the pixel values ​​of all pixels on the short-wave infrared image in A image units one by one, and add up and average the pixel values ​​of the pixels to obtain A brightness values; The resolution values, contrast values ​​and brightness values ​​of A image units are respectively assigned corresponding weight factors and added together to obtain A available coefficients; The image unit corresponding to the maximum value of the available coefficient is recorded as a matching unit, and the visible light image, infrared image and short-wave infrared image in the matching unit are recorded as images to be enhanced.

[0008] Furthermore, the edge feature extraction method is: Convert the visible light image in the image to be enhanced into a grayscale image, and read the grayscale image; Configure the Sobel convolution kernels in the horizontal and vertical directions, which are denoted as SP convolution kernel and SZ convolution kernel respectively; Use the SP convolution kernel and the SZ convolution kernel to perform convolution operations on the image to generate a horizontal gradient image and a vertical gradient image; Mark the gradient values ​​of B pixels in the grayscale image in the horizontal and vertical directions respectively, and calculate the gradient amplitude of the B pixels by the gradient amplitude formula; The expression of the gradient amplitude is: ; In the formula, For the The gradient amplitude of each pixel, =1,2,...,B, For the The gradient value of a pixel in the horizontal direction, For the The gradient value of each pixel in the vertical direction; The pixel points whose gradient amplitude is greater than the calibrated gradient threshold are recorded as target pixels, and all the target pixels are connected in sequence in the grayscale image to obtain edge features.

[0009] Furthermore, the method for extracting thermal radiation features is: The lower left corner of the infrared image is used as the coordinate origin, and the left boundary and bottom boundary of the infrared image are used as the vertical axis and horizontal axis respectively to construct the two-dimensional image coordinates; Mark the pixel values ​​of all pixels in the infrared image one by one, and convert the pixel values ​​into corresponding temperature values; Mark the horizontal and vertical coordinates of all pixel points through the two-dimensional image coordinates, and combine the horizontal and vertical coordinates into coordinate information; After matching the temperature values ​​with the corresponding coordinate information one by one, the thermal radiation characteristics are generated.

[0010] Furthermore, the sub-image segmentation method is: Convert the fused image into a grayscale fused image, and mark the pixel points corresponding to the D edge features one by one on the grayscale fused image; In a clockwise manner, pixel points corresponding to the D edge features are connected in sequence to form a closed area, and D first areas are obtained; Using a preset first length as a segmentation width and a preset second length as a segmentation height, a vertical segmentation line and a horizontal segmentation line are drawn on the grayscale fusion image respectively, and based on the vertical segmentation line and the horizontal segmentation line, the grayscale fusion image is divided into E images to be adjusted; The image to be adjusted including the first region is recorded as a target image, the first region in the target image is eliminated, and the remaining part of the target image, the remaining image to be adjusted and the D first regions are collectively referred to as sub-images to obtain E sub-images.

[0011] Furthermore, the calculation method of the enhancement coefficient is: Count the number of pixels in the E sub-images one by one to obtain E pixel values; After accumulating the pixel values ​​of all pixels in the E sub-images one by one, E total pixel values ​​are obtained, and the E total pixel values ​​are compared with the E pixel values ​​and averaged to obtain E enhancement coefficients; The expression of the enhancement coefficient is: ; In the formula, For the The enhancement factor of the sub-image is =1,2,...,E, For the The total pixel value of sub-images, For the The pixel value of each sub-image.

[0012] Furthermore, the method for generating the enhanced sub-image is as follows: According to the enhancement coefficient from large to small, the E sub-images are numbered in ascending order, with 1 being the first number; When sub-images have enhancement coefficients of the same size, the sub-images with the same enhancement coefficients are recorded as images to be sorted; Compare the total pixel values ​​of the images to be sorted one by one, and arrange all the images to be sorted in order from large to small total pixel values; According to the order of numbering from small to large, the E sub-images are input into the deep learning enhancement model one by one, E enhanced sub-images are output, and the numbers of the E sub-images are assigned to the E enhanced sub-images one by one.

[0013] Furthermore, the stitching method for enhancing low-light images is: Construct an image stitching frame identical to the grayscale fused image, segment the image stitching frame into stitching units identical to the E sub-images, and convert the numbers of the E sub-images to the E stitching units accordingly; Fill the E enhanced sub-images into the E stitching units in order according to the number from small to large, and obtain the basic stitching frame; In the basic stitching frame, the stitching boundaries between two adjacent stitching units are drawn one by one, and all the pixels on the stitching boundaries are marked to obtain F stitching pixels; The stitching units located on both sides of the F stitching pixels are recorded as adjacent units to obtain F groups of adjacent units, and the distance values ​​between the pixels in the F groups of adjacent units and the corresponding F stitching pixels are measured one by one, and all the distance values ​​of the F stitching pixels are summarized to obtain F distance value sets; Record two pixel points corresponding to the minimum values ​​in the F distance value sets as the first conversion point and the second conversion point respectively, and obtain F first conversion points and F second conversion points; The pixel values ​​of the F first conversion points and the corresponding F second conversion points are added and averaged in sequence to obtain F conversion values, and the F conversion values ​​are correspondingly assigned to the F splicing pixel points to obtain an enhanced low-light image.

[0014] Further, the annotation method to enhance the interpretation is: In the enhanced low-light image, the regions identical to the D first regions are recorded as height regions, and the remaining regions are recorded as regular regions; A remark box with two attention points is established on the D height areas and the conventional area respectively, and the two attention points are recorded as the first point and the second point respectively; The first and second points in the remark boxes of D height areas are both marked with "1", and D height attention areas are obtained; Add "1" to the first point in the remark box of the general area to obtain the general attention area.

[0015] The technical effects and advantages of the low-light vehicle-mounted image enhancement method based on multispectral fusion and deep learning of the present invention are as follows: The present invention divides the captured low-light images into subsets, and matches the images to be enhanced from the subsets based on the enhancement matching criterion. It can match the images that best meet the image fusion requirements from different types of low-light images, and effectively avoid the negative impact brought about by the participation of a large number of low-quality low-light images in image fusion. By extracting the spectral features of the images to be enhanced and fusing the spectral features based on the deep learning fusion model to generate a fused image, the spectral features of different types and dimensions can be effectively and accurately fused together, achieving the effect of overall fusion representation of the multi-spectral image. Based on the image segmentation criterion, the fused image is segmented into sub-images, and the enhancement coefficients of the sub-images are calculated one by one. The enhancement order is taken as the enhancement coefficient, and the fusion image is enhanced through deep learning. The model performs image enhancement on sub-images in turn to generate enhanced sub-images, which can not only divide the integrated image containing more information into multiple sub-images and reduce the workload of enhancing each sub-image, but also improve the detail fineness of the sub-image during the enhancement process, avoiding the problem of low detail enhancement accuracy that is easy to occur when the fused image is enhanced as a whole. At the same time, by splicing the enhanced sub-images into enhanced low-illuminance images and annotating the enhanced interpretation in the enhanced low-illuminance images, the enhanced low-illuminance images can be accurately annotated and classified, providing an accurate judgment basis for the vehicle-mounted image recognition system to recognize low-illuminance vehicle-mounted images, thereby achieving high-precision enhancement effects of low-illuminance vehicle-mounted images and improving the working performance of the vehicle-mounted image recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a process for a low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning provided in the first embodiment of the present invention; Figure 2 A schematic diagram of a low-light vehicle-mounted image enhancement system based on multi-spectral fusion and deep learning provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1: Please refer to Figure 1 As shown, the low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning described in this embodiment is applied to an image processor, including: S1: Divide the captured low-light image into subsets, and match the image to be enhanced from the subsets based on the enhancement matching criterion; Low-light images refer to the original images taken by the vehicle-mounted image acquisition system under poor lighting conditions without being optimized. At this time, low-light images usually show negative conditions such as low brightness, poor contrast, and high noise. Low-light images include visible light images, infrared images, and short-wave infrared images; visible light images refer to images captured by visible light equipment, infrared images refer to images captured by infrared equipment, and short-wave infrared images refer to images captured by short-wave infrared equipment; When visible light equipment, infrared equipment and short-wave infrared equipment are used to capture and collect low-light images, multiple low-light images are usually captured continuously in units of one shooting cycle. The duration of the shooting cycle at this time is extremely short, which can not only ensure the fast and sensitive shooting and collection of low-light images by the on-board image acquisition system, but also achieve the continuous shooting effect of multiple low-light images. For example, one shooting cycle is 0.03S, 10 shooting moments are marked in one shooting cycle, and 10 visible light images, infrared images and short-wave infrared images are continuously captured at 10 shooting moments.

[0019] Since there are many types of devices for image acquisition in the vehicle-mounted image acquisition system, the types and spectra corresponding to low-light images are not unique. When it is necessary to enhance and optimize the low-light images, it is necessary to classify the different types of low-light images that meet the enhanced optimization processing conditions, so as to aggregate the low-light images of the same type into a subset; The subsets include a first subset, a second subset, and a third subset; wherein the first subset is a set that only includes visible light type low-light images, the second subset is a set that only includes infrared type low-light images, and the third subset is a set that only includes short-wave infrared type low-light images; The method of dividing the subsets is: The attribute information of all low-light images taken by the vehicle-mounted image acquisition system is queried one by one, and the low-light images with the attribute information of "KJ", "HW" and "DB" are recorded as visible light images, infrared images and short-wave infrared images respectively; the attribute information is used to specifically represent the type represented by the low-light image, and serves as the basis for subsequent accurate distinction of the low-light images; respectively querying the shooting time of the visible light image, infrared image and short-wave infrared image, and eliminating the visible light images, infrared images or short-wave infrared images that do not have the same shooting time; According to the sequence of shooting time, the remaining visible light images, infrared images and short-wave infrared images are numbered in ascending order to generate a first subset, a second subset and a third subset.

[0020] The image to be enhanced refers to the combination of low-light images in each subset that can be combined and matched with each other and meet the requirements of subsequent fusion and enhancement processing, and is used as the object of subsequent fusion and enhancement. In order to ensure that the image to be enhanced can maintain rationality, it is necessary to ensure that different types of low-light images can match each other under the restriction of the enhancement matching criterion, and finally form a low-light image to be enhanced that meets the requirements of subsequent fusion and enhancement processing; The enhancement matching criterion is: match the visible light image, infrared image and short-wave infrared image corresponding to the maximum value of the available coefficient; this ensures that the subsequent image to be enhanced can be in the optimal selection as a whole, laying a good foundation for subsequent fusion and enhancement; The matching method of the image to be enhanced is: Combine the visible light images, infrared images and short-wave infrared images with the same numbers in the first subset, the second subset and the third subset to generate A image units; The resolution of the visible light image and the contrast of the infrared image in A image units are queried one by one through the image management system to obtain A resolution values ​​and A contrast values; Mark the pixel values ​​of all pixels on the short-wave infrared image in A image units one by one, and add up and average the pixel values ​​of the pixels to obtain A brightness values; The resolution values, contrast values ​​and brightness values ​​of A image units are respectively assigned corresponding weight factors and added together to obtain A available coefficients; The expression of the available coefficient is: ; In the formula, No. The available coefficients of the image units, =1,2,...,A, For the The resolution value of the image unit, For the The contrast value of each image unit, For the The brightness value of each image unit, , , All are weight factors greater than 0; in, , , , It is used to adjust the proportion of resolution value, contrast value and brightness value in the available coefficient, so that the resolution value, contrast value and brightness value can have different degrees of influence on the quality of each image unit; The image unit corresponding to the maximum value of the available coefficient is recorded as a matching unit, and the visible light image, infrared image and short-wave infrared image in the matching unit are recorded as images to be enhanced.

[0021] It should be noted that the shooting time of the visible light image, infrared image and short-wave infrared image in the matched image to be enhanced is the same, so that the image to be enhanced can maintain consistency on the timeline, thereby meeting the time requirements of subsequent fusion and enhancement.

[0022] S2: Extract the spectral features of the image to be enhanced, and fuse the spectral features based on the deep learning fusion model to generate a fused image; Spectral features refer to features that can represent image information of different types and dimensions in the image to be enhanced, that is, they can comprehensively represent the real meaning and data in the image to be enhanced, and serve as the basis for the subsequent fusion of the image to be enhanced. Spectral features include edge features, color features, thermal radiation features, and reflectivity features; Edge features are used to represent edge information in visible light images. They can accurately locate the edge of related objects in visible light images and improve the detail clarity and position accuracy of objects in visible light images. The edge feature extraction method is: Convert the visible light image in the image to be enhanced into a grayscale image, and read the grayscale image; Sobel convolution kernels in the horizontal and vertical directions are configured, which are recorded as SP convolution kernel and SZ convolution kernel respectively. It should be noted that the Sobel convolution kernel is an edge detection operator used in image processing. It detects edges by calculating the gradient of an image. The Sobel operator contains two convolution kernels, which are used to calculate the gradient of an image in the horizontal and vertical directions respectively. As an existing technology, the Sobel convolution kernel can be implemented through OpenCV. Use the SP convolution kernel and the SZ convolution kernel to perform convolution operations on the image to generate a horizontal gradient image and a vertical gradient image; Mark the gradient values ​​of B pixels in the grayscale image in the horizontal and vertical directions respectively, and calculate the gradient amplitude of the B pixels by the gradient amplitude formula; The expression of the gradient amplitude is: ; In the formula, For the The gradient amplitude of each pixel, =1,2,...,B, For the The gradient value of a pixel in the horizontal direction, For the The gradient value of each pixel in the vertical direction; The pixels whose gradient amplitude is greater than the calibrated gradient threshold are recorded as target pixels, and all target pixels are connected in sequence in the grayscale image to obtain edge features. The calibrated gradient threshold refers to the minimum value of the gradient amplitude of the pixel points that serve as edge features, which can be used to provide a basis for judging and identifying the size of the gradient amplitude of all pixels. The calibrated gradient threshold is obtained by collecting the minimum values ​​of the gradient amplitudes of a large number of pixel points that are recorded as edge features in history and calculating their average value.

[0023] Color features are used to represent the distribution of color information in visible light images. When extracting color features, it is necessary to quantize the R, G, and B color channels in the visible light image, and count the number of pixels in each quantized color channel one by one, draw a color histogram, and calculate the first-order moment, second-order moment, and third-order moment of the color histogram respectively. After combining the first-order moment, second-order moment, and third-order moment, the color features are generated.

[0024] Thermal radiation features are used to represent the temperature distribution information in infrared images, which can be used as a specific reference to the temperature and location of each point in the infrared image. The method for extracting thermal radiation features is: The lower left corner of the infrared image is used as the coordinate origin, and the left boundary and bottom boundary of the infrared image are used as the vertical axis and horizontal axis respectively to construct the two-dimensional image coordinates; Mark the pixel values ​​of all pixels in the infrared image one by one, and convert the pixel values ​​into corresponding temperature values; Mark the horizontal and vertical coordinates of all pixel points through the two-dimensional image coordinates, and combine the horizontal and vertical coordinates into coordinate information; After matching the temperature values ​​with the corresponding coordinate information one by one, the thermal radiation characteristics are generated.

[0025] It should be noted that by constructing the infrared image in the form of two-dimensional coordinates, the temperature value of any point in the infrared image can be accurately represented, thereby achieving the effect of accurately representing the temperature distribution of any point in the infrared image.

[0026] The reflectivity feature is used to represent the reflectivity information of different positions in the shortwave infrared image, so as to identify the reflectivity of each pixel in different bands. Specifically, after reading the reflectivity values ​​of all pixels in the shortwave infrared image in C different bands, the reflectivity values ​​in C different bands are accumulated and averaged, and the mean is recorded as the reflectivity feature.

[0027] After the spectral features are extracted, the extracted multi-dimensional spectral features can be fused together, so that the visible light image, infrared image and short-wave infrared image can be fused into an overall fused image, ensuring that the fused image can be used as a direct object for subsequent enhancement processing; When fusing the edge features, color features, thermal radiation features, and reflectivity features in the spectral features, it is necessary to use a deep learning fusion model for auxiliary fusion, that is, to effectively and accurately fuse spectral features of different types and dimensions together, thereby generating a fused image that can represent the visible light image, infrared image, and short-wave infrared image as a whole; The deep learning fusion model is a model trained and evolved based on the deep learning model in artificial intelligence technology, so that the deep learning fusion model can output a fusion image that matches the spectral features based on the input spectral features; Specifically, when training a deep learning fusion model, it is necessary to build training data based on the spectral features composed of a large number of different types of edge features, color features, thermal radiation features and reflectivity features, and take the fused images corresponding to these spectral features as the target. Then, the spectral features are used as input samples of the deep learning model, and the fused images are used as output samples of the deep learning model. After a large number of repeated input and output training, the deep learning model can output the corresponding fused images according to the input spectral features, and obtain the deep learning fusion model.

[0028] The extracted spectral features are input into the trained deep learning fusion model, so that the deep learning fusion model can output a fused image corresponding to the input spectral features, and then the fused image can be used as the integrated fusion image result of the visible light image, infrared image and short-wave infrared image.

[0029] S3: Based on the image segmentation criterion, the fused image is segmented into sub-images, and the enhancement coefficients of the sub-images are calculated one by one; A sub-image refers to an image representing a certain position of a fused image obtained by segmenting the entire fused image according to the calibration rules. Since the fused image is an image that integrates visible light images, infrared images, and short-wave infrared images, the fused image contains more types and dimensions of information, and the amount of data contained in each regional position in the fused image is also large, which makes it difficult to directly enhance the fused image. At the same time, the overall processing of a large amount of data may also cause omissions in the enhancement processing, resulting in the phenomenon that the local detail enhancement processing in the fused image is not accurate and complete enough. Therefore, it is necessary to segment the fused image and obtain sub-images with a smaller overall area and less data. When dividing the fused image into sub-images, it is necessary to ensure that two adjacent sub-images are in an adjacent state, and the data attributes referred to in each sub-image are single. Therefore, it is necessary to rely on image segmentation criteria for image segmentation processing; The image segmentation criterion is that there is no overlapping area between adjacent sub-images; this ensures the independence of each sub-image and avoids the situation where two sub-images contain the same object information, thereby improving the accuracy of subsequent sub-image enhancement processing; The sub-image segmentation method is: Convert the fused image into a grayscale fused image, and mark the pixel points corresponding to the D edge features one by one on the grayscale fused image; In a clockwise manner, pixel points corresponding to the D edge features are connected in sequence to form a closed area, and D first areas are obtained; A preset first length is used as a segmentation width, and a preset second length is used as a segmentation height. A vertical segmentation line and a horizontal segmentation line are drawn on the grayscale fusion image, and based on the vertical segmentation line and the horizontal segmentation line, the grayscale fusion image is divided into E images to be adjusted. The preset first length and the preset second length refer to the length basis for equally spaced segmentation of the grayscale fusion image in the horizontal direction and the vertical direction, respectively, which can provide a basis for accurate segmentation of the grayscale fusion image in the horizontal direction and the vertical direction, so as to ensure the consistency of the division of the images to be adjusted. The image to be adjusted including the first region is recorded as a target image, the first region in the target image is eliminated, and the remaining part of the target image, the remaining image to be adjusted and the D first regions are collectively referred to as sub-images to obtain E sub-images.

[0030] After the sub-images are obtained, the enhancement coefficients need to be calculated for each sub-image, so that the enhancement coefficients can be used as a basis for judging the order of sub-image enhancement, and the enhancement coefficients can also be used as a numerical representation of the difficulty of sub-image enhancement processing; The calculation method of the enhancement factor is: Count the number of pixels in the E sub-images one by one to obtain E pixel values; After accumulating the pixel values ​​of all pixels in the E sub-images one by one, E total pixel values ​​are obtained, and the E total pixel values ​​are compared with the E pixel values ​​and averaged to obtain E enhancement coefficients; The expression of the enhancement coefficient is: ; In the formula, For the The enhancement factor of the sub-image is =1,2,...,E, For the The total pixel value of sub-images, For the The pixel value of each sub-image.

[0031] After the enhancement coefficient of the sub-image is calculated, the enhancement order of the sub-images can be arranged according to the size of the enhancement coefficient. Specifically, the greater the enhancement coefficient, the more difficult it is to enhance the sub-image, and the higher the order of enhancement processing of the sub-image, and vice versa.

[0032] S4: Taking the enhancement coefficient as the enhancement order, the sub-images are enhanced in sequence through the deep learning enhancement model to generate enhanced sub-images; The enhanced sub-image refers to the sub-image enhanced by the deep learning enhancement model, so that the enhanced sub-image can perform detailed enhancement processing on the low-light image, and perform more detailed and accurate enhancement operations on the low-light image, avoiding the problem of insufficient details and accuracy of the enhancement processing caused by the overall enhancement processing of the low-light image; The deep learning enhancement model is a model trained and evolved based on the deep learning model in artificial intelligence technology, so that the deep learning enhancement model can output an enhanced sub-image adapted to the sub-image based on the sub-image, and the object information and details that are not clear and obvious in the sub-image can be more obvious and clear in the enhanced sub-image; Specifically, when training a deep learning enhancement model, it is necessary to construct training data based on a large number of different historical sub-images and take the enhanced sub-images corresponding to these sub-images as the target, and then use the sub-images as input samples of the deep learning model, and the enhanced sub-images as output samples of the deep learning model. After a large number of repeated input and output training, the deep learning model can output the corresponding enhanced sub-image according to the input sub-image, and obtain the deep learning enhancement model.

[0033] After the deep learning enhancement model is trained, the obtained sub-images are sequentially input into the deep learning enhancement model to obtain enhanced sub-images; The method for generating the enhanced sub-image is: According to the enhancement coefficient from large to small, the E sub-images are numbered in ascending order, with 1 being the first number; When sub-images have enhancement coefficients of the same size, the sub-images with the same enhancement coefficients are recorded as images to be sorted; Compare the total pixel values ​​of the images to be sorted one by one, and arrange all the images to be sorted in order from large to small total pixel values; According to the order of numbering from small to large, the E sub-images are input into the deep learning enhancement model one by one, E enhanced sub-images are output, and the numbers of the E sub-images are assigned to the E enhanced sub-images one by one.

[0034] It should be noted that the acquired enhanced sub-image cannot be directly used as the enhanced image of the low-light vehicle-mounted image. It can be used as a prerequisite for subsequent use, and through operations such as splicing of the enhanced sub-image, it can meet subsequent vehicle-mounted use needs.

[0035] S5: splicing the enhanced sub-images into an enhanced low-light image, and annotating the enhanced interpretation in the enhanced low-light image; The enhanced low-light image refers to the final image after the low-light vehicle image is enhanced and processed, and is used as the image output by the image processor. At the same time, it can provide the vehicle image recognition system with relevant driving judgment basis. Since the enhanced low-light image is used to represent the low-light image as a whole, the enhanced low-light image needs to splice the E enhanced sub-images obtained after segmentation and enhancement; The stitching method for enhancing low-light images is: Construct an image stitching frame identical to the grayscale fusion image, segment the stitching units identical to the E sub-images in the image stitching frame, and convert the numbers of the E sub-images to the E stitching units accordingly; the image stitching frame refers to a blank infrastructure without any substantial image, and is used as a definition of the stitching boundary and area of ​​the enhanced low-light image; the stitching unit is the smallest component of the image stitching frame, and provides a position guide for the subsequent import and stitching of the enhanced sub-images; Fill the E enhanced sub-images into the E stitching units in order according to the number from small to large, and obtain the basic stitching frame; In the basic stitching frame, the stitching boundaries between two adjacent stitching units are drawn one by one, and all the pixels on the stitching boundaries are marked to obtain F stitching pixels; the stitching boundary refers to the overall structure of the points at the connection between any two adjacent stitching units, which can be used to distinguish different stitching units; The stitching units located on both sides of the F stitching pixels are recorded as adjacent units to obtain F groups of adjacent units, and the distance values ​​between the pixels in the F groups of adjacent units and the corresponding F stitching pixels are measured one by one, and all the distance values ​​of the F stitching pixels are summarized to obtain F distance value sets; Record two pixel points corresponding to the minimum values ​​in the F distance value sets as the first conversion point and the second conversion point respectively, and obtain F first conversion points and F second conversion points; The pixel values ​​of the F first conversion points and the corresponding F second conversion points are added and averaged in sequence to obtain F conversion values, and the F conversion values ​​are correspondingly assigned to the F splicing pixel points to obtain an enhanced low-light image.

[0036] It should be noted that, by assigning corresponding conversion values ​​to the stitching pixel points, the pixel value of each stitching pixel point can be in an intermediate state with the pixel value of the corresponding first conversion point and the second conversion point, so that the brightness of the stitching boundary can maintain a gradual decrease or increase effect with the brightness on the adjacent units, avoiding a large gap in brightness between the stitching boundary and the adjacent units, thereby providing a positive effect on seamless stitching between adjacent units.

[0037] After the enhanced low-light image is stitched together, the useful information in the enhanced low-light image needs to be annotated, so as to help the vehicle-mounted image recognition system to quickly and accurately recognize the image, and achieve the effect of accurate enhancement and clear interpretation of the low-light vehicle-mounted image. At this time, the interpretation annotated on the enhanced low-light image is the enhanced interpretation; Enhancement includes high attention area and regular attention area; high attention area refers to the location and information that need special attention in the enhanced low-light image, and regular attention area refers to the location and information that do not need special attention in the enhanced low-light image; The annotation method for enhanced interpretation is: In the enhanced low-light image, the regions identical to the D first regions are recorded as height regions, and the remaining regions are recorded as regular regions; A remark box with two attention points is established on the D height areas and the regular area, and the two attention points are recorded as the first point and the second point respectively; the remark box is a text box used to represent the content of the enhanced interpretation as a whole, and the attention point is the smallest unit constituting the remark box, thereby providing a remark basis for subsequent enhanced interpretations of different types; The first and second points in the remark boxes of D height areas are both marked with "1", and D height attention areas are obtained; Add "1" to the first point in the remark box of the general area to obtain the general attention area.

[0038] It should be noted that when a high-attention area is noted on the enhanced low-light image, it means that information that may threaten driving safety has appeared in the environment outside the vehicle, and the driver needs to pay more attention to the situation. When a regular-attention area is noted on the enhanced low-light image, it means that no information that may threaten driving safety has appeared in the environment outside the vehicle, and the driver only needs to pay regular attention to the situation.

[0039] In this embodiment, by dividing the captured low-light images into subsets, and matching the images to be enhanced from the subsets based on the enhancement matching criteria, it is possible to match the images that best meet the image fusion requirements from different types of low-light images, effectively avoiding the negative impact of a large number of low-quality low-light images participating in image fusion, and by extracting the spectral features of the images to be enhanced and fusing the spectral features based on the deep learning fusion model to generate a fused image, spectral features of different types and dimensions can be effectively and accurately fused together, achieving the effect of overall fusion representation of the multi-spectral image, and based on the image segmentation criteria, the fused image is segmented into sub-images, and the enhancement coefficients of the sub-images are calculated one by one, with the enhancement coefficients as the enhancement order, through deep learning The enhancement model performs image enhancement on sub-images in sequence to generate enhanced sub-images, which can not only divide the integrated image containing more information into multiple sub-images and reduce the workload of enhancing each sub-image, but also improve the detail fineness of the sub-image during the enhancement processing, avoiding the problem of low detail enhancement accuracy that is easy to occur when the fused image is enhanced as a whole. At the same time, by splicing the enhanced sub-images into enhanced low-light images and annotating the enhanced interpretation in the enhanced low-light images, the enhanced low-light images can be accurately annotated and classified, providing an accurate judgment basis for the vehicle-mounted image recognition system to recognize low-light vehicle-mounted images, thereby achieving high-precision enhancement effects of low-light vehicle-mounted images and improving the working performance of the vehicle-mounted image recognition system.

[0040] Example 2: Please refer to Figure 2 As shown, the part not described in detail in this embodiment is described in the first embodiment, and a low-light vehicle-mounted image enhancement system based on multi-spectral fusion and deep learning is provided, which is applied to an image processor to implement a low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning, including an image matching module, an image fusion module, an image segmentation module, an image enhancement module and an image stitching module, wherein each module is connected via a wired or wireless network; An image matching module is used to divide the captured low-light images into subsets, and match the image to be enhanced from the subsets based on an enhancement matching criterion; The image fusion module is used to extract the spectral features of the image to be enhanced, and fuse the spectral features based on the deep learning fusion model to generate a fused image; An image segmentation module is used to segment the fused image into sub-images based on an image segmentation criterion, and calculate enhancement coefficients of the sub-images one by one; An image enhancement module is used to enhance the sub-images in sequence by using the enhancement coefficients as the enhancement order through a deep learning enhancement model to generate enhanced sub-images; The image stitching module is used to stitch the enhanced sub-images into an enhanced low-light image and annotate the enhanced interpretation in the enhanced low-light image.

[0041] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A low-light vehicle-mounted image enhancement method based on multispectral fusion and deep learning, applied to an image processor, characterized in that: include: S1: Divide the captured low-light image into subsets, and match the image to be enhanced from the subset based on the enhancement matching criterion. The enhancement matching criterion is: match the visible light image, infrared image and short-wave infrared image corresponding to the maximum value of the available coefficient; S2: Extract the spectral features of the image to be enhanced, including edge features, color features, thermal radiation features and reflectivity features, and fuse the spectral features based on the deep learning fusion model to generate a fused image; S3: Based on the image segmentation criterion, the fused image is segmented into sub-images, and the enhancement coefficients of the sub-images are calculated one by one. The image segmentation criterion is that there is no overlapping area between adjacent sub-images; S4: Taking the enhancement coefficient as the enhancement order, the sub-images are enhanced in sequence through the deep learning enhancement model to generate enhanced sub-images; S5: stitching the enhanced sub-images into an enhanced low-light image, and annotating the enhanced interpretation in the enhanced low-light image, where the enhanced interpretation includes a high-attention area and a normal-attention area.

2. The low-light vehicle-mounted image enhancement method based on multispectral fusion and deep learning according to claim 1 is characterized in that: The subsets include a first subset, a second subset, and a third subset; The method of dividing the subsets is: The attribute information of all low-light images taken by the vehicle-mounted image acquisition system is queried one by one, and the low-light images with the attribute information of "KJ", "HW" and "DB" are recorded as visible light images, infrared images and short-wave infrared images respectively; respectively querying the shooting time of the visible light image, infrared image and short-wave infrared image, and eliminating the visible light images, infrared images or short-wave infrared images that do not have the same shooting time; According to the sequence of shooting time, the remaining visible light images, infrared images and short-wave infrared images are numbered in ascending order to generate a first subset, a second subset and a third subset.

3. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 2 is characterized in that: The matching method of the image to be enhanced is: Combine the visible light images, infrared images and short-wave infrared images with the same numbers in the first subset, the second subset and the third subset to generate A image units; The resolution of the visible light image and the contrast of the infrared image in A image units are queried one by one through the image management system to obtain A resolution values ​​and A contrast values; Mark the pixel values ​​of all pixels on the short-wave infrared image in A image units one by one, and add up and average the pixel values ​​of the pixels to obtain A brightness values; The resolution values, contrast values ​​and brightness values ​​of A image units are respectively assigned corresponding weight factors and added together to obtain A available coefficients; The image unit corresponding to the maximum value of the available coefficient is recorded as a matching unit, and the visible light image, infrared image and short-wave infrared image in the matching unit are recorded as images to be enhanced.

4. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 3 is characterized in that: The edge feature extraction method is: Convert the visible light image in the image to be enhanced into a grayscale image, and read the grayscale image; Configure the Sobel convolution kernels in the horizontal and vertical directions, which are denoted as SP convolution kernel and SZ convolution kernel respectively; Use the SP convolution kernel and the SZ convolution kernel to perform convolution operations on the image to generate a horizontal gradient image and a vertical gradient image; Mark the gradient values ​​of B pixels in the grayscale image in the horizontal and vertical directions respectively, and calculate the gradient amplitude of the B pixels by the gradient amplitude formula; The expression of the gradient amplitude is: ； In the formula, For the The gradient amplitude of each pixel, =1,2,...,B, For the The gradient value of a pixel in the horizontal direction, For the The gradient value of each pixel in the vertical direction; The pixel points whose gradient amplitude is greater than the calibrated gradient threshold are recorded as target pixels, and all the target pixels are connected in sequence in the grayscale image to obtain edge features.

5. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 4 is characterized in that: The method for extracting thermal radiation features is: The lower left corner of the infrared image is used as the coordinate origin, and the left boundary and bottom boundary of the infrared image are used as the vertical axis and horizontal axis respectively to construct the two-dimensional image coordinates; Mark the pixel values ​​of all pixels in the infrared image one by one, and convert the pixel values ​​into corresponding temperature values; Mark the horizontal and vertical coordinates of all pixel points through the two-dimensional image coordinates, and combine the horizontal and vertical coordinates into coordinate information; After matching the temperature values ​​with the corresponding coordinate information one by one, the thermal radiation characteristics are generated.

6. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 5 is characterized in that: The sub-image segmentation method is: Convert the fused image into a grayscale fused image, and mark the pixel points corresponding to the D edge features one by one on the grayscale fused image; In a clockwise manner, pixel points corresponding to D edge features are connected in sequence to form a closed area, and D first areas are obtained; Using a preset first length as a segmentation width and a preset second length as a segmentation height, a vertical segmentation line and a horizontal segmentation line are drawn on the grayscale fusion image respectively, and based on the vertical segmentation line and the horizontal segmentation line, the grayscale fusion image is divided into E images to be adjusted; The image to be adjusted including the first region is recorded as a target image, the first region in the target image is eliminated, and the remaining part of the target image, the remaining image to be adjusted and the D first regions are collectively referred to as sub-images to obtain E sub-images.

7. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 6 is characterized in that: The calculation method of the enhancement factor is: Count the number of pixels in the E sub-images one by one to obtain E pixel values; After accumulating the pixel values ​​of all pixels in the E sub-images one by one, E total pixel values ​​are obtained, and the E total pixel values ​​are compared with the E pixel values ​​and averaged to obtain E enhancement coefficients; The expression of the enhancement coefficient is: ； In the formula, For the The enhancement factor of the sub-images, =1,2,...,E, For the The total pixel value of sub-images, For the The pixel value of each sub-image.

8. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 7 is characterized in that: The method for generating the enhanced sub-image is: According to the enhancement coefficient from large to small, the E sub-images are numbered in ascending order, with 1 being the first number; When sub-images have enhancement coefficients of the same size, the sub-images with the same enhancement coefficients are recorded as images to be sorted; Compare the total pixel values ​​of the images to be sorted one by one, and arrange all the images to be sorted in order from large to small total pixel values; According to the order of numbering from small to large, the E sub-images are input into the deep learning enhancement model one by one, E enhanced sub-images are output, and the numbers of the E sub-images are assigned to the E enhanced sub-images one by one.

9. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 8, characterized in that: The stitching method for enhancing low-light images is: Construct an image stitching frame identical to the grayscale fused image, segment the image stitching frame into stitching units identical to the E sub-images, and convert the numbers of the E sub-images to the E stitching units accordingly; Fill the E enhanced sub-images into the E splicing units in order according to the number from small to large, and obtain the basic splicing frame; In the basic stitching frame, the stitching boundaries between two adjacent stitching units are drawn one by one, and all the pixels on the stitching boundaries are marked to obtain F stitching pixels; The stitching units located on both sides of the F stitching pixels are recorded as adjacent units to obtain F groups of adjacent units, and the distance values ​​between the pixels in the F groups of adjacent units and the corresponding F stitching pixels are measured one by one, and all the distance values ​​of the F stitching pixels are summarized to obtain F distance value sets; Record two pixel points corresponding to the minimum values ​​in the F distance value sets as the first conversion point and the second conversion point respectively, and obtain F first conversion points and F second conversion points; The pixel values ​​of the F first conversion points and the corresponding F second conversion points are added and averaged in sequence to obtain F conversion values, and the F conversion values ​​are correspondingly assigned to the F splicing pixel points to obtain an enhanced low-light image.

10. The low-light vehicle-mounted image enhancement method based on multi-spectral fusion and deep learning according to claim 9, characterized in that: The annotation method for enhanced interpretation is: In the enhanced low-light image, the regions identical to the D first regions are recorded as height regions, and the remaining regions are recorded as regular regions; A remark box with two attention points is established on the D height areas and the conventional area respectively, and the two attention points are recorded as the first point and the second point respectively; The first and second points in the remark boxes of D height areas are both marked with "1", and D height attention areas are obtained; Add "1" to the first point in the remark box of the general area to obtain the general attention area.

Images