Multi-source image information fusion method for low-illumination scene robot positioning

Abstract

The invention discloses a multi-source image information fusion method for low-illumination scene robot positioning, and belongs to the technical field of low-illumination scene image processing. The method aims at solving the problem that in a low-illumination environment, an existing image fusion method cannot adjust the image fusion scale in a self-adaptive mode, and image features cannot be utilized to the maximum degree. Comprising the following steps: performing two-dimensional wavelet image decomposition on an acquired infrared light image and a low-illumination image to obtain an infrared light high-frequency image sub-band, a low-frequency image sub-band, a low-illumination high-frequency image sub-band and a low-frequency image sub-band; respectively fusing the infrared light, the low-illumination high-frequency image sub-band and the low-frequency image sub-band to obtain a high-frequency first-level fused image sub-band and a low-frequency first-level fused image sub-band; performing threshold processing and histogram equalization processing to obtain a high-frequency fusion sub-band and a low-frequency fusion sub-band; and based on the illuminance, selecting the high-frequency weight coefficient and the low-frequency weight coefficient to carry out image weighted fusion to obtain a final fused image. The method is used for restoring the low-illumination image.

Description

Technical Field

[0001] The invention relates to a multi-source image information fusion method for low-illumination scene robot positioning, and belongs to the technical field of low-illumination scene image processing. Background Art

[0002] Low illumination environment generally refers to low illumination environment with weak or dim light level. Usually, the illumination of low illumination environment is less than 100 Lux. In such a lighting environment, the mobile robot cannot capture enough feature points in real time, resulting in the robot being unable to complete positioning in real time, thus affecting the subsequent SLAM task (Simultaneous Localization and Mapping Task). At present, under the premise of meeting the real-time requirements of the robot, the effect of single processing of low illumination images is not ideal, and the image feature point recovery effect is poor. However, image enhancement methods that can meet the real-time requirements and have excellent processing effects are rarely reported. In contrast, image fusion has great potential, considering that infrared cameras can obtain clear targets at any illumination, but the image details are seriously lost; although low illumination images taken by ordinary cameras cannot obtain clear targets, the details are relatively intact. Therefore, multi-source image fusion is often used for low illumination environments.

[0003] Existing research on infrared image fusion can be mainly divided into traditional methods and learning-based methods. Among them, the traditional method mainly removes the noise in the image by image domain transformation, and decomposes and fuses the image by scale transformation. Although it can bring a certain fusion effect, for robot tasks in low-light environments, specific fusion rules need to be set and a large amount of calculations need to be performed. The situation of too low illumination is poorly handled and the image feature points cannot be restored well, which greatly limits the promotion and application of robots in practical applications. Most importantly, for the situation of illumination changes in actual situations, traditional methods cannot adaptively adjust the scale of image fusion, which results in the inability to maximize the use of the only image features, making it impossible for the robot to capture enough feature points in the fused image, making it difficult to complete the robot's self-positioning and subsequent tasks. Learning-based methods often extract detailed features of images through deep learning and use the network to generate decision diagrams. This method can obtain fused images with better visual perception, but rarely optimizes image fusion rules for low-light environments; or uses the generator and discriminator of the adversarial network to play games and continuously increase the detailed features of the fused image. Although it can complete specific tasks, it requires setting reasonable parameters and a large number of simulations, resulting in poor generalization, too low robustness for applications in actual scenarios, and unstable algorithm performance. Summary of the invention

[0004] In view of the problem that the existing image fusion method cannot adaptively adjust the image fusion scale in a low-light environment, so that image features cannot be utilized to the maximum extent, the present invention provides a multi-source image information fusion method for robot positioning in low-light scenes.

[0005] A multi-source image information fusion method for low-light scene robot positioning of the present invention comprises:

[0006] Perform two-dimensional wavelet image decomposition on the collected infrared light image and low-illumination image respectively to obtain infrared light high-frequency image subband, infrared light low-frequency image subband, low-illumination high-frequency image subband and low-illumination low-frequency image subband; fuse the infrared light high-frequency image subband and the low-illumination high-frequency image subband to obtain a high-frequency first-level fused image subband, and then perform threshold processing to obtain a high-frequency fused subband; fuse the infrared light low-frequency image subband and the low-illumination low-frequency image subband to obtain a low-frequency first-level fused image subband, and then perform histogram equalization processing to obtain a low-frequency fused subband;

[0007] The high-frequency weight coefficient and the low-frequency weight coefficient are selected based on the illumination of the low-illumination scene, and the high-frequency fusion sub-band and the low-frequency fusion sub-band are weightedly fused to obtain the final fused image.

[0008] According to the multi-source image information fusion method for low-illumination scene robot positioning of the present invention, the threshold processing is Haar wavelet VisuShrink hard threshold processing.

[0009] According to the multi-source image information fusion method for low-light scene robot positioning of the present invention, the histogram equalization processing is a combined processing of a segmentation grid size of 2×2 and a histogram threshold of 1.5.

[0010] According to the multi-source image information fusion method for robot positioning in low-illuminance scenes of the present invention, the method for selecting high-frequency weight coefficients and low-frequency weight coefficients based on the illumination of the low-illuminance scene is to select high-frequency weight coefficients and low-frequency weight coefficients based on a preset illumination-weight coefficient curve.

[0011] According to the multi-source image information fusion method for low-illumination scene robot positioning of the present invention, the method for obtaining the preset illumination-weight coefficient curve is:

[0012] In a low-illumination scene, infrared light image samples and low-illumination image samples are collected, and high-frequency fusion sub-band samples and low-frequency fusion sub-band samples are obtained;

[0013] Set the initial value of the high-frequency weight coefficient ω1 and the initial value of the low-frequency weight coefficient ω2, and iteratively update the high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2:

[0014] Under the current illumination, the high-frequency fusion sub-band samples and the low-frequency fusion sub-band samples are weightedly fused based on the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 to obtain the current fused image; the feature points of the current fused image are extracted, and the number of current feature points is determined;

[0015] If the current number of feature points is greater than the number of feature points obtained in the previous round of iteration, the final high-frequency weight coefficient and the final low-frequency weight coefficient are updated, and the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are updated to continue the iteration; otherwise, the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are directly updated to continue the iteration until the iteration termination condition is reached, and the final high-frequency weight coefficient and the final low-frequency weight coefficient are output;

[0016] The final high-frequency weight coefficient and the final low-frequency weight coefficient corresponding to different illuminations are fitted by a neural network regression method to obtain a preset illumination-weight coefficient curve.

[0017] According to the multi-source image information fusion method for robot positioning in low-illumination scenes of the present invention, the low-illumination scenes include indoor scenes, outdoor scenes and complex background scenes; in each low-illumination scene, a preset illumination-weight coefficient curve is obtained respectively, and then the three preset illumination-weight coefficient curves are fused by using the cubic spline interpolation method to obtain the fused preset illumination-weight coefficient curve.

[0018] According to the multi-source image information fusion method for low-illumination scene robot positioning of the present invention, the method of selecting high-frequency weight coefficients and low-frequency weight coefficients based on the illumination of the low-illumination scene is:

[0019] Set the initial value of the high-frequency weight coefficient ω1 and the initial value of the low-frequency weight coefficient ω2, and iteratively update the high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2:

[0020] Under the current illumination, the high-frequency fusion subband and the low-frequency fusion subband are weightedly fused based on the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 to obtain the current fused image; the feature points of the current fused image are extracted, and the number of current feature points is determined;

[0021] If the current number of feature points is greater than the number of feature points obtained in the previous round of iteration, the final high-frequency weight coefficient and the final low-frequency weight coefficient are updated, and the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are updated to continue the iteration; otherwise, the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are directly updated to continue the iteration until the iteration termination condition is reached, and the final high-frequency weight coefficient and the final low-frequency weight coefficient are output.

[0022] According to the multi-source image information fusion method for robot positioning in low-illumination scenes of the present invention, ω1+ω2=1.

[0023] According to the multi-source image information fusion method for low-light scene robot positioning of the present invention, the final fused image is represented as X(x, y):

[0024]

[0025] Where (i, j) is the row and column coefficient of the block index corresponding to the image subband, H L (i, j) is the wavelet coefficient of the low-illumination high-frequency image subband, is the wavelet basis function of the low-illumination high-frequency image subband, H I (i, j) is the wavelet coefficient of the infrared high-frequency image subband, is the wavelet basis function of the infrared high-frequency image subband; L L (i, j) is the wavelet coefficient of the low-illumination low-frequency image subband, is the wavelet basis function of the low-illumination low-frequency image subband, L I (i,j) is the wavelet coefficient of the infrared low-frequency image subband, is the wavelet basis function of the infrared low-frequency image subband; where (x, y) is the horizontal and vertical coordinates of the image;

[0026] in is the high frequency fusion subband, It is the low frequency fusion subband.

[0027] According to the multi-source image information fusion method for low-light scene robot positioning of the present invention, the high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2 are updated according to a set step size.

[0028] The beneficial effects of the present invention are as follows: both low-light images and infrared images can be decomposed into high-frequency image subbands and low-frequency image subbands through filters, and each of them has obvious feature differences, and the filtering operation can also complete the denoising of the image. In the fusion of different infrared image ratios, the image feature point recovery effect varies greatly. Therefore, considering the different illuminations, the required fusion ratio of high-frequency image information and low-frequency image information will directly affect the quality of image fusion. The present invention combines the principle of feature point extraction to construct an infrared image information fusion framework for robot positioning in low-light scenes, and realizes image information fusion in low-light environments through visual input; through the cooperation of photometric sensors, the adaptive control weighting ratio is realized according to different illumination values ​​to complete image fusion; the method of the present invention makes the most of the feature information of high- and low-frequency image subbands, and provides sufficient image feature points for low-light robot positioning tasks, so that the robot can also perform operation tasks in low-light scenes.

[0029] The method of the present invention performs weighted fusion of high-frequency image subbands and low-frequency image subbands by coordinating the input image with the illumination information, thereby greatly improving the number of feature points of the fused image and reducing the impact of the low-illumination environment on the robot positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is an overall flow chart of the multi-source image information fusion method for robot positioning in low-light scenes according to the present invention;

[0031] Figure 2 It is a schematic diagram of the optional method of threshold processing of the high-frequency first-level fusion image subband;

[0032] Figure 3 It is a schematic diagram of selectable combinations of histogram equalization processing of low-frequency first-level fusion image subbands;

[0033] Figure 4 It is a schematic diagram of the iterative process of obtaining the final high-frequency weight coefficient and the final low-frequency weight coefficient;

[0034] Figure 5 It is a schematic diagram of obtaining a fused preset illuminance-weight coefficient curve from three preset illuminance-weight coefficient curves;

[0035] Figure 6 It is a schematic diagram of the implementation flow of the method of the present invention;

[0036] Figure 7 It is a comparison diagram of the fusion results of the image fusion method of the present invention and the existing method; the numbers in the figure are the number of feature points; Original represents the original low-light image; Ours represents the fusion result of the method of the present invention;

[0037] Figure 8 It is a schematic diagram of the number and distribution of ORB feature points at four consecutive moments t1 to t4; the first image in each row is a low-light image, the second is an infrared image, and the third is a fused image; the illumination of the low-light image at moment t1 is 19 Lux, the illumination of the low-light image at moment t2 is 17 Lux, the illumination of the low-light image at moment t3 is 12 Lux, and the illumination of the low-light image at moment t4 is 7 Lux; the number corresponding to each image is the number of feature points;

[0038] Fig. 9 is a comparison diagram of information entropy EN of image fusion results using the method of the present invention and the existing method; wherein IMAGE represents the number of experiments, and VALUE represents the value;

[0039] Fig.10 It is a structural similarity SSIM comparison diagram of image fusion results using the method of the present invention and the existing method;

[0040] Fig.11 It is a comparison diagram of the average gradient AG of the image fusion results using the method of the present invention and the existing method;

[0041] Fig.12 This is an ORB comparison chart of the number of feature points of image fusion results using the method of the present invention and the existing method. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0044] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0045] Specific implementation method 1. Combination Figures 1 to 6 As shown, the present invention provides a multi-source image information fusion method for robot positioning in low-light scenes, comprising:

[0046] Perform two-dimensional wavelet image decomposition on the collected infrared light image and low-illumination image respectively to obtain infrared light high-frequency image subband, infrared light low-frequency image subband, low-illumination high-frequency image subband and low-illumination low-frequency image subband; fuse the infrared light high-frequency image subband and the low-illumination high-frequency image subband to obtain a high-frequency first-level fused image subband, and then perform threshold processing to obtain a high-frequency fused subband; fuse the infrared light low-frequency image subband and the low-illumination low-frequency image subband to obtain a low-frequency first-level fused image subband, and then perform histogram equalization processing to obtain a low-frequency fused subband;

[0047] The high-frequency weight coefficient and the low-frequency weight coefficient are selected based on the illumination of the low-illumination scene, and the high-frequency fusion sub-band and the low-frequency fusion sub-band are weightedly fused to obtain the final fused image.

[0048] The multi-source image information fusion method described in this embodiment can be implemented by three modules, namely, an image preprocessing module, a weight module and an image fusion module. Figure 1As shown in the figure, the image preprocessing module obtains high-frequency fusion subbands and low-frequency fusion subbands based on infrared images and low-illumination images; the weight module includes the prediction of weighting coefficients and the sorting of data, and works with the photometer to determine the most appropriate final high-frequency weight coefficients and low-frequency weight coefficients according to different illumination values; finally, the image fusion module is used for weighted fusion.

[0049] The image preprocessing module includes image decomposition and image processing. First, the image is divided into four image subbands through two-dimensional wavelet image decomposition, and the high and low frequency parts of each image are randomly fused to obtain two parts: high-frequency first-level fused image subband and low-frequency first-level fused image subband. The image processing part includes selecting the best image processing method from different image processing and processing the first-level fused image subband. As an example, the threshold processing is Haar wavelet VisuShrink hard threshold processing. Figure 2 As shown, the Haar wavelet VisuShrink hard threshold is finally selected from different threshold combination processing (16 kinds) to process the decomposed high-frequency primary fusion image subband.

[0050] As an example, the histogram equalization process is a combination process of a segmentation grid size of 2×2 and a histogram threshold of 1.5. Figure 3 As shown in the figure, in different histogram equalization processes (12 types), the combination of segmentation grid size (2×2) and histogram threshold value of 1.5 is used to process the low-frequency first-level fusion image subband. This processing combination has the best information value under the premise of ensuring the removal of image noise and not deviating from the original image structure. Therefore, the image preprocessing module provides clean and tidy image data for subsequent processing.

[0051] This implementation provides two methods for selecting high-frequency weight coefficients and low-frequency weight coefficients:

[0052] The first method is to select the high-frequency weight coefficient and the low-frequency weight coefficient based on the illuminance of the low-illuminance scene, which is to select the high-frequency weight coefficient and the low-frequency weight coefficient based on a preset illuminance-weight coefficient curve.

[0053] The method for obtaining the preset illumination-weight coefficient curve is:

[0054] In a low-illumination scene, infrared light image samples and low-illumination image samples are collected, and high-frequency fusion sub-band samples and low-frequency fusion sub-band samples are obtained;

[0055] Set the initial value of the high-frequency weight coefficient ω1 and the initial value of the low-frequency weight coefficient ω2, and iteratively update the high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2:

[0056] Under the current illumination, the high-frequency fusion sub-band samples and the low-frequency fusion sub-band samples are weightedly fused based on the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 to obtain the current fused image; the feature points of the current fused image are extracted, and the number of current feature points is determined;

[0057] If the current number of feature points is greater than the number of feature points obtained in the previous round of iteration, the final high-frequency weight coefficient and the final low-frequency weight coefficient are updated, and the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are updated to continue the iteration; otherwise, the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are directly updated to continue the iteration until the iteration termination condition is reached, and the final high-frequency weight coefficient and the final low-frequency weight coefficient are output;

[0058] The final high-frequency weight coefficient and the final low-frequency weight coefficient corresponding to different illuminations are fitted by a neural network regression method to obtain a preset illumination-weight coefficient curve.

[0059] The low illumination scenes include indoor scenes, outdoor scenes and complex background scenes; in each low illumination scene, a preset illumination-weight coefficient curve is obtained respectively, and then the three preset illumination-weight coefficient curves are fused by using the cubic spline interpolation method to obtain the fused preset illumination-weight coefficient curve.

[0060] The weight module is mainly responsible for solving the problem of changes in weight coefficients caused by changes in illumination. This implementation method determines the optimal weight coefficient under the illumination through an iterative process in conjunction with a photometer. Figure 4 As shown, after iterating according to the process, scattered points in the form of "illuminance value-weight coefficient" are obtained, and then a curve corresponding to the illumination value and the weight coefficient is obtained through neural network regression.

[0061] In order to make the obtained data more extensive, this implementation method selects three different low-light scenes (indoor, outdoor, and complex background scenes) to obtain three "illuminance value-weight coefficient" curves, such as Figure 5 As shown. Finally, the three curves are fused and sorted using the cubic spline interpolation method to obtain a fused "illuminance value-weight coefficient" curve, which is pre-stored in the weight module. When the robot is working, the photometer transmits the illuminance information to the weight module in real time. Then the weight module will call the corresponding weight coefficient according to the illuminance value, and then pass it to the image fusion module to complete the weighted fusion of the image.

[0062] The second method is to select high-frequency weight coefficients and low-frequency weight coefficients based on the illumination of low-illumination scenes:

[0063] Set the initial value of the high-frequency weight coefficient ω1 and the initial value of the low-frequency weight coefficient ω2, and iteratively update the high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2:

[0064] Under the current illumination, the high-frequency fusion subband and the low-frequency fusion subband are weightedly fused based on the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 to obtain the current fused image; the feature points of the current fused image are extracted, and the number of current feature points is determined;

[0065] If the current number of feature points is greater than the number of feature points obtained in the previous round of iteration, the final high-frequency weight coefficient and the final low-frequency weight coefficient are updated, and the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are updated to continue the iteration; otherwise, the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are directly updated to continue the iteration until the iteration termination condition is reached, and the final high-frequency weight coefficient and the final low-frequency weight coefficient are output.

[0066] In this embodiment, the two methods for determining the weight coefficient are the same. The first method obtains the corresponding curve in advance, and directly determines the weight coefficient based on the curve in actual use; the second method determines the optimal weight coefficient online in real time.

[0067] In this embodiment, ω1+ω2=1.

[0068] Going further, the final fused image is represented as X(x,y):

[0069]

[0070] Where (i, j) is the row and column coefficient of the block index corresponding to the image subband, H L (i, j) is the wavelet coefficient of the low-illumination high-frequency image subband, is the wavelet basis function of the low-illumination high-frequency image subband, H I (i, j) is the wavelet coefficient of the infrared high-frequency image subband, is the wavelet basis function of the infrared high-frequency image subband; L L (i, j) is the wavelet coefficient of the low-illumination low-frequency image subband, is the wavelet basis function of the low-illumination low-frequency image subband, L I (i,j) is the wavelet coefficient of the infrared low-frequency image subband, is the wavelet basis function of the infrared low-frequency image subband; where (x, y) is the horizontal and vertical coordinates of the image;

[0071] in is the high frequency fusion subband, It is the low frequency fusion subband.

[0072] The high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2 are updated according to a set step size.

[0073] This embodiment can control the fusion ratio of high-frequency image information to low-frequency image information by selecting weight coefficients while fusing infrared light images with low-illuminance images. The change in illumination is determined by a photometer.

[0074] Figure 6 As shown, first the light sensor measures a reading of l n Lux, then the illumination information will be passed to the weight module, and the weighting coefficient under the corresponding illumination is found to be ω n (high-frequency weight coefficient), and then determine the corresponding low-frequency weight coefficient, import it into the image fusion module, and complete the weighted fusion of the image.

[0075] The method of the present invention completes the information fusion of infrared light images and low-light images through the coordinated cooperation of three modules, and can control the weighted fusion of high-frequency and low-frequency image information of different proportions. At the same time, the method realizes the maximum utilization of limited image information features by cooperating with the photometer to achieve the maximum utilization of limited image information features, thereby significantly improving the restoration effect of image feature points. The image restoration effect is compared with the current mainstream methods (including traditional methods and methods based on deep learning) (DWT, NSST, IFCNN, FLGC-FusionGAN) and the effect is good, and the feature points are significantly improved, such as Figure 7 In addition, robot positioning requires not only the number of feature points, but also the consistency of feature points captured at consecutive moments. The number and distribution of image feature points at four consecutive moments in the method of the present invention are shown in Figure 2. Figure 8 Considering the real-time requirements and computational requirements of the robot positioning task, quantitative and qualitative experiments are carried out. The results are shown in Table 1. Figures 9 to 12 It is verified that the result generated by the method of the present invention has a better task completion effect.

[0076] Table 1 Real-time comparison

[0077]

[0078] Example:

[0079] 1) Select the Intel Realsense D435i camera as the image input source, and use the API provided by the Intel Realsense SDK to configure the camera to output infrared stream (infrared image) and RGB stream (low-light image) and control the image capture speed and resolution. The photometer model used is SonbestSM-8763B, and the photometer follows the RS485 communication protocol. Connect the camera and photometer to the computer for communication.

[0080] 2) Start the photometer and camera in the computer, set the photometer working frequency and sampling frequency, and perform image acquisition and photometer data acquisition.

[0081] 3) After collecting the infrared light image, the low-light image and the photometer reading, the program used by the method of the present invention is started.

[0082] 4) Image preprocessing module: First, two-dimensional wavelet image decomposition is used to decompose the infrared image and low-light image to obtain four image subbands. Then, the two high-frequency image subbands are fused to obtain a high-frequency first-level fused image subband, and the two low-frequency image subbands are fused to obtain a low-frequency first-level fused image subband. The Haar wavelet VisuShrink hard threshold is used to denoise the high-frequency first-level fused image subband to effectively remove the salt and pepper noise and Gaussian noise in the image; the low-frequency first-level fused image subband is enhanced by using a combination of a segmentation grid size of (2×2) and a histogram threshold of 1.5, so that the background information of the image is richer, thereby improving the quality of subsequent image fusion.

[0083] 5) Weight module: Before executing the task, use a photometer and a camera to take images with illumination labels in advance, use the optimal weight coefficient fusion process designed by the method of the present invention to obtain the optimal fusion coefficient under a specific illumination, then collect several optimal weight coefficient points, use the neural network regression method to fit the data points into a curve, and then put it into the weight module for calling. Alternatively, the data curve after multi-scene fusion can be used by the present invention and directly put into the weight module for calling.

[0084] 6) Image fusion module: After the input image passes through the image preprocessing module, it will be input into the image fusion module. At the same time, the photometer obtains the illumination information, and then the illumination information will be passed into the weight module to find the best weighting coefficient under the illumination and pass it into the image fusion module. Finally, the image will be fused with the image subband according to their respective weighting coefficients.

[0085] 7) As the robot moves, the camera and photometer will collect data at a synchronized speed. When the photometer senses a change in illumination information, it will pass the information to the weight module again, and then call up a new weighting coefficient to complete the image fusion process and input a new fused image into the robot positioning network.

[0086] 8) Experimental results:

[0087] The data sampling frequency in the experiment is 30Hz. The low-light scenes used in this experiment are indoor low-light scenes, outdoor low-light scenes, and low-light scenes with complex backgrounds, with illumination values ​​between 0-100Lux. The experimental computing platform is configured with 13th Gen Intel(R)Core(TM)i5-13400F, memory 2.50GHz, and built-in 16.00GB RAM. The operating system is Ubuntu18.04, based on python 3.7.4, OpenCV4.7.0.68 and Pywavelets1.7.0. The image fusion effects and algorithm consumption time of DWT, NSST, IFCNN, and FLGC-FusionGAN are compared, and the processing results are objectively evaluated using structural similarity, information entropy, average gradient, and the number of ORB feature points. SSIM is also an indicator for measuring image quality. It takes into account the structural information of the image and is more in line with the perception of the human visual system. The value range of SSIM is usually between -1 and 1. The closer to 1, the more similar the two images are, and the closer to -1, the less similar the two images are. Information entropy is used to measure the information richness after image fusion. After image fusion processing, image blocks with high information entropy may contain more image information. The average gradient reflects the rate of change of contrast of tiny details in the image, that is, the rate of change of density in the multi-dimensional direction of the image, and represents the relative clarity of the image. The number of ORB feature points directly affects the positioning accuracy of the robot and whether it can complete the positioning task. The higher the value of these four objective indicators, the better.

[0088] Combined with Table 1, Figures 9 to 12 As shown, the method of the present invention has advantages over other existing methods in image feature point recovery and time. Some traditional methods can achieve fast fusion, but the effect is not good and the information contained is less. For some fusion methods based on deep learning, their robustness is poor, and poor fusion effect often occurs, or the calculation is too complicated to meet the real-time requirements of robot positioning. The infrared light information fusion method proposed in the method of the present invention makes the best use of the high and low frequency information of the image, so that the fusion effect is excellent.

[0089] From the image fusion results, for scenes with complex illumination and obvious illumination changes, the traditional method has problems such as artifacts, excessive noise, and serious feature loss. The method of the present invention benefits from its adaptive change of weight coefficient, which makes it more robust, and the fused image has a better visual perception and number of feature points. At the same time, the existing methods of other objective evaluation indicators are lower than the method of the present invention. Therefore, comprehensively speaking, the method of the present invention is better than other solutions.

[0090] In summary, the method of the present invention has significant improvements in real-time performance, image feature point recovery rate, and fused image quality compared with other existing methods, and is suitable for robot positioning tasks in low-light scenes.

[0091] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A multi-source image information fusion method for robot positioning in low-light scenes, characterized in that: include: Perform two-dimensional wavelet image decomposition on the collected infrared light image and low-illumination image respectively to obtain infrared light high-frequency image subband, infrared light low-frequency image subband, low-illumination high-frequency image subband and low-illumination low-frequency image subband; fuse the infrared light high-frequency image subband and the low-illumination high-frequency image subband to obtain a high-frequency first-level fused image subband, and then perform threshold processing to obtain a high-frequency fused subband; fuse the infrared light low-frequency image subband and the low-illumination low-frequency image subband to obtain a low-frequency first-level fused image subband, and then perform histogram equalization processing to obtain a low-frequency fused subband; The high-frequency weight coefficient and the low-frequency weight coefficient are selected based on the illumination of the low-illumination scene, and the high-frequency fusion sub-band and the low-frequency fusion sub-band are weightedly fused to obtain the final fused image.

2. The multi-source image information fusion method for low-light scene robot positioning according to claim 1, characterized in that: The threshold processing is Haar wavelet VisuShrink hard threshold processing.

3. The multi-source image information fusion method for low-light scene robot positioning according to claim 1, characterized in that: The histogram equalization process is a combination process of a segmentation grid size of 2×2 and a histogram threshold of 1.

5.

4. The multi-source image information fusion method for low-light scene robot positioning according to claim 1, characterized in that: The method for selecting the high-frequency weight coefficient and the low-frequency weight coefficient based on the illuminance of the low-illuminance scene is to select the high-frequency weight coefficient and the low-frequency weight coefficient based on a preset illuminance-weight coefficient curve.

5. The multi-source image information fusion method for low-light scene robot positioning according to claim 4, characterized in that: The method for obtaining the preset illumination-weight coefficient curve is: In a low-illumination scene, infrared light image samples and low-illumination image samples are collected, and high-frequency fusion sub-band samples and low-frequency fusion sub-band samples are obtained; Set the initial value of the high-frequency weight coefficient ω1 and the initial value of the low-frequency weight coefficient ω2, and iteratively update the high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2: Under the current illumination, the high-frequency fusion sub-band samples and the low-frequency fusion sub-band samples are weightedly fused based on the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 to obtain the current fused image; the feature points of the current fused image are extracted, and the number of current feature points is determined; If the current number of feature points is greater than the number of feature points obtained in the previous round of iteration, the final high-frequency weight coefficient and the final low-frequency weight coefficient are updated, and the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are updated to continue the iteration; otherwise, the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are directly updated to continue the iteration until the iteration termination condition is reached, and the final high-frequency weight coefficient and the final low-frequency weight coefficient are output; The final high-frequency weight coefficient and the final low-frequency weight coefficient corresponding to different illuminations are fitted by a neural network regression method to obtain a preset illumination-weight coefficient curve.

6. The multi-source image information fusion method for low-light scene robot positioning according to claim 5, characterized in that: The low illumination scenes include indoor scenes, outdoor scenes and complex background scenes; in each low illumination scene, a preset illumination-weight coefficient curve is obtained respectively, and then the three preset illumination-weight coefficient curves are fused by using the cubic spline interpolation method to obtain the fused preset illumination-weight coefficient curve.

7. The multi-source image information fusion method for low-light scene robot positioning according to claim 1, characterized in that: The method for selecting high-frequency weight coefficients and low-frequency weight coefficients based on the illumination of low-light scenes is: Set the initial value of the high-frequency weight coefficient ω1 and the initial value of the low-frequency weight coefficient ω2, and iteratively update the high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2: Under the current illumination, the high-frequency fusion subband and the low-frequency fusion subband are weightedly fused based on the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 to obtain the current fused image; the feature points of the current fused image are extracted, and the number of current feature points is determined; If the current number of feature points is greater than the number of feature points obtained in the previous round of iteration, the final high-frequency weight coefficient and the final low-frequency weight coefficient are updated, and the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are updated to continue the iteration; otherwise, the current high-frequency weight coefficient ω1 and the current low-frequency weight coefficient ω2 are directly updated to continue the iteration until the iteration termination condition is reached, and the final high-frequency weight coefficient and the final low-frequency weight coefficient are output.

8. The multi-source image information fusion method for low-light scene robot positioning according to claim 5, 6 or 7, characterized in that: ω1+ω2=1.

9. The multi-source image information fusion method for low-light scene robot positioning according to claim 8, characterized in that: The final fused image is represented as X(x,y): Where (i, j) is the row and column coefficient of the block index corresponding to the image subband, H L (i, j) is the wavelet coefficient of the low-illumination high-frequency image subband, is the wavelet basis function of the low-illumination high-frequency image subband, H I (i, j) is the wavelet coefficient of the infrared high-frequency image subband, is the wavelet basis function of the infrared high-frequency image subband; L L (i, j) is the wavelet coefficient of the low-illumination low-frequency image subband, is the wavelet basis function of the low-illumination low-frequency image subband, L I (i,j) is the wavelet coefficient of the infrared low-frequency image subband, is the wavelet basis function of the infrared low-frequency image subband; where (x, y) is the horizontal and vertical coordinates of the image; in is the high frequency fusion subband, It is the low frequency fusion subband.

10. The multi-source image information fusion method for low-light scene robot positioning according to claim 9, characterized in that: The high-frequency weight coefficient ω1 and the low-frequency weight coefficient ω2 are updated according to a set step size.

Images