Visual positioning method based on dark image fusion and related equipment

Abstract

Embodiments of the present application disclose a visual positioning method based on dark light image fusion and related equipment, the method comprising: acquiring a first image of a to-be-recognized region and a second image of the to-be-recognized region, and determining first brightness information and color difference information of the first image, second brightness information and infrared data of the second image; performing dimensional fusion on the first brightness information and the second brightness information to obtain a first fused image; obtaining a second fused image according to the color difference information; performing proportional operation on the second fused image and a historical image, calibrating operation data through the infrared data, dark light enhancing the second image according to the calibrated operation data, and then generating a target image of the to-be-recognized region, and completing visual positioning of the to-be-recognized region based on the target image. Through dark light enhancement of the fused image, image data with better display effect is obtained, and visual positioning can be better performed on the image obtained in a dark light environment.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, and in particular to a visual positioning method and related equipment based on low-light image fusion. Background Technology

[0002] Visual localization is one of the key technologies for robots to achieve autonomous obstacle avoidance and localization. Visual localization robots collect data through camera sensors and then use visual recognition and localization to detect, identify, and locate obstacles. Typical visual localization robots employ visual recognition algorithms such as contour matching-based target recognition and deep neural network-based instance segmentation and target recognition algorithms. However, when robots perform special tasks or in extreme weather conditions, such as fire rescue or heavy fog and rain, data from camera sensors alone is often insufficient. Ambient lighting can make obstacle outlines or features less distinct, ultimately reducing the effectiveness of target recognition.

[0003] Currently, many researchers using deep learning-based visual localization technologies are attempting to improve the performance of robots in target recognition and localization under harsh environments such as low light, heavy rain, and fog, by employing low-light enhancement or dehazing algorithms. However, due to the limited data acquired by camera sensors, these algorithms have relatively limited application environments and low robustness. In existing technologies, other researchers use infrared cameras to obtain infrared data, fusing this data with RGB data acquired by the camera to increase the amount of environmental data collected, ultimately improving the accuracy of visual recognition.

[0004] In complex environments (such as high noise, low light, heavy fog, heavy rain, etc.), visual positioning is limited when using a single sensor, while the image outline after image fusion technology is still blurry when using multiple sensors, resulting in a decrease in the robot's own visual positioning accuracy. Summary of the Invention

[0005] In view of this, the present invention provides a visual positioning method and related equipment based on low-light image fusion, which solves the problem that the outline of the image after image fusion is still blurry in the prior art, resulting in a decrease in the visual positioning accuracy of the robot itself.

[0006] To achieve one or more of the above objectives or other objectives, in a first aspect, the present invention proposes a visual positioning method based on low-light image fusion, comprising: acquiring a first image captured by a camera device for a region to be identified and a second image captured by an infrared sensor device for the region to be identified, and determining first brightness information of the first image, color difference information of the first image, second brightness information of the second image and infrared data of the second image.

[0007] The first brightness information and the second brightness information are dimensionally fused to obtain a first fused image, and the first fused image is rendered according to the color difference information to obtain a second fused image;

[0008] The second fused image and the historical image obtained after dark enhancement processing are proportionally calculated to obtain calculation data. The calculation data is then calibrated using the infrared data. Based on the calibrated calculation data, the second image is enhanced in dark light to obtain target enhancement data.

[0009] A target image of the region to be identified is generated based on the target augmentation data, and the visual localization of the region to be identified is completed based on the target image.

[0010] Optionally, the step of determining the first brightness information and the color difference information of the first image includes:

[0011] The initial image format of the first image is determined based on the basic attribute information of the first image, and it is determined whether the initial image format is the target image format;

[0012] If the initial image format is the target image format, then the first image of the target image format is identified according to the information storage rules of the target image format to obtain the first brightness information and the color difference information of the first image.

[0013] If the initial image format is not the target image format, then the target conversion relationship between the initial image format and the target image format is determined according to the preset format conversion table, and the first image in the initial image format is converted into the target image in the target image format through the target conversion relationship. The preset format conversion table includes the conversion relationship between different image formats.

[0014] The target image is identified according to the information storage rules of the target image format to obtain the first brightness information and the color difference information of the first image.

[0015] Optionally, the step of determining the infrared data of the second image includes:

[0016] The second image is calibrated using a calibration function to obtain initial data;

[0017] The initial data is input into a preset parameterized network for assignment processing to obtain the infrared data, which contains a preset calibration value corresponding to the preset parameterized network.

[0018] Optionally, the step of dimensionally fusing the first brightness information and the second brightness information to obtain a first fused image includes:

[0019] The first brightness information and the second brightness information are dimensionally fused to obtain initial fused data;

[0020] The initial fusion data is input into a preset fusion network to obtain target fusion data. The preset fusion network includes at least five convolutional neural layers, and the convolutional neural layers are connected in a feedforward manner.

[0021] The target fusion data is processed by an activation function to obtain activation data, and the first fusion image is obtained based on the activation data.

[0022] Optionally, the step of performing a scaling operation on the second fused image and the historical image obtained after dark enhancement processing to obtain the calculated data includes:

[0023] Acquire historical images after the aforementioned low-light enhancement processing;

[0024] The historical image is divided pixel by pixel by the second fused image to obtain the computational data.

[0025] Optionally, the step of calibrating the computational data using the infrared data and performing low-light enhancement on the second image based on the calibrated computational data to obtain target enhancement data includes:

[0026] The infrared data and the calculated data are added pixel by pixel to obtain the calibrated calculated data;

[0027] The calibrated computational data is then dimensionally concatenated with the second image to obtain the concatenated data.

[0028] The stitched data is input into a preset brightness adaptive adjustment network to obtain the target enhanced data. The preset brightness adaptive adjustment network implements the brightness adaptive adjustment algorithm through at least three layers of convolutional neural networks.

[0029] Optionally, the step of generating a target image of the region to be identified based on the target augmentation data includes:

[0030] The calibrated computational data is added pixel by pixel to the target enhancement data to obtain target enhancement data containing the calibrated computational data;

[0031] A target image of the region to be identified is generated based on target augmentation data containing calibrated computational data.

[0032] Secondly, this application provides a visual positioning device based on low-light image fusion, the device comprising:

[0033] The data acquisition module is used to acquire a first image captured by the camera device for the area to be identified and a second image captured by the infrared sensor device for the area to be identified, and to determine the first brightness information of the first image, the color difference information of the first image, the second brightness information of the second image, and the infrared data of the second image.

[0034] The fusion module is used to perform dimensional fusion of the first brightness information and the second brightness information to obtain a first fused image, and to render the first fused image according to the color difference information to obtain a second fused image;

[0035] An enhancement module is used to perform a ratio calculation on the second fused image and the historical image obtained after dark light enhancement processing to obtain calculation data, and to calibrate the calculation data using the infrared data. Based on the calibrated calculation data, the second image is then enhanced in dark light to obtain target enhancement data.

[0036] The localization module is used to generate a target image of the region to be identified based on the target augmentation data, and to complete the visual localization of the region to be identified based on the target image.

[0037] Thirdly, this application provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and when the machine-readable instructions are executed by the processor, the steps of the visual localization method based on dark light image fusion as described above are performed.

[0038] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the visual localization method based on dark-light image fusion as described above.

[0039] Implementing the embodiments of the present invention will have the following beneficial effects:

[0040] By acquiring a first image of the area to be identified by a camera device and a second image of the area to be identified by an infrared sensor device, and determining the first brightness information, color difference information, second brightness information, and infrared data of the second image, the first brightness information and the second brightness information are dimensionally fused to obtain a first fused image. The first fused image is then rendered based on the color difference information to obtain a second fused image. A proportional calculation is performed between the second fused image and a historical image obtained through dark light enhancement processing to obtain computational data. This computational data is then calibrated using the infrared data. Based on the calibrated computational data, the second image is subjected to dark light enhancement to obtain target enhancement data. A target image of the area to be identified is generated based on the target enhancement data, and visual positioning of the area to be identified is completed based on the target image. Combining image fusion with dark light enhancement makes the image fused by image fusion technology clearer in complex environments, such as noisy, low-light, foggy, or rainy scenes, improving the accuracy of visual positioning in complex environments. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] in:

[0043] Figure 1 This is a scene diagram of a visual positioning method based on low-light image fusion provided in an embodiment of this application;

[0044] Figure 2 This is a flowchart of a visual localization method based on low-light image fusion provided in an embodiment of this application;

[0045] Figure 3 This is an operational diagram of a visual localization method based on low-light image fusion provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of a visual positioning device based on low-light image fusion provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a storage medium provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The visual localization method based on low-light image fusion provided in this invention can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server via a network. The server can acquire a first image of the area to be identified captured by the client through a camera device and a second image of the area to be identified captured by the client through an infrared sensor device. The server determines the first brightness information, the first color difference information, the second brightness information, and the infrared data of the second image. It then performs dimensional fusion of the first and second brightness information to obtain a first fused image, and renders the first fused image based on the color difference information to obtain a second fused image. The server performs a proportional calculation between the second fused image and a historical image obtained through dark light enhancement processing to obtain computational data. This computational data is then calibrated using the infrared data. Based on the calibrated computational data, the second image undergoes dark light enhancement to obtain target enhancement data. A target image of the area to be identified is generated based on the target enhancement data, and visual positioning of the area to be identified is completed based on the target image. The visual positioning result is sent to the client. Alternatively, the server generates control commands based on the visual recognition results and controls the client based on the control commands. This invention combines image fusion with low-light enhancement, resulting in clearer images in complex environments such as noisy, low-light, foggy, or rainy conditions, thus improving the accuracy of visual positioning in such environments. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices equipped with multiple sensors that collect data. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0051] To alleviate the computational burden on the server, the visual localization method based on low-light image fusion provided in this embodiment of the invention can also be applied to... Figure 1The client in the process acquires a first image captured by a camera device targeting the area to be identified and a second image captured by an infrared sensor targeting the area to be identified. It then determines first brightness information, color difference information, second brightness information, and infrared data of the second image. The client performs dimensional fusion of the first and second brightness information to obtain a first fused image, and renders the first fused image based on the color difference information to obtain a second fused image. It performs a proportional calculation between the second fused image and a historical image obtained through dark light enhancement processing to obtain computational data, and calibrates the computational data using the infrared data. Based on the calibrated computational data, it performs dark light enhancement on the second image to obtain target enhancement data. Based on the target enhancement data, it generates a target image of the area to be identified, and completes visual positioning of the area to be identified based on the target image. The visual positioning result is then sent to the client's display device, or a control command is generated based on the visual recognition result, and other devices on the client are controlled based on the control command.

[0052] like Figure 2 As shown, this application provides a visual localization method based on low-light image fusion, including:

[0053] S101. Acquire a first image captured by the camera device for the area to be identified and a second image captured by the infrared sensor device for the area to be identified, and determine the first brightness information of the first image, the color difference information of the first image, the second brightness information of the second image and the infrared data of the second image.

[0054] For example, as the application of visual positioning technology becomes more widespread, the requirements for the results of visual positioning are also becoming more accurate. Since the data obtained by the camera sensor is limited, the basic data required for visual positioning has been supplemented with infrared images collected by infrared sensing devices, i.e., the second image. By acquiring the infrared image collected by the region to be identified, the pixel image collected by the camera device for the region to be identified, i.e., the first image, is supplemented with data. This allows more of the corresponding data of the region to be identified to be added to the visual positioning process, thereby taking into account the actual situation of the region to be identified more comprehensively and obtaining a more accurate visual positioning result.

[0055] For example, since the infrared image essentially represents the brightness information of each part of the area to be identified, and the first image captured by the camera device for the area to be identified can also reflect the brightness information of each part of the area to be identified, the brightness parameter is used as a bridge when the first image and the second image are fused.

[0056] S102. The first brightness information and the second brightness information are dimensionally fused to obtain a first fused image, and the first fused image is rendered according to the color difference information to obtain a second fused image.

[0057] For example, image fusion refers to combining image data about the same target collected from multiple sources (i.e., the first image and the second image) through image processing and computer technology to extract the most useful information from each source and finally synthesize them into a high-quality image. This improves the utilization rate of image information, enhances the accuracy and reliability of computer interpretation, and increases the spatial and spectral resolution of the original image, thus facilitating monitoring.

[0058] For example, by color space conversion and luminance channel replacement, the first luminance information and the second luminance information are dimensionally fused to obtain a first fused image. The first fused image is then rendered according to the color difference information by color space inverse transformation to obtain a second fused image.

[0059] S103. Perform a ratio calculation on the second fused image and the historical image obtained after dark light enhancement processing to obtain calculation data, and calibrate the calculation data using the infrared data. Based on the calibrated calculation data, perform dark light enhancement on the second image to obtain target enhancement data.

[0060] For example, the process of performing dark enhancement on the second fused image is preprocessed using the ratio data between the second fused image and the historical image obtained after dark enhancement processing, so that the dark enhancement process on the second fused image is more accurate.

[0061] For example, calibrating the computational data using the infrared data ensures the reliability of the preprocessing process and avoids excessive deviation in the preprocessing process, which could lead to errors when enhancing the second fused image in low light.

[0062] S104. Generate a target image of the region to be identified based on the target enhancement data, and complete the visual localization of the region to be identified based on the target image.

[0063] By acquiring a first image of the area to be identified by a camera and a second image of the area to be identified by an infrared sensor, and determining the first brightness information, color difference information, second brightness information, and infrared data of the second image, the first brightness information and the second brightness information are dimensionally fused to obtain a first fused image. The first fused image is then rendered based on the color difference information to obtain a second fused image. A proportional calculation is performed between the second fused image and a historical image obtained through dark light enhancement processing to obtain computational data. This computational data is then calibrated using the infrared data. Based on the calibrated computational data, the second image is dark-light enhanced to obtain target enhancement data. A target image of the area to be identified is generated based on the target enhancement data, and visual positioning of the area to be identified is completed based on the target image. This cascaded approach combines image fusion with dark light enhancement, resulting in clearer images in complex environments, such as noisy, low-light, foggy, or rainy conditions, thus improving the accuracy of visual positioning in complex environments.

[0064] In one possible implementation, the step of determining the first brightness information and the color difference information of the first image includes:

[0065] The initial image format of the first image is determined based on the basic attribute information of the first image, and it is determined whether the initial image format is the target image format;

[0066] If the initial image format is the target image format, then the first image of the target image format is identified according to the information storage rules of the target image format to obtain the first brightness information and the color difference information of the first image.

[0067] If the initial image format is not the target image format, then the target conversion relationship between the initial image format and the target image format is determined according to the preset format conversion table, and the first image in the initial image format is converted into the target image in the target image format through the target conversion relationship. The preset format conversion table includes the conversion relationship between different image formats.

[0068] The target image is identified according to the information storage rules of the target image format to obtain the first brightness information and the color difference information of the first image.

[0069] For example, since different camera devices store acquired images in different image formats, such as Joint Photographic Experts Group (JPG) format, Red-Green-Blue (RGB) format, YCrCb color coding format, and Color Model (LAB) format, in order to ensure the wide applicability of the method described in this application, different image formats are converted into a target image format. For example, YCrCb format is used as the target image format. When the initial image format of the first image is RGB format, the conversion relationship between RGB format and YCrCb format is determined by a preset format conversion table, and the RGB format is converted into YCrCb format. The YCrCb format information storage rule is that the Y channel carries the brightness information of the image, and the CrCb channel carries the color difference information of the image. Therefore, the first brightness information and the color difference information of the first image are read through the YCrCb format information storage rule.

[0070] For example, converting RGB format to YCrCb format, for instance:

[0071] Y=0.2126*R+0.7152*G+0.0722*B

[0072] Cr=1.4749*(0.2213*R-0.3390*G+0.1177*B)+128

[0073] Cb=0.6245*(0.1949*R+0.6057*G-0.8006*B)+128.

[0074] In one possible implementation, the step of determining the infrared data of the second image includes:

[0075] The second image is calibrated using a calibration function to obtain initial data;

[0076] The initial data is input into a preset parameterized network for assignment processing to obtain the infrared data, which contains a preset calibration value corresponding to the preset parameterized network.

[0077] For example, the second image is subjected to A / D conversion. The A / D conversion process is the process of converting analog quantities or continuously changing quantities into corresponding digital quantities to obtain the original digital image. A target matrix is ​​constructed based on the original digital image, and filter weight parameters are constructed based on the target matrix. Then, an image non-uniformity modeling formula is obtained based on the filter weight parameters, and gain correction parameters and bias correction parameters are obtained respectively. The gain correction parameters and bias correction parameters are substituted into the non-uniformity correction function to obtain the initial data through the non-uniformity correction function.

[0078] For example, the initial data is input into a preset parameterization network, which adds a preset calibration value to the initial data and outputs infrared data containing the preset calibration value.

[0079] In one possible implementation, the step of dimensionally fusing the first brightness information and the second brightness information to obtain a first fused image includes:

[0080] The first brightness information and the second brightness information are dimensionally fused to obtain initial fused data;

[0081] The initial fusion data is input into a preset fusion network to obtain target fusion data. The preset fusion network includes at least five convolutional neural layers, and the convolutional neural layers are connected in a feedforward manner.

[0082] The target fusion data is processed by an activation function to obtain activation data, and the first fusion image is obtained based on the activation data.

[0083] For example, the first brightness information and the second brightness information are dimensionally fused to obtain initial fused data; the initial fused data is input into a DenseNet network composed of 7 layers of convolutional neural networks, the DenseNet network being a convolutional neural network with dense connections, to obtain target fused data, and then activated by the Tanh activation function derived from hyperbolic sine and hyperbolic cosine to output the fused image, i.e., the first fused image.

[0084] For example, the first fused image is only an image representing brightness information. Therefore, the chromatic difference information of the image carried by the CrCb channel of the first image is dimensionally fused with the first fused image to obtain the second fused image.

[0085] For example, the second fused image is in YCrCb format. To facilitate the subsequent low-light enhancement process, the YCrCb format of the second fused image is converted to RGB format.

[0086] In one possible implementation, the step of performing a scaling operation on the second fused image and the historical image obtained after dark enhancement processing to obtain the calculated data includes:

[0087] Acquire historical images after the aforementioned low-light enhancement processing;

[0088] The historical image is divided pixel by pixel by the second fused image to obtain the computational data.

[0089] For example, the second fused image is divided pixel by pixel with the historical image after the dark light enhancement processing, forming a cyclic whole by using the idea of ​​looping.

[0090] In one possible implementation, the step of calibrating the computational data using the infrared data, and then performing low-light enhancement on the second image based on the calibrated computational data to obtain target enhancement data includes:

[0091] The infrared data and the calculated data are added pixel by pixel to obtain the calibrated calculated data;

[0092] The calibrated computational data is then dimensionally concatenated with the second image to obtain the concatenated data.

[0093] The stitched data is input into a preset brightness adaptive adjustment network to obtain the target enhanced data. The preset brightness adaptive adjustment network implements the brightness adaptive adjustment algorithm through at least three layers of convolutional neural networks.

[0094] For example, the infrared data and the computational data are added pixel by pixel, and the computational data is calibrated to ensure the reliability of the preprocessing of the preset brightness adaptive adjustment network after calibration. The stitched data of the second image and the calibrated computational data are used as the input to the dark light enhancement process. The dark light enhancement process is completed by the preset brightness adaptive adjustment network, which is a preset network that implements the brightness adaptive adjustment algorithm with at least three layers of convolutional neural networks.

[0095] In one possible implementation, the step of generating a target image of the region to be identified based on the target enhancement data includes:

[0096] The calibrated computational data is added pixel by pixel to the target enhancement data to obtain target enhancement data containing the calibrated computational data;

[0097] A target image of the region to be identified is generated based on target augmentation data containing calibrated computational data.

[0098] For example, in order to ensure that the quality of the image after low-light enhancement is not inferior to that of the image without low-light enhancement, the concept of residual is used to add the calibrated computational data to the target enhancement data pixel by pixel to obtain target enhancement data containing the calibrated computational data, and to generate the target image of the region to be identified.

[0099] In one possible implementation, such as Figure 3As shown, this application provides a deep learning framework algorithm for a visual localization method based on low-light image fusion. The framework algorithm includes an image fusion algorithm and a low-light enhancement algorithm. The image fusion algorithm includes the following steps: receiving an RGB image output from a camera and an infrared image output from an infrared camera; converting the RGB image into a YCrCb image format, where the Y channel carries the brightness information of the image, and the infrared image itself contains brightness information; performing dimensional fusion on the brightness information of the input RGB image and infrared image and inputting it into a DenseNet network composed of 7 layers of convolutional neural networks; then activating it through the Tanh activation function and outputting the fused image, which is equivalent to the first fused image mentioned above; since the fused image only represents brightness information, the Cr and Cb image information in the RGB image is dimensionally fused with the fused image to obtain a second fused image, which is equivalent to the second fused image mentioned above.

[0100] The low-light enhancement algorithm operates by: combining the fused image Y with the previously enhanced image X. t This is equivalent to performing a pixel-by-pixel division operation on the aforementioned historical image data, resulting in the previously enhanced image X. t Set the original input visible light image; take the input infrared image G, which is equivalent to the second image mentioned above, and apply the calibration function Z. t The data after (x) is input into the calibration module. The calibration module is a parameterized network used to add a calibration value to the input data and output infrared data S containing the calibration value. t ; calibrated infrared data S t V is obtained by adding the data obtained from the previous division operation pixel by pixel. t This is equivalent to the calibrated calculation data mentioned above, and then V t After performing a dimensional stitching operation with the input infrared image G, it is used as a low-light enhancement module. The input image is a low-light enhancement module consisting of a 3-layer convolutional neural network that uses an adaptive brightness adjustment algorithm. Through parameter learning, it optimizes the brightness information of the input image and outputs an enhanced image, U. t Finally, to ensure that the image quality after low-light enhancement is not inferior to the image data before the low-light enhancement module, the concept of residuals is used to adjust V... t Compared with the enhanced image U t The final enhanced image X is obtained by performing pixel-by-pixel addition. t+1 Using a cyclical approach, the enhanced image X... t+1 X, as the next dark light enhancement operation t This forms a complete cycle.

[0101] In one possible implementation, such as Figure 4As shown, this application provides a visual positioning device based on low-light image fusion, the device comprising:

[0102] The data acquisition module 201 is used to acquire a first image captured by the camera device for the area to be identified and a second image captured by the infrared sensor device for the area to be identified, and to determine the first brightness information of the first image, the color difference information of the first image, the second brightness information of the second image and the infrared data of the second image;

[0103] The fusion module 202 is used to perform dimensional fusion of the first brightness information and the second brightness information to obtain a first fused image, and to render the first fused image according to the color difference information to obtain a second fused image;

[0104] The enhancement module 203 is used to perform a ratio calculation on the second fused image and the historical image obtained after dark light enhancement processing to obtain calculation data, and to calibrate the calculation data through the infrared data. Based on the calibrated calculation data, the second image is enhanced in dark light to obtain target enhancement data.

[0105] The positioning module 204 is used to generate a target image of the region to be identified based on the target enhancement data, and to complete the visual positioning of the region to be identified based on the target image.

[0106] By combining image fusion with low-light enhancement, the image fused by the image fusion technology is displayed more clearly in complex environments, such as noisy, low-light, foggy, or rainy scenes, thus improving the accuracy of visual positioning in complex environments.

[0107] One possible implementation, such as Figure 5As shown, this application embodiment provides an electronic device 300, including: a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it performs the following: acquiring a first image captured by a camera device for a region to be identified and a second image captured by an infrared sensor for the region to be identified, and determining first brightness information, color difference information, second brightness information, and infrared data of the second image; performing dimensional fusion of the first brightness information and the second brightness information to obtain a first fused image, and rendering the first fused image according to the color difference information to obtain a second fused image; performing a proportional calculation on the second fused image and a historical image obtained through dark light enhancement processing to obtain calculation data, and calibrating the calculation data using the infrared data; performing dark light enhancement on the second image according to the calibrated calculation data to obtain target enhancement data; generating a target image of the region to be identified based on the target enhancement data, and completing visual positioning of the region to be identified based on the target image.

[0108] By combining image fusion with low-light enhancement, the image fused by the image fusion technology is displayed more clearly in complex environments, such as noisy, low-light, foggy, or rainy scenes, thus improving the accuracy of visual positioning in complex environments.

[0109] In one possible implementation, such as Figure 6 As shown, this application embodiment provides a computer-readable storage medium 400 storing a computer program 411. When executed by a processor, the computer program 411 performs the following: acquiring a first image captured by a camera device for a region to be identified and a second image captured by an infrared sensor for the region to be identified; determining first brightness information, color difference information, second brightness information, and infrared data of the second image; performing dimensional fusion of the first brightness information and the second brightness information to obtain a first fused image; rendering the first fused image according to the color difference information to obtain a second fused image; performing a proportional calculation on the second fused image and a historical image obtained through dark light enhancement processing to obtain calculation data; calibrating the calculation data using the infrared data; performing dark light enhancement on the second image according to the calibrated calculation data to obtain target enhancement data; generating a target image of the region to be identified based on the target enhancement data; and completing the visual positioning of the region to be identified based on the target image.

[0110] By combining image fusion with low-light enhancement, the image fused by the image fusion technology is displayed more clearly in complex environments, such as noisy, low-light, foggy, or rainy scenes, thus improving the accuracy of visual positioning in complex environments.

[0111] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0112] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0113] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0114] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0116] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

[0117] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A visual localization method based on low-light image fusion, characterized in that, include: Acquire a first image captured by a camera device for the area to be identified and a second image captured by an infrared sensor for the area to be identified, and determine the first brightness information, the color difference information, the second brightness information, and the infrared data of the second image of the first image; The step of determining the first brightness information and the color difference information of the first image includes: The initial image format of the first image is determined based on the basic attribute information of the first image, and it is determined whether the initial image format is the target image format. If the initial image format is the target image format, then the first image of the target image format is identified according to the information storage rules of the target image format to obtain the first brightness information and the color difference information of the first image. If the initial image format is not the target image format, then the target conversion relationship between the initial image format and the target image format is determined according to the preset format conversion table, and the first image in the initial image format is converted into the target image in the target image format through the target conversion relationship. The preset format conversion table includes the conversion relationship between different image formats. The target image is identified according to the information storage rules of the target image format to obtain the first brightness information and the color difference information of the first image. The first brightness information and the second brightness information are dimensionally fused to obtain a first fused image, and the first fused image is rendered according to the color difference information to obtain a second fused image; The step of dimensionally fusing the first brightness information and the second brightness information to obtain a first fused image includes: The first brightness information and the second brightness information are dimensionally fused to obtain initial fused data; The initial fusion data is input into a preset fusion network to obtain target fusion data. The preset fusion network includes at least five convolutional neural layers, and the convolutional neural layers are connected in a feedforward manner. The target fusion data is processed by an activation function to obtain activation data, and the first fusion image is obtained based on the activation data. The second fused image and the historical image obtained after dark enhancement processing are proportionally calculated to obtain calculation data. The calculation data is then calibrated using the infrared data. Based on the calibrated calculation data, the second image is enhanced in dark light to obtain target enhancement data. The step of calibrating the computational data using the infrared data, and then performing low-light enhancement on the second image based on the calibrated computational data to obtain target enhancement data includes: The infrared data and the calculated data are added pixel by pixel to obtain the calibrated calculated data; The calibrated computational data is then dimensionally concatenated with the second image to obtain the concatenated data. The spliced ​​data is input into a preset brightness adaptive adjustment network to obtain the target enhanced data. The preset brightness adaptive adjustment network implements the brightness adaptive adjustment algorithm through at least three layers of convolutional neural networks. A target image of the region to be identified is generated based on the target augmentation data, and the visual localization of the region to be identified is completed based on the target image.

2. The visual localization method based on low-light image fusion as described in claim 1, characterized in that, The steps for determining the infrared data of the second image include: The second image is calibrated using a calibration function to obtain initial data; The initial data is input into a preset parameterized network for assignment processing to obtain the infrared data, which contains a preset calibration value corresponding to the preset parameterized network.

3. The visual localization method based on low-light image fusion as described in claim 1, characterized in that, The step of performing a ratio calculation on the second fused image and the historical image obtained after dark enhancement processing to obtain the calculation data includes: Acquire historical images after the aforementioned low-light enhancement processing; The historical image is divided pixel by pixel by the second fused image to obtain the computational data.

4. The visual localization method based on low-light image fusion as described in claim 1, characterized in that, The step of generating a target image of the region to be identified based on the target enhancement data includes: The calibrated computational data is added pixel by pixel to the target enhancement data to obtain target enhancement data containing the calibrated computational data; A target image of the region to be identified is generated based on target augmentation data containing calibrated computational data.

5. A visual positioning device based on low-light image fusion, characterized in that, The device includes: A data acquisition module is used to acquire a first image captured by a camera device for an area to be identified and a second image captured by an infrared sensor for the area to be identified, and to determine the first brightness information, the first color difference information, the second brightness information, and the infrared data of the second image of the first image. The step of determining the first brightness information and the first color difference information of the first image includes: determining the initial image format of the first image based on the basic attribute information of the first image, and determining whether the initial image format is a target image format; if the initial image format is a target image format, then identifying the first image of the target image format according to the information storage rules of the target image format to obtain the first brightness information and the first color difference information of the first image; if the initial image format is not the target image format, then determining the target conversion relationship between the initial image format and the target image format according to a preset format conversion table, and converting the first image of the initial image format into a target image of the target image format through the target conversion relationship, wherein the preset format conversion table includes conversion relationships between different image formats, and identifying the target image according to the information storage rules of the target image format to obtain the first brightness information and the first color difference information of the first image. A fusion module is used to perform dimensional fusion of the first brightness information and the second brightness information to obtain a first fused image, and to render the first fused image according to the color difference information to obtain a second fused image. The step of performing dimensional fusion of the first brightness information and the second brightness information to obtain the first fused image includes: performing dimensional fusion of the first brightness information and the second brightness information to obtain initial fusion data; inputting the initial fusion data into a preset fusion network to obtain target fusion data, wherein the preset fusion network includes at least five convolutional neural layers and the convolutional neural layers are connected in a feedforward manner; processing the target fusion data through an activation function to obtain activation data, and obtaining the first fused image according to the activation data. An enhancement module is used to perform proportional calculations on the second fused image and a historical image obtained through dark light enhancement processing to obtain computational data, and to calibrate the computational data using infrared data. Based on the calibrated computational data, dark light enhancement is performed on the second image to obtain target enhancement data. The step of calibrating the computational data using infrared data and performing dark light enhancement on the second image based on the calibrated computational data to obtain target enhancement data includes: adding the infrared data and the computational data pixel by pixel to obtain calibrated computational data; performing dimensional concatenation of the calibrated computational data and the second image to obtain concatenated data; and inputting the concatenated data into a preset brightness adaptive adjustment network to obtain the target enhancement data. The preset brightness adaptive adjustment network implements a brightness adaptive adjustment algorithm through at least three layers of convolutional neural networks. The localization module is used to generate a target image of the region to be identified based on the target augmentation data, and to complete the visual localization of the region to be identified based on the target image.

6. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the visual localization method based on dark-light image fusion as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the visual localization method based on dark-light image fusion as described in any one of claims 1 to 4.

Images