Visible light near-infrared fusion positioning method and system based on feature consistency

Abstract

The invention provides a visible light near-infrared fusion positioning method and system based on feature consistency, and belongs to the technical field of indoor positioning, and the method comprises the steps: obtaining a visible light image and an original near-infrared image at the same time; obtaining a near-infrared image aligned with the visible light image; obtaining a near-infrared image which is denoised and enhanced; setting a feature extraction threshold, and obtaining a feature point set of the current frame image; judging whether the current external illumination condition is ideal or not; selecting a current input image according to a smooth switching principle; performing feature matching; and realizing autonomous positioning according to a feature tracking result. According to the invention, an active imaging near-infrared image is introduced to realize feature tracking of illumination robustness, and indoor continuous robust positioning is realized in an over-bright or over-dark extreme illumination environment and a complex change illumination environment.

Description

Technical Field

[0001] The present invention belongs to the field of indoor positioning technology, and in particular relates to a visible light near-infrared fusion positioning method and system based on feature consistency. Background Art

[0002] After years of development, the Beidou satellite navigation system has become a stable and reliable positioning solution. However, in indoor scenarios, due to the influence of Beidou signal obstruction, continuous and effective positioning cannot be achieved by relying solely on Beidou. Therefore, autonomous positioning methods that do not rely on external satellite signals have become the main way to solve indoor problems. Simultaneous Localization and Mapping (SLAM) technology is often used in the field of autonomous positioning, which uses vision, laser and other sensors to achieve perception and positioning tasks. Laser-based positioning methods require lidar. Although their positioning accuracy is higher, they are expensive and large in size. Vision-based positioning methods only require visual sensors, which are low-cost, compact and more practical.

[0003] Existing visual positioning methods are based on visible light images, which mainly use feature point matching methods and optical flow methods for feature extraction and tracking, such as feature extraction and matching based on ORB (Oriented FAST and Rotated BRIEF, the orientation corner feature of the rotation descriptor), and feature tracking based on FAST (Features From Accelerated Segment Test, fast detection corner) corner points and optical flow methods. Both mainstream feature tracking methods are affected by the imaging effect of visible light images. When the external lighting conditions are poor, the image quality is poor, and it is difficult to track features. For example, in extreme lighting such as too dark or too strong, and in scenes with changing lighting, the visible light image will appear too dark or overexposed, resulting in the inability to extract and track features, and thus the inability to position. In order to solve the dependence on visible light, non-visible light images began to be used for visual positioning.

[0004] Existing visual positioning methods based on non-visible light images mainly rely on long-wave infrared imaging. Long-wave infrared image capture The radiation emitted in the wavelength range, the captured image depends on the temperature and is not affected by the ambient lighting, so it is called a "thermal infrared" image. However, its image quality is poor and the noise is strong, making it difficult to track features effectively, so the final positioning accuracy is relatively poor. In addition, it requires the introduction of an additional visual sensor in the positioning system - a long-wave infrared camera, which is expensive and complicated to use, posing a challenge to the actual usability of the system.

[0005] In indoor scenes, due to the severe obstruction of Beidou signals, most Beidou positioning results will be seriously offset, and the traditional method of calibrating the inertial PDR (Pedestrian Dead Reckoning) positioning results based on time-continuous and frequency-stable Beidou position points will be seriously ineffective. The vision-based autonomous positioning method can complete positioning through feature tracking methods, but in extreme lighting and scenes with complex lighting changes, visible light images fail to perform feature tracking. At the same time, current non-visible light visual positioning relies on long-wave infrared images, which not only have poor image quality, but also require the introduction of additional sensors, increasing the complexity of the system; due to the large difference in the frequency bands of long-wave infrared and visible light, the imaging properties are very different, so it is difficult to perform effective fusion to deal with scenes with changing lighting. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a visible light and near-infrared fusion positioning method and system based on feature consistency, introduces near-infrared images in the non-visible light frequency band of active light source imaging, performs feature tracking when the visible light image fails, and realizes visual autonomous positioning in extreme lighting environments; utilizes the imaging principle of a depth camera, simultaneously acquires near-infrared images and visible light images according to a single sensor, and completes a dual-light positioning system according to the two visual images under different lighting conditions; according to the adjacent nature of the near-infrared image and visible light image frequency bands, a cross-modal feature association method based on feature consistency is proposed, thereby correlating the visible light image and the near-infrared image at the feature level, completing continuous feature tracking of different images under different lighting conditions, and realizing robust visual autonomous positioning in complex lighting scenes.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A visible light near-infrared fusion positioning method based on feature consistency includes the following steps:

[0009] Step 1: Based on the RGB-D camera, simultaneously obtain visible light images and the original near infrared image ; RGB-D stands for color-depth;

[0010] Step 2: Complete the prior calibration based on the calibration plate to obtain the external parameters from the near-infrared image to the visible light image , and according to external reference , register each frame of near-infrared image acquired to the visible light image , obtain the visible light image Aligned NIR images ;

[0011] Step 3: For visible light images Aligned NIR images , based on binary mask, fast detection and removal of near infrared images The hollow noise is removed and gamma correction is performed to obtain the denoised and enhanced near-infrared image. ;

[0012] Step 4: Perform deep learning-based feature extraction on the current input image. Set the feature extraction threshold according to the source type of the current input image: a low threshold is used for near-infrared images. , high threshold is used for visible light images , get the feature point set of the current frame image ;

[0013] Step 5: Based on the visible light image , calculate the average pixel brightness and histogram uniformity U, to determine whether the current external lighting conditions are ideal;

[0014] Step 6: According to the principle of smooth switching, select the current input image: when the lighting is ideal, input the visible light image When the lighting is not ideal, the input is the denoised and enhanced near-infrared image ;

[0015] Step 7: According to the deep learning feature matching method, the feature point set of the current frame image is The image and the feature point set of the previous frame image When the two frames of images are of the same modality, single-modality feature tracking is achieved; when the two frames of images are of different modality types, dual-modality feature association is achieved;

[0016] Step 8: Triangulate the spatial map points based on the feature tracking results and calculate the current camera pose. The solution can realize autonomous positioning.

[0017] The present invention also provides a visible light near-infrared fusion positioning system based on feature consistency, comprising the following modules:

[0018] Image acquisition module, based on RGB-D camera, simultaneously acquires visible light images and the original near infrared image ; RGB-D stands for color-depth;

[0019] The registration module completes the prior calibration based on the calibration plate and obtains the external parameters from the near-infrared image to the visible light image. , and according to external reference , register each frame of near-infrared image acquired to the visible light image , obtain the visible light image Aligned NIR images ;

[0020] Denoising enhancement module for visible light images Aligned NIR images , based on binary mask, fast detection and removal of near infrared images The hollow noise is removed and gamma correction is performed to obtain the denoised and enhanced near-infrared image. ;

[0021] The collection acquisition module performs deep learning-based feature extraction on the current input image and sets the feature extraction threshold according to the current input image source type: a low threshold is used for near-infrared images , high threshold is used for visible light images , get the feature point set of the current frame image ;

[0022] Light judgment module, based on visible light image , calculate the average pixel brightness and histogram uniformity U, to determine whether the current external lighting conditions are ideal;

[0023] Smooth switching module, according to the principle of smooth switching, selects the current input image: when the lighting is ideal, inputs the visible light image When the lighting is not ideal, the input is the denoised and enhanced near-infrared image ;

[0024] The association module uses the deep learning feature matching method to match the feature point set of the current frame image. The image and the feature point set of the previous frame image When the two frames of images are of the same modality, single-modality feature tracking is achieved; when the two frames of images are of different modality types, dual-modality feature association is achieved;

[0025] The positioning module triangulates the spatial map points based on the feature tracking results and calculates the current camera position. The solution can realize autonomous positioning.

[0026] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned visible light near-infrared fusion positioning method based on feature consistency are implemented.

[0027] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned visible light near-infrared fusion positioning method based on feature consistency.

[0028] Beneficial effects:

[0029] 1. Compared with the existing visible light vision indoor positioning method that has the problem of visible light failure in extreme lighting environments, the present invention introduces active imaging of near-infrared images to achieve lighting robust feature tracking, and can also achieve indoor continuous robust positioning in extreme lighting conditions such as too bright or too dark and complex changing lighting environments.

[0030] 2. Compared with the existing long-wave infrared-based non-visible light vision indoor positioning methods, which are faced with problems such as poor image quality, strong noise, low positioning accuracy, the need to introduce additional sensors, and inability to be used with visible light, the present invention is based on a single depth camera. Through its imaging principle, it can obtain visible light and near-infrared dual-light images with only one visual sensor, without the introduction of additional high-cost sensors. It has more universal convenience and versatility. At the same time, near-infrared images have better imaging quality than long-wave infrared images, and the positioning results are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the visible light near infrared fusion positioning method based on feature consistency of the present invention;

[0032] Figure 2 This is a schematic diagram of the principle of the visible light near-infrared fusion positioning method based on feature consistency of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] like Figure 1 As shown, a visible light near-infrared fusion positioning method based on feature consistency in an embodiment of the present invention includes the following steps:

[0035] Step 1: Based on the RGB-D (RGB-Depth, color-depth) camera, simultaneously obtain visible light images and the original near infrared image ,Right now Figure 1 The near-infrared and visible light images are acquired simultaneously.

[0036] Step 2: Complete the prior calibration based on the calibration plate to obtain the external parameters from the near-infrared image to the visible light image , and according to the external parameters calibrated a priori , register each frame of near-infrared image acquired to the visible light image , obtain the visible light image Aligned NIR images ,Right now Figure 1 Near-infrared-visible image registration in .

[0037] Step 3: For the registered visible light image Aligned NIR images , based on binary mask, fast detection and removal of near infrared images The hollow noise is removed and gamma correction is performed to obtain the denoised and enhanced near-infrared image. ,Right now Figure 1 Near infrared image enhancement and denoising.

[0038] Step 4: Perform deep learning-based feature extraction on the current input image. Set the feature extraction threshold according to the source type of the current input image: a low threshold is used for near-infrared images. , high threshold is used for visible light images , get the feature point set of the current frame image ,Right now Figure 1 Dynamic dual threshold feature extraction in .

[0039] Step 5: Based on the visible light image , calculate the average pixel brightness And the histogram uniformity U, to determine whether the current external lighting conditions are ideal, that is, Figure 1 Calculate the external lighting conditions in .

[0040] Step 6: According to the principle of smooth switching, select the current input image: when the lighting is ideal, input the visible light image , when the lighting is not ideal, the input and visible light images Align, denoise and enhance near-infrared images ,Right now Figure 1 Smooth switching images in .

[0041] Step 7: According to the deep learning feature matching method, the feature point set of the current frame image is The image and the feature point set of the previous frame image When the two frames are of the same modality, single-modality feature tracking is achieved; when the two frames are of different modality, dual-modality feature association is achieved. Figure 1 Deep learning feature matching in .

[0042] Step 8: Triangulate the spatial map points based on the feature tracking results and calculate the current camera pose. The solution of can realize autonomous positioning, that is, Figure 1 Pose solution in .

[0043] Among them, steps 2 to 4 constitute a dual-light feature dynamic extraction method, and steps 5 to 7 constitute a dual-light smooth switching method.

[0044] Specifically, the step 1 includes:

[0045] RGB-D cameras can be divided into two imaging modules, one is the RGB image imaging module that relies on external visible light, and the other is the near-infrared imaging module that is equipped with an active near-infrared light source and receives actively emitted near-infrared light. The two are not homologous imaging, but considering that the two are integrated in the same sensor, the position is very close, and the imaging perspective is similar, so the captured scene structure is consistent in features and has feature consistency; the visible light image is obtained through the sensor at the same time and the original near infrared image .

[0046] Specifically, the step 2 includes:

[0047] set up is the spatial coordinate of a point under the near-infrared imaging module coordinates, is the projection coordinate of the point on the image plane, is the internal parameter matrix of the near-infrared imaging module. According to the pinhole imaging model, the three satisfy the following relationship:

[0048] ;

[0049] Also set is the spatial coordinate of the same point in the coordinates of the RGB image imaging module of visible light, is the projection coordinate of the point on the visible light image plane, is the internal parameter matrix of the visible light RGB image imaging module. Since the coordinate system of the near-infrared camera is different from that of the visible light camera, the two can be connected by a rotation and translation transformation, that is, the external parameter matrix between the two for:

[0050] ;

[0051] Among them, the external parameter T is composed of two parts: the rotation matrix R and the translation vector t. Finally, the internal parameter matrix Space coordinates Projection can get the visible light coordinates corresponding to the point ,Right now:

[0052] ;

[0053] Among them, the projection coordinates and visible light coordinates Homogeneous coordinates are used, so when constructing projection coordinates When the original pixel coordinates (x, y) should be multiplied by the depth value, and the final visible light pixel coordinates must be the visible light coordinates Divide by the z component, that is (x / z, y / z), and the value of the z component is the depth of the point to the visible light camera.

[0054] In summary, through external parameter calibration, the external parameter conversion matrix from near-infrared image to visible light image can be obtained , and according to the external parameter conversion matrix , the original near infrared image Convert to visible light image Next, obtain the registered visible light image Aligned NIR images :

[0055] ;

[0056] in, , is the intrinsic parameter matrix of the near-infrared image.

[0057] Specifically, the step 3 includes:

[0058] In order to enhance the performance of near-infrared images in visual tracking and improve the positioning performance, it is necessary to perform denoising and enhancement on the images. The hole noise in near-infrared images is usually distributed in flat areas, which are characterized by significant black areas with pixel values ​​of 0. These noises may lead to errors in feature point extraction due to their significant differences from normal grayscale values. Therefore, a hole noise detection and removal method based on binarization and region growing is designed.

[0059] The input near infrared image is ,in represents the pixel position, and its grayscale range is [0,255]. First, for the near infrared image Perform binarization processing and define a threshold Tm. The near-infrared image above the threshold Tm The near-infrared image is white and below the threshold Tm is black, thus generating a binary mask :

[0060] ;

[0061] Through binary mask , quickly locate potential hollow noise areas in the image. These areas are further processed by the region growing algorithm. All pixels with a value of 1 in the mask are selected as the initial seed point set ,in is the pixel coordinate of the mth point in the point set, and then the seed point set is expanded through the neighborhood growth strategy until a set of all connected pure black areas is generated ,in is the kth pure black area in the set.

[0062] To avoid overfilling, the kth pure black area Area As the judgment criterion, only the area that meets the area threshold condition is retained as the hole noise area. Less than , then continue to keep it, otherwise it will be removed from the pure black area collection Among them, Is the preset area threshold. The set of pure black areas that have been removed That is, the required set of hollow noise areas .

[0063] Get the set of hollow noise regions Finally, in order to achieve smooth pixel filling, a distance-weighted neighbor template filtering method is designed to perform smooth filtering from the outside to the inside. First, a neighbor template is established. , where n is the size of the template, which changes according to the image size. Then traverse the non-black pixels in the template and weight the neighboring pixel values ​​by the inverse of the distance to obtain the target pixel filling pixel value :

[0064] ;

[0065] in, , represents the radius of the template, Is a flag, representing the current pixel value Is it 0, where i and j represent the increments of the horizontal and vertical coordinates of the pixel, respectively, so as to exclude the black pixels and finally obtain the denoised near-infrared image .

[0066] Since the near-infrared image has insufficient brightness and contrast, it is not conducive to grayscale distinction and feature point extraction. After denoising, the near-infrared image is gamma-corrected. It maps the pixel values ​​to the [0,1] interval, quickly improves the brightness and contrast through non-linear changes, and then obtains the denoised and enhanced near-infrared image. :

[0067] ;

[0068] in, is the gamma parameter, and its value is [0,1].

[0069] Specifically, the step 4 includes:

[0070] For the current frame image, the SuperPoint (super feature point) model based on the convolutional neural network is used for automatic feature point extraction. It uses an encoder-decoder architecture, where the encoder realizes the dimensionality reduction and feature extraction of the image through multiple convolutional layers and max pooling layers, and the decoder is responsible for decoding the feature information compressed by the encoder to obtain the required information type; after passing through the SuperPoint model, a series of feature point sets in the current frame image are obtained. , for any feature point in the set , where ii represents the ii-th point in the set, and it is composed of its position in the image pixel coordinate system , a 256 1D descriptor and its feature score to form.

[0071] Considering the feature extraction of bimodal images, where the quality of the near-infrared image is low, the distinguishability is poor, and the feature extraction is difficult. To ensure sufficient and stable feature extraction on both visible light and near-infrared images, a feature point extraction module based on dynamic double thresholds is designed, and high and low thresholds are set according to different image sources , and its value is taken according to the input image type: a high threshold is used for visible light images , and a low threshold is used for near-infrared images , and based on the feature score obtained from the feature position decoder , the feature points greater than the high and low thresholds are retained :

[0072] ;

[0073] ;

[0074] At the same time, considering that the change of the illumination condition will also affect the feature tracking, which is ultimately reflected in the number of successfully tracked features. To prevent the decline of the feature extraction quantity under different situations with a fixed threshold, the threshold is dynamically updated according to the change of the historical feature tracking quantity to improve its feature extraction ability under different illumination conditions.

[0075] Considering that the feature matching quantities of the current K-th frame, the previous K-1 frame, and the previous two K-2 frames are M0, M1, and M2 respectively. If M0 < M1 < M2, it means that the feature matching decays at this time. Therefore, the feature extraction threshold is reduced, and at the same time, a double lowest threshold is set to prevent the confidence of feature point extraction from being too low, and the update process is as follows:

[0076] ;

[0077] in, is a hyperparameter to adjust its dynamic changes, and its value is as follows ;

[0078] The core of the dual-light positioning system is to choose to use near-infrared images or visible light images according to different lighting conditions: visible light images are used under ideal lighting conditions, which have good image quality and high positioning accuracy, and near-infrared images are used under extreme lighting conditions, which do not rely on external visible light. Based on this idea, a dual-light smooth switching method is designed to achieve the use of images under different lighting conditions and continuous feature tracking.

[0079] Specifically, the step 5 includes:

[0080] External light directly affects visible light image formation, so external light can be detected through visible light images. First, calculate the average brightness of the visible light image. and histogram distribution , let the pixel value of the visible light image be , the image size is .

[0081] ;

[0082] ;

[0083] in, Represents the number of pixels with pixel value d.

[0084] When the average brightness of the visible light image Does not meet normal range threshold When , it is directly judged as abnormal lighting, otherwise the histogram uniformity U is calculated, where The lowest value in the normal range. The highest value in the normal range.

[0085] ;

[0086] When U is greater than the uniformity threshold, it is also judged as abnormal lighting; when U is less than the uniformity threshold, it can be judged as normal lighting.

[0087] Specifically, the step 6 includes:

[0088] Acquire visible light images simultaneously And complete denoising and enhancement of near infrared images According to the external light detection result, when the light state changes, the image source is switched.

[0089] When switching from a visible light image to a near-infrared image, the system adheres to the principle of "decisiveness". Once the visible light image does not meet the brightness distribution threshold, it will immediately switch to the near-infrared image. When switching from the near-infrared image to the visible light image, the system adheres to the principle of "hesitation". When the visible light image meets the switching threshold for more than 5 consecutive frames, it will switch to the visible light image.

[0090] Specifically, the step 7 includes:

[0091] SuperGlue is a feature matching graph neural network based on the attention mechanism, which mainly consists of two parts: the attention-based graph neural network and the optimal matching layer. And the feature point set of the previous frame image Perform feature matching.

[0092] For the iiith feature point in the feature point set , whose pixel position is , the descriptor is First, the input pixel position and descriptors The aggregate expression is performed to obtain the input of the iiith feature point of the network :

[0093] ;

[0094] Among them, MLP stands for multi-layer perceptron, and then the input of the iii-th feature point Input to the attention graph neural network. It fully mines the information between feature points by alternating self-attention and cross-attention. The core formula of attention is:

[0095] ;

[0096] ;

[0097] Among them, Softmax is a normalized exponential function, represents the attention weight between the iii-th feature point and the jjj-th feature point, Represents the information after all other feature points are aggregated to the iiith feature point, Represents the input of the iiith feature point The calculated index information, the superscript T indicates the transpose of the matrix, Represents the input of the jjjth feature point The calculated key information, Represents the input of the jjjth feature point Calculated value information. The calculation formula is as follows:

[0098] ;

[0099] ;

[0100] in, , , and , , Represent different weight matrices and biases respectively.

[0101] The information after aggregating all other feature points to the iiith feature point And the input of the iiith feature point When the feature points iii and jjj are from the same image, self-attention is calculated to aggregate the relationship between the feature points in a single image. When the feature points are from two different images, cross-attention is calculated to capture the relationship between key points across images.

[0102] After that, the optimal matching layer is passed to complete the feature point set of the current frame image And the feature point set of the previous frame image Matching results , which is an array, each array element is a two-tuple , where query represents The query feature point in , train is The trainth feature point in the image is matched successfully.

[0103] Specifically, the step 8 includes:

[0104] According to the matching results , triangulate the matching feature points of the two frames to get the spatial point position , and complete the 3D-2D pose solution based on the camera projection model to obtain the current pose.

[0105] The present invention also provides a visible light near-infrared fusion positioning system based on feature consistency, comprising the following modules:

[0106] Image acquisition module, based on RGB-D camera, simultaneously acquires visible light images and the original near infrared image ; RGB-D stands for color-depth;

[0107] The registration module completes the prior calibration based on the calibration plate and obtains the external parameters from the near-infrared image to the visible light image. , and according to external reference , register each frame of near-infrared image acquired to the visible light image , obtain the visible light image Aligned NIR images ;

[0108] Denoising enhancement module for visible light images Aligned NIR images , based on binary mask, fast detection and removal of near infrared images The hollow noise is removed and gamma correction is performed to obtain the denoised and enhanced near-infrared image. ;

[0109] The collection acquisition module performs deep learning-based feature extraction on the current input image and sets the feature extraction threshold according to the current input image source type: a low threshold is used for near-infrared images , high threshold is used for visible light images , get the feature point set of the current frame image ;

[0110] Light judgment module, based on visible light image , calculate the average pixel brightness and histogram uniformity U, to determine whether the current external lighting conditions are ideal;

[0111] Smooth switching module, according to the principle of smooth switching, selects the current input image: when the lighting is ideal, inputs the visible light image When the lighting is not ideal, the input is the denoised and enhanced near-infrared image ;

[0112] The association module uses the deep learning feature matching method to match the feature point set of the current frame image. The image and the feature point set of the previous frame image When the two frames of images are of the same modality, single-modality feature tracking is achieved; when the two frames of images are of different modality types, dual-modality feature association is achieved;

[0113] The positioning module triangulates the spatial map points based on the feature tracking results and calculates the current camera position. The solution can realize autonomous positioning.

[0114] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the above-mentioned visible light near-infrared fusion visual positioning method based on feature consistency in a complex lighting scene are implemented.

[0115] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned visible light near-infrared fusion visual positioning method based on feature consistency in a complex lighting scene.

[0116] like Figure 2 As shown, under normal lighting, the imaging effect of the visible light camera is normal, so the traditional visual positioning method is still applicable. However, under complex lighting, such as extremely bright or extremely dark lighting, the visible light cannot be imaged normally, and the visible light image is invalid, and visual positioning cannot be performed at this time. The present invention introduces near-infrared images, which can still complete imaging under complex lighting, and the near-infrared images are still valid. Through the feature consistency of visible light images and near-infrared images, they are integrated into the positioning system, thereby realizing fusion positioning under complex lighting.

[0117] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0118] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0122] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A visible light near-infrared fusion positioning method based on feature consistency, characterized in that: The steps include: Step 1: Based on the RGB-D camera, simultaneously obtain visible light images and the original near infrared image ; RGB-D stands for color-depth; Step 2: Complete the prior calibration based on the calibration plate to obtain the external parameters from the near-infrared image to the visible light image , and according to external reference , register each frame of near-infrared image acquired to the visible light image , obtain the visible light image Aligned NIR images ; Step 3: For visible light images Aligned NIR images , based on binary mask, fast detection and removal of near infrared images The hollow noise is removed and gamma correction is performed to obtain the denoised and enhanced near-infrared image. ; Step 4: Perform deep learning-based feature extraction on the current input image. Set the feature extraction threshold according to the source type of the current input image: a low threshold is used for near-infrared images. , high threshold is used for visible light images , get the feature point set of the current frame image ; Step 5: Based on the visible light image , calculate the average pixel brightness and histogram uniformity U, to determine whether the current external lighting conditions are ideal; Step 6: According to the principle of smooth switching, select the current input image: when the lighting is ideal, input the visible light image When the lighting is not ideal, the input is the denoised and enhanced near-infrared image ; Step 7: According to the deep learning feature matching method, the feature point set of the current frame image is The image and the feature point set of the previous frame image When the two frames of images are of the same modality, single-modality feature tracking is achieved; when the two frames of images are of different modality types, dual-modality feature association is achieved; Step 8: Triangulate the spatial map points based on the feature tracking results and calculate the current camera pose. The solution can realize autonomous positioning.

2. The visible light near infrared fusion positioning method based on feature consistency according to claim 1 is characterized in that: The step 1 includes: the RGB-D camera is divided into an RGB image imaging module that relies on external visible light and a near-infrared imaging module that is equipped with an active near-infrared light source and receives actively emitted near-infrared light to form an image. The two are integrated in the same sensor, and the captured scene structure has feature consistency; the visible light image is simultaneously acquired through the sensor and the original near infrared image .

3. The visible light near infrared fusion positioning method based on feature consistency according to claim 2 is characterized in that: The step 2 comprises: set up is the spatial coordinate of a point under the near-infrared imaging module coordinates, is the projection coordinate of the point on the image plane, is the internal parameter matrix of the near-infrared imaging module. According to the pinhole imaging model, the three satisfy the following relationship: ； set up is the spatial coordinate of the same point in the coordinates of the RGB image imaging module of visible light, is the projection coordinate of the point on the visible light image plane, is the internal parameter matrix of the visible light RGB image imaging module, the projection coordinate With space coordinates External reference The calculation formula is: ； Among them, external reference It is composed of the rotation matrix R and the translation vector t; Using the internal reference matrix Space coordinates Projection to obtain the visible light coordinates corresponding to the point ; Obtain the extrinsic conversion matrix from near-infrared image to visible light image through extrinsic calibration , and according to the external parameter conversion matrix The original near infrared image Convert to visible light image Next, the registered visible light image is obtained. Aligned NIR images : ； in, .

4. The visible light near infrared fusion positioning method based on feature consistency according to claim 1 is characterized in that: The step 3 comprises: The input near infrared image is ,in Represents the pixel position, and its grayscale range is [0,255]; Near infrared images Perform binarization processing and define a threshold Tm. The near-infrared image above the threshold Tm The near-infrared image is white and below the threshold Tm is black, thus generating a binary mask ; Through binary mask Quickly locate potential hole noise areas in images; Select all pixels with a value of 1 in the binary mask as the initial seed point set ,in is the pixel coordinate of the mth point in the point set, and then the seed point set is expanded through the neighborhood growth strategy until a set of all connected pure black areas is generated ,in is the kth pure black area in the pure black area set; The kth pure black area Area As the judgment criterion, only the area that meets the area threshold condition is retained as the hole noise area. Less than , then continue to retain the pure black area, otherwise remove the pure black area from the pure black area set Among them, is the preset area threshold; the set of pure black areas that have been removed That is, the required set of hollow noise areas ; Get the set of hollow noise regions Finally, a distance-weighted neighbor template filtering method is used to perform smoothing filtering from the outside to the inside. First, a neighbor template is established. , where n is the size of the template; then traverse the non-black pixels in the template and weight the neighboring pixel values ​​by the inverse of the distance to obtain the target pixel filling pixel value : ； in, , represents the radius of the template, Is a flag, representing the current pixel value Whether it is 0, i and j represent the increment of the horizontal and vertical coordinates of the pixel, thereby excluding black pixels and finally obtaining a denoised near-infrared image ; The denoised near infrared image Gamma correction is performed to map pixel values ​​to the [0,1] interval, quickly improving brightness and contrast, and ultimately obtaining a denoised and enhanced near-infrared image. .

5. The visible light near infrared fusion positioning method based on feature consistency according to claim 1 is characterized in that: The step 4 comprises: For the current frame image, the SuperPoint model based on convolutional neural network is used to automatically extract feature points. It uses an encoder-decoder architecture, in which the encoder is composed of multiple convolutional layers and maximum pooling layers to achieve image dimensionality reduction and feature extraction, and the decoder decodes the feature information compressed by the encoder to obtain the required information type; after passing through the SuperPoint model, a series of feature point sets in the current frame image are obtained. , for any feature point in the set , are determined by their positions in the image pixel coordinate system , a 256 1-dimensional descriptor and its feature scores Composition; where ii represents the ii-th point in the set; Design a feature point extraction module based on dynamic dual thresholds, and set high and low thresholds according to different image sources , high and low thresholds Include high threshold and low threshold , according to the input image type: visible light images use high threshold , near infrared images use a low threshold , and based on the feature scores obtained by the feature position decoder , keep the value greater than the upper and lower thresholds Feature points ; Tracking changes in quantity based on historical features to set high and low thresholds Perform dynamic updates; The dual-light smooth switching method is used to achieve the use of images under different illumination and continuous feature tracking.

6. The visible light near infrared fusion positioning method based on feature consistency according to claim 1 is characterized in that: The step 5 comprises: First, calculate the average brightness of the visible light image and histogram distribution , let the pixel value of the visible light image be , the image size is ; ； ； in, Represents the number of pixels with pixel value d; represents the pixel position; When the average brightness of the visible light image Does not meet normal range threshold When , it is directly judged as abnormal lighting, otherwise continue to calculate the histogram uniformity U, where The lowest value in the normal range, The highest value in the normal range; ； When the histogram uniformity U is greater than the uniformity threshold, it is judged as abnormal lighting; when the histogram uniformity U is less than the uniformity threshold, it is judged as normal lighting.

7. The visible light near infrared fusion positioning method based on feature consistency according to claim 1 is characterized in that: The step 6 comprises: Acquire visible light images simultaneously And complete denoising and enhancement of near infrared images ; According to the external light detection results, when the light state changes, the image source is switched; When switching from a visible light image to a near-infrared image, the "decisive" principle is followed. Once the visible light does not meet the brightness distribution threshold, the image is immediately switched to the near-infrared image. When switching from a near-infrared image to a visible light image, the "hesitant" principle is followed. When the visible light image meets the switching threshold for more than 5 consecutive frames, the image is switched to the visible light image.

8. The visible light near infrared fusion positioning method based on feature consistency according to claim 1 is characterized in that: The step 7 comprises: The attention-based feature matching graph neural network SuperGlue is used, which includes an attention-based graph neural network and an optimal matching layer; the feature point set of the current frame image is And the feature point set of the previous frame image Perform feature matching; For the iiith feature point in the feature point set , whose pixel position is , the descriptor is ; First, the pixel position and descriptors Through the multi-layer perceptron, the input of the iiith feature point is obtained. , and then the input of the iiith feature point The input is sent to the feature matching graph neural network SuperGlue based on the attention mechanism, which fully mines the information between feature points by alternating self-attention and cross-attention. The core formula of attention is: ； ； Among them, Softmax is a normalized exponential function, represents the attention weight between the iii-th feature point and the jjj-th feature point, Represents the information after all other feature points are aggregated to the iiith feature point, Represents the input of the iiith feature point The calculated index information, Represents the input of the jjjth feature point The calculated key information, Indicates that the jjjth feature point is input Calculated value information; the superscript T indicates the transpose of the matrix; Aggregate all other feature points of the current layer to the information of the iiith feature point And the input of the iiith feature point Connect them to get the input of the next layer; when the iii-th feature point and the jjj-th feature point come from the same image, calculate self-attention to aggregate the relationship between feature points in a single image; when the feature points come from two different images, calculate cross-attention to capture the relationship between key points across images; After that, the optimal matching layer is passed to complete the feature point set of the current frame image And the feature point set of the previous frame image Matching results , which is an array, each array element is a two-tuple , where query represents The query feature point in , train is The trainth feature point in is used to achieve successful matching.

9. The visible light near infrared fusion positioning method based on feature consistency according to claim 8 is characterized in that: The step 8 comprises: According to the matching results , triangulate the matching feature points of the two frames to get the spatial point position , and complete the 3D-2D pose solution based on the camera projection model to obtain the current pose.

10. A visible light near infrared fusion positioning system based on feature consistency, characterized in that: Includes the following modules: Image acquisition module, based on RGB-D camera, simultaneously acquires visible light images and the original near infrared image ; RGB-D stands for color-depth; The registration module completes the prior calibration based on the calibration plate and obtains the external parameters from the near-infrared image to the visible light image. , and according to external reference , register each frame of near-infrared image acquired to the visible light image , obtain the visible light image Aligned NIR images ; Denoising enhancement module for visible light images Aligned NIR images , based on binary mask, fast detection and removal of near infrared images The hollow noise is removed and gamma correction is performed to obtain the denoised and enhanced near-infrared image. ; The collection acquisition module performs deep learning-based feature extraction on the current input image and sets the feature extraction threshold according to the current input image source type: a low threshold is used for near-infrared images , high threshold is used for visible light images , get the feature point set of the current frame image ; Light judgment module, based on visible light image , calculate the average pixel brightness and histogram uniformity U, to determine whether the current external lighting conditions are ideal; Smooth switching module, according to the principle of smooth switching, selects the current input image: when the lighting is ideal, inputs the visible light image When the lighting is not ideal, the input is the denoised and enhanced near-infrared image ; The association module uses the deep learning feature matching method to match the feature point set of the current frame image. The image and the feature point set of the previous frame image When the two frames of images are of the same modality, single-modality feature tracking is achieved; when the two frames of images are of different modality types, dual-modality feature association is achieved; The positioning module triangulates the spatial map points based on the feature tracking results and calculates the current camera position. The solution can realize autonomous positioning.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a visible light near-infrared fusion positioning method based on feature consistency as described in any one of claims 1 to 9 are implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a visible light near-infrared fusion positioning method based on feature consistency as described in any one of claims 1 to 9 are implemented.

Images