Infrared-visible light fusion visual positioning method and system based on large-scale visual language model

Abstract

The invention discloses an infrared-visible light fusion visual positioning method and system based on a large-scale visual language model, and relates to the field of infrared and visible light feature fusion and visual language large models. The method solves the problems that an existing visual language model is prominent in presentation in generative tasks such as semantic understanding, reasoning and planning, but still has obvious defects in the aspect of accurate target positioning, and the like, and is suitable for the field of generative tasks. The method comprises the following steps: inputting a registered infrared-visible light fusion image in the generative task field and a keyword prompt into a large-scale visual language model; traversing candidate attention heads, processing the registered infrared-visible light fusion image, comparing the attention map of each attention head with a real label mask, and screening out the attention head most related to target positioning; and extracting an attention map of each mode by using the screened attention head, and carrying out fusion, filtering and binarization processing to obtain a final fusion mask.

Description

Technical Field

[0001] This invention relates to the field of infrared and visible light feature fusion and large-scale visual language model technology, specifically to an infrared-visible light fusion visual positioning method and system based on a large-scale visual language model. Background Technology

[0002] Visual grounding aims to accurately locate and label target regions (such as bounding boxes or segmented regions) in an image based on a given natural language description. It is a fundamental capability of large-vision-language models in complex tasks. While existing visual-language models excel in generative tasks such as semantic understanding, reasoning, and planning, they still have significant shortcomings in accurate target localization, especially in applications requiring high spatial accuracy. In real-world scenarios such as autonomous driving and industrial automation, complex lighting conditions (such as strong light, low light, and fog) frequently occur, making it difficult for single-modal visual grounding methods to maintain stable and reliable performance over long periods, thus limiting the effectiveness of large-vision models in real-world environments.

[0003] To enhance visual localization capabilities in complex environments, introducing multimodal perception strategies is an effective approach. Visible light images possess rich texture details and high resolution, but are highly sensitive to changes in illumination; infrared images, on the other hand, exhibit stronger robustness in harsh conditions such as darkness and smoke, but lack fine structural information. Therefore, the effective fusion of infrared and visible light modalities is considered an important direction for improving localization robustness. Existing technologies typically employ a hybrid paradigm, introducing traditional detection or segmentation modules at the input or output stages of large visual language models to compensate for their insufficient localization capabilities. However, such methods often rely on additional model structures and cross-modal annotation data, which not only increases system complexity but also reduces deployment flexibility under various scene conditions.

[0004] Recent research has revealed that the self-attention mechanism within large visual language models implicitly incorporates information related to the spatial location of the target during inference, providing new insights into unlocking their inherent visual localization capabilities. Some methods attempt to utilize attention maps from intermediate layers of the model as localization cues to mitigate the illusion problem in the generated results. However, these methods typically only provide coarse-grained spatial responses and often require post-processing using external visual models, thus diminishing the end-to-end inference advantages of large visual language models.

[0005] Given the dual requirements of semantic understanding and spatial positioning for infrared and visible light fusion visual positioning, this invention proposes for the first time a technical solution that can directly utilize the internal attention mechanism of a large visual language model to achieve multimodal fusion and accurate positioning without additional training or external modules. Summary of the Invention

[0006] This invention addresses the shortcomings of existing visual language models, which excel in generative tasks such as semantic understanding, reasoning, and planning, but still suffer from significant deficiencies in precise target localization, particularly in applications requiring high spatial accuracy. In real-world scenarios like autonomous driving and industrial automation, complex lighting conditions (such as strong light, low illumination, and fog) frequently occur, making it difficult for single-modal visual localization methods to maintain stable and reliable performance over long periods, thus limiting the effectiveness of large-scale visual models in real-world environments. Currently, there is a severe shortage of infrared and visible light images for training datasets to generate models, and most existing large-scale visual language models are primarily trained on visible light datasets. However, both infrared and visible light images are optical images with fundamentally similar properties; therefore, this invention proposes an infrared-visible light fusion visual localization method and system based on a large-scale visual language model.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: Option 1: This invention proposes an infrared-visible light fusion visual localization method based on a large-scale visual language model. This method is applicable to generative task domains, including semantic understanding, reasoning, and planning. The method includes the following steps: Step 1: Input the registered infrared-visible fusion images and keyword prompts from the generative task domain into the large-scale visual language model; Step 2: Traverse the candidate attention heads, process the registered infrared-visible fusion image described in Step 1, and compare the attention map of each attention head with the real label mask to select the attention head most relevant to the target localization. Step 3: Use the attention heads selected in Step 2 to extract the attention maps of each modality, convert the sparse attention maps into binary masks, and after fusion, filtering and binarization, obtain the final fused mask.

[0008] Furthermore, in a preferred embodiment, step 1 further includes quantifying the contribution of each attention head to the target region. The steps, namely,

[0009] in, and These are the length and width of the image, respectively. In order to be in The true value of the pixel position. This represents the attention intensity at the corresponding location.

[0010] Furthermore, a preferred embodiment is provided, wherein the method for comparing the attention map of each attention head with the real label mask in step 2 to select the attention head most relevant to target localization is as follows: Step 2.1: Measure the attention concentration of different numbers of attention heads and obtain the attention concentration score;

[0011] Step 2.2: Calculate the proportion of attention quality located within the marked area based on the attention concentration score obtained in Step 2.1; .

[0012] Furthermore, in a preferred embodiment, step 3 further includes an interpolation and scaling process for the attention map, namely, mapping the output pixel coordinates to a continuous coordinate system of the input plane using bilinear interpolation, achieved by selecting the four nearest integer grid points in the neighborhood of each coordinate point. .

[0013] Furthermore, a preferred embodiment is provided, wherein the interpolation scaling is performed using two-dimensional bilinear interpolation for image scaling. Sampling is conducted in a 2×2 neighborhood at the mapping location, and the output result is a weighted combination of four adjacent pixels, with the weights determined by the vertical direction. and horizontal The decision is that this operation can be performed separately, i.e., linear interpolation is performed along the horizontal and vertical axes separately to obtain a smooth resampling result, defined as follows:

[0014] in, , , , . The matrix is ​​around neighboring elements, and These are the weights or normalization factors in the vertical and horizontal directions, which control the interpolation intensity in the vertical and horizontal directions during the interpolation process, respectively.

[0015] Furthermore, in a preferred embodiment, step 3 further includes a step of eliminating noise using a Gaussian filter, that is,

[0016] in, This represents the interpolated attention map. Its reflection extension portion is obtained through reflection filling. Indicates the output spatial location. Sum the indices of the input positions. This represents the attention response at spatial location p after Gaussian smoothing.

[0017] Furthermore, a preferred embodiment is provided in which the sparse attention map is converted into a binary mask in step 3 using the Otsu thresholding method, i.e.,

[0018] in, Indicates the end category The cumulative class probability, It is the corresponding cumulative mean. This represents the population mean.

[0019] Option 2: An infrared-visible light fusion visual positioning system based on a large-scale visual language model, the system being applicable to generative task domains, including semantic understanding, reasoning, and planning, the system comprising: The attention mechanism module is used to input the registered infrared-visible fusion image and keyword prompts from the generative task domain into a large-scale visual language model. The visual positioning attention head filtering module is used to traverse candidate attention heads, process the registered infrared-visible light fusion image described in the attention mechanism module, and compare the attention map of each attention head with the real label mask to filter out the attention head most relevant to the target positioning. The multimodal attention map fusion module is used to extract attention maps of each modality using the attention heads selected in the filtering module, convert the sparse attention map into a binary mask, and obtain the final fused mask after fusion, filtering and binarization.

[0020] Option 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0021] Option 4: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0022] The advantages of this invention are: The infrared-visible light fusion visual localization method and system based on a large-scale visual language model described in this invention can be plug-and-play adapted to multiple series of open-source visual language models, such as the LLaVA series, Qwen2VL series, Qwen2.5VL series, and DeepseekVL2 series. Comparative experiments were conducted on multiple general-purpose large visual language models and their different variants. Experimental results show that, on the vast majority of datasets and evaluation metrics (including accuracy, recall, and F1 score), the method described in this invention achieves stable performance improvements compared to the original inference process, verifying the effectiveness of the method. Furthermore, since the method described in this invention does not require the execution of all generation steps during the inference phase, it has a significant efficiency advantage. When performing the same number of inference tasks (FMB-dataset test set: 2602 times) on all experimental models, the method described in this invention consistently takes less time.

[0023] The perception method proposed in this invention can leverage the complementary advantages of visible light and infrared modes to compensate for each other's limitations, exhibiting significant advantages in extreme environments, particularly suitable for complex lighting or low-visibility scenarios. Experimental results show that under strong light and backlight conditions, strong light interferes with the model's ability to perceive target contours, leading to inaccurate attention allocation in some areas. Although infrared images are also affected by strong light, as strong light sources emit infrared radiation that may cause signal saturation and loss of perception information in some areas, the fusion of visible light and infrared modes effectively integrates their complementary advantages, improving overall perception performance. On the other hand, in foggy environments, infrared images can easily achieve target detection, while visible light images struggle to demonstrate their advantages. In such scenarios, the fusion result exhibits high fault tolerance, ensuring the stability of the target recognition task. Attached Figure Description

[0024] Figure 1 This is a schematic diagram comparing the method of the present invention described in Embodiment 1 with existing reasoning processes.

[0025] Figure 2 This is a schematic diagram of the algorithm flow described in Implementation Method 1.

[0026] (a) is a schematic diagram of key attention head extraction, and (b) is a schematic diagram of multimodal fusion localization mask generation.

[0027] Figure 3 A visualization of multimodal attention under different lighting conditions.

[0028] Figure 4 This is a schematic diagram of the most relevant attention heads for different models of visual language models in the daytime scene described in Implementation Method 1.

[0029] Figure 5This is a schematic diagram of the most relevant attention heads for different models of visual language models in the night scene described in Implementation Method 1.

[0030] Figure 6 This is a schematic diagram illustrating the impact of the number of fusion heads on the fusion attention of key regions, as described in Implementation Method 1.

[0031] Figure 7 This is a comparative diagram of the original localization and inference performance of multiple visual language models described in Implementation Method 1.

[0032] Figure 8 This is a diagram illustrating the improved inference efficiency described in Implementation Method 1.

[0033] Figure 9 This is a performance analysis diagram for extreme lighting scenarios as described in Implementation Method 1. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0035] Implementation Method 1: This implementation method proposes an infrared-visible light fusion visual localization method and system based on a large-scale visual language model. The method specifically includes the following steps: This implementation inputs paired visible light and infrared images into a large visual language model that has not been specifically trained for this task. For example... Figure 1 As shown, traditional multi-image inference visual localization methods generate object detection box information through text generation. Existing research shows that there are often a lot of illusions in the language generation layer, which does not fully utilize the localization ability of the visual language model. Therefore, this implementation method abandons the text generation part that is prone to illusions and directly uses the model's attention to generate mask for key regions, which can maximize the model's localization potential without training.

[0036] like Figure 2 As shown, this embodiment discloses an infrared-visible light fusion visual localization method based on a large-scale visual language model. The method described in this embodiment is mainly applied to multimodal complex scene perception, and typical applicable scenarios include ground target detection (covering vehicles, pedestrians, streetlights, and various street facilities) under complex weather conditions, such as urban streets, nighttime environments, and complex weather conditions. The data formats used mainly include paired infrared-visible light images, text prompts describing the target, and corresponding ground truth masks.

[0037] The core logic of this invention comprises two stages: key attention head selection and multimodal fusion inference. 1. In the key attention head selection stage: The registered infrared and visible light dual-modal images are input into the large visual language model along with the target text keywords. By traversing the candidate attention heads in the model, the correlation or overlap between the attention map generated by each attention head and the real label mask is calculated, thereby accurately selecting the set of key attention heads with the most discriminative power for target localization and eliminating redundant or noisy heads.

[0038] 2. In the reasoning and localization stage: Using the key attention heads selected above, attention feature maps are extracted in infrared and visible light modes respectively. Subsequently, the dual-modal feature maps are weighted and fused, and then subjected to spatial filtering (such as Gaussian filtering) and adaptive binarization processing to generate a high-precision final fusion mask.

[0039] To verify the effectiveness of this method, extensive tests were conducted on public datasets such as MSRS, FMB-Dataset, and MFNet. The generated fusion mask was compared with the ground truth mask, and accuracy, recall, and F1 score were calculated. Quantitative and qualitative experimental results show that the method described in this embodiment significantly outperforms existing benchmark visual language models on multimodal visual localization tasks, effectively solving the problem of inaccurate localization of single modal models in harsh environments, and providing a new technical path for the application of large models in the field of multimodal fusion perception.

[0040] The infrared-visible light fusion visual localization method based on a large-scale visual language model described in this invention specifically includes the following steps: Step 1: Exploring the Attention Mechanism As research into the interpretability of language models deepens, analyzing and visualizing their internal self-attention mechanisms has become an effective means to improve model reliability and gain a deeper understanding of the underlying decision-making processes. Existing research shows that attention layers and attention heads typically possess distinct functional specializations. In language models, attention heads can be broadly categorized into different functional types: some attention heads primarily focus on repetitive markers to reinforce text repetition patterns, others focus on low-frequency markers, and still others model sequence dependencies by capturing the relative positional relationships between markers. Similarly, language models include visually relevant attention heads (often called localizing heads), which are particularly sensitive to salient object regions. Notably, these heads are not uniformly distributed across layers but tend to cluster in specific stages of the network, typically concentrated in the middle layers. Based on this finding, we quantified the contribution of each attention head to the target region. This is used as the standard for evaluating the head:

[0041] in and These are the length and width of the image, respectively. Represents in The truth value of the pixel position (1 for the correct position, 0 for other positions). This represents the attention intensity at the corresponding location. It's worth noting that the position and behavior of the effective visual head are not fixed. They vary depending on the model architecture and are also influenced by the modal content. We... Figure 3 This was further verified by comparing the spatial focusing of the attention head under visible and infrared light inputs. Visible light and infrared images exhibit significant differences in perceptual characteristics under day and night conditions. Because most infrared-visible light benchmark data clearly indicate the acquisition time (daytime and nighttime), we divided the data into two scenarios for separate evaluation and selection of attention heads: such as... Figure 5 and Figure 6 As shown, this invention randomly selected 100 samples from the daytime and nighttime datasets of the MSRS infrared-visible light dataset to evaluate eight mainstream open-source visual localization models (including LLaVA, Qwen2-VL, Qwen2.5-VL, and the DeepSeek-VL2 series). For each model, we selected the top eight attention heads with the highest relevance to the target localization region, laying the foundation for further multimodal attention head fusion.

[0042] Step Two: Visual Positioning and Attention Target Selection To quantitatively determine the optimal number of attention heads in a fusion task, this invention establishes a set of evaluation metrics to quantify performance. It is important to note that in multi-objective scenarios, the attention alignment evaluation cannot directly reuse the cross-entropy objective function commonly used in single-objective scenarios, because the supervision definition involves multiple spatial regions rather than a single dominant objective. We measure the attention concentration under different attention head numbers using the following formula, applicable to both single-objective and multi-objective scenarios:

[0043] Here, and represents the number of attention heads selected from the visible (V) and infrared (I) modes, respectively. For each selected attention head, we extract a normalized non-negative spatial attention map. and ,in Indicates spatial resolution. Given a binary label mask. We calculate the attention score as the percentage of attention quality located within the marked area:

[0044] When the focus of the fusion is more concentrated on the key target area Increase. Therefore, this metric provides a direct and model-independent method for evaluating Choose how to influence attention alignment related to positioning. For example... Figure 4 As shown, when fusing two attention heads from each modality... Attention is most focused on the key areas.

[0045] Step 3: Multimodal Attention Map Fusion Method In image fusion processing, the pixel values ​​of the attention map often differ from those of the input image. These attention maps are typically compressed into square images (e.g., 336×336 LLaVA). Therefore, interpolation scaling is required for the attention map. Specifically, bilinear interpolation is used to map the output pixel coordinates to a continuous coordinate system in the input plane, achieved by selecting the four nearest integer grid points in the neighborhood of each coordinate point. The weights of these points are linearly determined by their distance. Let the input feature map be... (Target size is) The output is The corresponding coordinates Mapped to the input plane:

[0046] After coordinate mapping, this invention employs two-dimensional bilinear interpolation for image scaling. A (2×2) neighborhood is sampled around the mapped location, and the output is calculated as a weighted combination of four adjacent pixels, with the weights determined by the vertical direction. and horizontal The decision is made. This operation can be performed separately, i.e., linear interpolation is performed along the horizontal and vertical axes respectively, to obtain a smooth resampling result, defined as follows:

[0047] Here, , , , . The matrix represents the surrounding neighboring elements, and These are the weights or normalization factors in the vertical and horizontal directions, controlling the interpolation intensity in the vertical and horizontal directions during the interpolation process, respectively. During the resizing process, when sampling points exceed the boundary, a boundary expansion strategy (reflection filling) is used to define... Values ​​are used to avoid boundary artifacts.

[0048] In Transformer-based models, attention activation maps often exhibit significant noise, which reduces visualization accuracy. This noise can be caused by attention collapse, missing registration information, and redundant features across categories. This type of noise is primarily characterized by salt-and-pepper noise. To effectively eliminate this noise, we employ a Gaussian filter, with the specific filtering formula as follows:

[0049] in This represents the interpolated attention map. Its reflection extension portion is obtained through reflection filling. Indicates the output spatial location. Sum the indices of the input positions. This represents the Gaussian smoothed attention response at spatial location p, which is calculated by weighted aggregation of the extended attention map within the local neighborhood. By mirroring and padding the attention map boundary values, stable convolutions are achieved in edge regions, reducing boundary artifacts. The Gaussian kernel is defined as follows:

[0050] in Control the smoothing intensity, Spatial dimension (usually) This filtering method uses reflection filling to process boundary pixels, avoiding abrupt truncation, maintaining the smoothness of the boundary region, and preventing the generation of boundary artifacts.

[0051] Step 4: Automatic Threshold Binarization Strategy After extracting the attention map, accurately and appropriately converting the sparse attention map into a binary mask is a key challenge. Through comparative analysis of different automatic thresholding methods, we found that the Otsu thresholding method performed the most balanced in the tests. Its core advantage lies in being completely data-driven and parameter-free, selecting the optimal threshold by maximizing the inter-class variance (equivalent to minimizing the intra-class variance). This makes the method easy to deploy, computationally inexpensive, and robust in scenes with good foreground / background distribution discriminability. However, Otsu also has significant limitations: it implicitly assumes a bimodal intensity distribution, and the selected threshold may become unstable when the attention map exhibits a multimodal distribution, low contrast, or strong noise. To mitigate these shortcomings while retaining Otsu's simplicity and automation advantages, we further optimized the process by adding an edge smoothing post-processing step after thresholding. Specifically, we employed boundary refinement techniques to suppress false isolated pixels, fill in minor discontinuities, and smooth jagged boundary structures, thereby reducing the Otsu algorithm's sensitivity to local noise and improving the geometric consistency of the final mask. The calculation formula is shown below:

[0052] in, Indicates the end category The cumulative class probability, It is the corresponding cumulative mean. This represents the overall mean. By selecting the threshold that maximizes the inter-class variance, the effective binarization threshold for the image is determined. Using the above method, a key target mask that can effectively integrate infrared and visible light modal information can be obtained.

[0053] In summary, this invention demonstrates that by utilizing the internal attention mechanism of a pre-trained visual language model, effective infrared-visible light fusion and robust visual localization can be achieved without training, providing a low-cost and highly efficient feasible approach for multimodal visual fusion.

[0054] The original method proposed in this implementation is plug-and-play compatible with multiple series of open-source visual language models (e.g., LLaVA series, Qwen2VL series, Qwen2.5VL series, DeepseekVL2 series, etc.), and comparative experiments have been conducted on multiple general-purpose large visual language models and their different variants, such as... Figure 7 As shown, experimental results demonstrate that the proposed method achieves stable performance improvements over the original inference process on the vast majority of datasets and evaluation metrics (including accuracy, recall, and F1 score), validating the effectiveness of the method. Furthermore, since the proposed method does not require executing all generation steps during the inference phase, it possesses a significant efficiency advantage. Figure 5 As shown, our method consistently takes less time when performing the same number of inference tasks on all experimental models (FMB-dataset test set: 2602 times).

[0055] Meanwhile, the sensing method proposed in this invention can utilize visible light and infrared modes to compensate for each other's sensing limitations, exhibiting significant advantages in extreme environments, and is particularly suitable for complex lighting or low-visibility scenarios. This is evident in... Figure 6 The results were further validated in a visualization comparison experiment. The results show that under strong backlight conditions, strong light interferes with the model's ability to perceive target contours, leading to inaccurate attention allocation in some areas. Although infrared images are also affected by strong light—due to the infrared radiation emitted by strong light sources potentially causing signal saturation and loss of perceptual information in some areas—the fusion of visible and infrared modes can effectively integrate their complementary advantages and improve overall perception performance. On the other hand, in foggy conditions, infrared images can easily achieve target detection, while visible light images struggle to leverage their advantages. In such scenarios, the fusion result exhibits high fault tolerance, ensuring the stability of the target recognition task.

[0056] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0057] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. An infrared-visible light fusion visual localization method based on a large-scale visual language model, characterized in that, The method is applicable to generative task domains, including semantic understanding, reasoning, and planning, and includes the following steps: Step 1: Input the registered infrared-visible fusion images and keyword prompts from the generative task domain into the large-scale visual language model; Step 2: Traverse the candidate attention heads, process the registered infrared-visible fusion image described in Step 1, and compare the attention map of each attention head with the real label mask to select the attention head most relevant to the target localization. Step 3: Use the attention heads selected in Step 2 to extract the attention maps of each modality, convert the sparse attention maps into binary masks, and after fusion, filtering and binarization, obtain the final fused mask.

2. The infrared-visible light fusion visual localization method based on a large-scale visual language model according to claim 1, characterized in that, Step 1 also includes quantifying the contribution of each attention head to the target region. The steps, namely, in, and These are the length and width of the image, respectively. In order to be in The true value of the pixel position. This represents the attention intensity at the corresponding location.

3. The infrared-visible light fusion visual localization method based on a large-scale visual language model according to claim 1, characterized in that, In step 2, the method for comparing the attention map of each attention head with the real label mask to select the attention head most relevant to target localization is as follows: Step 2.1: Measure the attention concentration of different numbers of attention heads and obtain the attention concentration score; Step 2.2: Calculate the proportion of attention quality located within the marked area based on the attention concentration score obtained in Step 2.1; 。 4. The infrared-visible light fusion visual localization method based on a large-scale visual language model according to claim 1, characterized in that, Step 3 also includes an interpolation and scaling process for the attention map, which involves using bilinear interpolation to map the output pixel coordinates to a continuous coordinate system on the input plane. This is achieved by selecting the four nearest integer grid points in the neighborhood of each coordinate point. .

5. The infrared-visible light fusion visual localization method based on a large-scale visual language model according to claim 4, characterized in that, The interpolation scaling is performed using two-dimensional bilinear interpolation. At the mapped location, a 2×2 neighborhood is sampled, and the output is a weighted combination of four adjacent pixels. The weights are determined by the vertical direction. and horizontal The decision is that this operation can be performed separately, i.e., linear interpolation is performed along the horizontal and vertical axes separately to obtain a smooth resampling result, defined as follows: in, , , , . The matrix is ​​around neighboring elements, and These are the weights or normalization factors in the vertical and horizontal directions, which control the interpolation intensity in the vertical and horizontal directions during the interpolation process, respectively.

6. The infrared-visible light fusion visual localization method based on a large-scale visual language model according to claim 1, characterized in that, Step 3 also includes the step of eliminating noise using a Gaussian filter, that is, in, This represents the interpolated attention map. Its reflection extension portion is obtained through reflection filling. Indicates the output spatial location. Sum the indices of the input positions. This represents the attention response at spatial location p after Gaussian smoothing.

7. The infrared-visible light fusion visual localization method based on a large-scale visual language model according to claim 4, characterized in that, In step 3, the Otsu thresholding method is used to convert the sparse attention map into a binary mask, that is, in, Indicates the end category The cumulative class probability, It is the corresponding cumulative mean. This represents the population mean.

8. An infrared-visible light fusion visual positioning system based on a large-scale visual language model, characterized in that, The system is applicable to generative task domains, including semantic understanding, reasoning, and planning. The system comprises: The attention mechanism module is used to input the registered infrared-visible fusion image and keyword prompts from the generative task domain into a large-scale visual language model. The visual positioning attention head filtering module is used to traverse candidate attention heads, process the registered infrared-visible light fusion image described in the attention mechanism module, and compare the attention map of each attention head with the real label mask to filter out the attention head most relevant to the target positioning. The multimodal attention map fusion module is used to extract attention maps of each modality using the attention heads selected in the filtering module, convert the sparse attention map into a binary mask, and obtain the final fused mask after fusion, filtering and binarization.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

10. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.

Images