Method and system for constructing infrared target tracking retrieval knowledge base in dark environment

Abstract

The invention discloses a method and a system for constructing an infrared target tracking retrieval knowledge base in a dark environment, and relates to the technical field of infrared detection tracking. The method comprises the following steps: acquiring an infrared image in the dark environment; inputting the infrared image into a target tracking model optimized based on a YOLOv8 model, and outputting target tracking information; constructing a knowledge vector library by utilizing a retrieval enhancement generation method based on the infrared image and the target tracking information; and performing multi-modal feature fusion by using the knowledge vector library, and storing infrared data after multi-modal feature fusion. According to the method, the YOLOv8 model is optimized, the target tracking model is constructed, and the target tracking information is obtained by using the target tracking model, so that the detection and tracking accuracy of the target tracking model is improved; and constructing and utilizing a knowledge vector library to carry out multi-modal feature fusion, and storing the infrared data, so that localized efficient storage of the infrared data in a dynamic and complex multi-modal scene is realized.

Description

Technical Field

[0001] The present application relates to the field of infrared detection and tracking technology, and in particular to a method and system for constructing an infrared target tracking and retrieval knowledge base in a dim environment. Background Art

[0002] Facing complex and changeable dim environments, accurately obtaining the real-time position trajectory and category information of dynamic targets is the top priority for realizing intelligent autonomous perception of unknown environments. After a natural disaster occurs, efficient target search is of practical significance for real-time and accurate completion of infrared dynamic target recognition, search and tracking rescue in dark environments to ensure personnel safety. In order to meet the requirements of all-weather, good concealment, smoke penetration and blind spot detection, infrared imaging search and tracking technology (IRST) plays a vital role in military and civilian fields. Infrared image target detection methods can be divided into filtering local information detection methods and deep learning-based pixel feature representation capability detection methods. Deep learning-based infrared image target detection algorithms have become the main trend of current research. They are mainly designed and optimized by using convolutional neural networks (CNNs) and Transformers models as basic models to ensure the balance between infrared target tracking accuracy and reasoning ability in dark environments.

[0003] Current infrared image target detection and tracking methods have problems such as being unable to effectively and real-time detect and track targets in dim environments, low target detection position accuracy, offset in trajectory tracking, and the inability to efficiently save tracked infrared data in multimodal scenarios. Summary of the invention

[0004] The purpose of this application is to provide a method and system for constructing an infrared target tracking retrieval knowledge base in a dim environment, which can improve the accuracy of target tracking model detection and tracking, and realize the localization and efficient preservation of infrared data in dynamic and complex multimodal scenes.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] In the first aspect, the present application provides a method for constructing an infrared target tracking retrieval knowledge base in a dim environment, the method comprising: acquiring an infrared image in a dim environment; the dim environment is an environment under conditions where visibility is lower than a preset value; inputting the infrared image into a target tracking model, and outputting target tracking information; the target tracking model is optimized based on a YOLOv8 model; the target tracking information comprises: the position, category and trajectory information of all detected targets; based on the infrared image and the target tracking information, constructing a knowledge vector library using a retrieval enhancement generation method; using the knowledge vector library to perform multimodal feature fusion, and saving the infrared data after the multimodal feature fusion.

[0007] In a second aspect, the present application provides a computer system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for constructing a knowledge base for infrared target tracking and retrieval in a dim environment.

[0008] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0009] This application optimizes the YOLOv8 model to construct a target tracking model, and uses the target tracking model to obtain target tracking information, thereby improving the accuracy of detection and tracking of the target tracking model; constructs and uses a knowledge vector library for multimodal feature fusion, and saves infrared data, thereby achieving localized and efficient preservation of infrared data in dynamic and complex multimodal scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 A flowchart of a method for constructing an infrared target tracking and retrieval knowledge base in a dim environment provided in an embodiment of the present application.

[0012] Figure 2 An improved Yolov8 model is provided in the embodiments of the present application.

[0013] Figure 3 A schematic diagram of the structure of the Bottleneck CSP network provided in an embodiment of the present application.

[0014] Figure 4 A schematic diagram of the structure of the RCCA module provided in an embodiment of the present application.

[0015] Figure 5 A schematic diagram of the structure of the DASI dimension perception selective integration module provided in an embodiment of the present application.

[0016] Figure 6 A schematic diagram of the structure of a computer system provided for an embodiment of the application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0019] Embodiment 1, as Figure 1-Figure 2 As shown, this embodiment provides a method for constructing an infrared target tracking and retrieval knowledge base in a dim environment, and the method includes the following steps.

[0020] S1. Acquire an infrared image in a dim environment; a dim environment is an environment where visibility is lower than a preset value.

[0021] Furthermore, the infrared image is an image captured by binocular infrared cameras (IR-Camera1 and IR-Camera2); the model of the binocular infrared camera module is InfiSense S2640-P20004S00.

[0022] Optionally, the external environment of this embodiment is: under the ambient temperature T0=25°C, the noise equivalent temperature difference (NETD) of the device is less than 50mK, and the working band is [8μm, 14μm]. In this environment, the depth of the target is estimated, and the feature ID is matched and tracked.

[0023] S2. Input the infrared image into the target tracking model and output the target tracking information; the target tracking model is optimized based on the YOLOv8 model; the target tracking information includes: the location, category and trajectory information of all detected targets.

[0024] Furthermore, the target tracking model includes a backbone network, a neck network and a head network connected in sequence; the structure of the backbone network is a CSPDarknet53 structure; the backbone network includes an offset learning module, a BottleneckCSP structure and a DASI module connected in sequence; wherein, Figure 3 As shown, the BottleneckCSP structure includes a convolution operation module, an AKConv module and an RCCA module connected in sequence; the neck network is a PANet structure; and the head network includes multiple multi-scale dynamic detection heads.

[0025] Optionally, the target tracking model can be optimized and verified by real-time reasoning based on low-computing edge embedded devices and binocular infrared camera modules. By optimizing the model parameters, the computational overhead can be reduced while ensuring the detection and tracking performance of the model. Through real-time reasoning verification in actual environments, the accuracy and robustness of the model in detecting and tracking dynamic targets in complex dark environments can be ensured.

[0026] In the actual application process, this embodiment uses CSPDarknet53 instead of Darknet53 as the structure of the backbone network, and solves the redundancy problem of the network when fusing features at different levels through the cross-stage partial network structure (CSPNet). The BottleneckCSP structure replaces the Conv layer in Bottleneck2 with the AKConv, RCCA, and DASI modules, and the feature output value is the weighted average of the surrounding positions. Integrating the AKConv, RCCA, and DASI modules into the C2F network structure enhances the network's ability to detect and track dynamic features in complex dark environments.

[0027] Step S2 specifically includes the following steps.

[0028] S21. Utilize the offset learning module and the convolution operation module to extract the multi-scale initial feature map of the infrared image according to the offset of the convolution kernel sampling point.

[0029] In actual applications, the Offset Learning Module learns the offset of sampling points from the input feature map through convolution or fully connected layers. These offsets dynamically adjust the sampling grid of the convolution kernel, allowing each convolution kernel to sample the input feature map at different positions and obtain the tensor of offsets of each sampling point of the convolution kernel. The Convolution Operation Module uses the feature map combined with the offset tensor for dynamic convolution. The offset tensor adjusts the sampling position of the convolution kernel so that it can perform more flexible convolution operations according to the structure and characteristics of the input feature map.

[0030] S22. The neck network based on the PANet (Path Aggregation Network) structure uses a path aggregation mechanism to fuse features at different levels in the multi-scale initial feature map to obtain a fused multi-scale feature map.

[0031] In actual applications, the application of PANet structure enables better combination of multi-scale features at different levels and enhances the target detection effect.

[0032] S23. According to the fused multi-scale feature map, the variable and convolution AKConv module is used to extract geometric shape features through convolution kernels of variable shape and size to obtain a variable kernel enhanced feature map.

[0033] In actual application, the AKConv module improves the model's feature extraction capabilities by using convolution kernels of variable shapes and sizes. The AKConv module can more flexibly adapt to the geometric features of the input data by learning the location of the sampling points, and can capture more subtle and irregular shape features than standard convolution. Similar to standard convolution, AKConv also contains multiple convolution kernels, and the sampling points of these convolution kernels are not fixed, but dynamically adjusted according to the features of the input image and the learned offset.

[0034] S24. Figure 4 As shown, the RCCA module is used to calculate the relationship between the target feature pixel in the variable kernel enhanced feature map and all other points in the feature map, and the features of the target pixel are weighted according to the relationship to obtain the weighted variable kernel enhanced feature map.

[0035] In practical applications, an RCCA module (Recurrent Criss Cross Attention Moudle) can be added during the convolution operation. The RCCA module is composed of two CCA (Criss Cross Attention Moudle) modules in series. By calculating the relationship between the target feature pixel and all other points in the feature map, and using this relationship to weight the features of the target pixel, a more effective target feature can be obtained.

[0036] S25. Figure 5 As shown, the DASI module is used to perform feature fusion on the weighted variable kernel enhanced feature map in an adaptive screening manner to obtain a dimension-aware fused feature map.

[0037] In actual application, the DASI module effectively solves the problems of feature information loss and background clutter in infrared small target detection by adaptively selecting fused features, thereby improving the saliency of small targets and ultimately improving the detection performance.

[0038] S26. Use multiple multi-scale motion detection heads to extract multi-scale semantic information from the dimension-aware fusion feature map to obtain a multi-scale motion detection feature map.

[0039] Step S26 specifically includes the following steps.

[0040] S261. Use a multi-scale motion detection head to perform multi-scale semantic information processing on the dimension-aware fusion feature map to obtain a multi-scale motion detection feature map, which specifically includes the following steps.

[0041] S262. Use the five-frame difference model combined with dynamic area and depth estimation to calculate the pixel intensity difference of the dimension-aware fusion feature map of consecutive frames and generate a difference image.

[0042] Furthermore, the formula of the five-frame difference model is as follows.

[0043] .

[0044] in, The difference image after five frames of difference, For the Frame image, For the Frame image, For the Frame image.

[0045] S263. Determine the target area of ​​the differential image through binarization to obtain a binarized image.

[0046] Optionally, the target area determination formula is as follows.

[0047] .

[0048] .

[0049] in, is a binary image, is the three-frame difference image, and τ is the coefficient value.

[0050] S264. Based on the camera intrinsic parameter matrix, a calibration method is used to map the pixel coordinates in the binary image to the normalized camera coordinate system.

[0051] S265. Map the binary image in the camera coordinate system to the world coordinate system to obtain a multi-scale dynamic detection feature map.

[0052] In actual application, by adding a multi-scale dynamic detection head, the multi-scale semantic information processing capability is improved in the output part to avoid occlusion or missed detection of small objects. The multi-scale feature detection head is used to enhance the feature detection and positioning performance.

[0053] The conversion process between the camera coordinate system and the world coordinate system is as follows.

[0054] First, to reduce the possible distortion in the streaming image, the pixel coordinates in the image are ) is mapped to the normalized camera coordinate system through the calibration method. The pixel coordinates are converted to normalized camera coordinates through the camera intrinsic parameter matrix , the normalized camera coordinate formula is as follows.

[0055] .

[0056] in, and is the focal length of the left and right lenses of the binocular infrared camera, The coordinate offset of the main point.

[0057] Next, the normalized camera coordinates are mapped to the complete 3D camera coordinates by combining the target’s depth value. ). Then, the spatial transformation is performed through the external parameter matrix to map the points in the camera coordinate system to the world coordinate system.

[0058] The mapping formula from camera coordinates to world coordinates is as follows.

[0059] .

[0060] Among them, R represents the rotation matrix and T represents the translation vector.

[0061] Finally, the comprehensive transformation formula from pixel coordinates to world coordinates is as follows.

[0062] .

[0063] The mapping function of the moving target is defined by the function , associate the position of the moving target with its three-dimensional space coordinates, and define the function The formula is as follows.

[0064] .

[0065] The mapping function combines the detection results and depth estimation of the motion area, and can achieve efficient tracking and acquisition of dynamic targets in three-dimensional space.

[0066] S27. Perform target detection and target tracking on the multi-scale dynamic detection feature map obtained at multiple scales through a dual-branch network to obtain target tracking information; the dual-branch network includes a detection branch and a tracking branch; wherein the detection branch is used for target classification and positioning, and the tracking branch is used for target trajectory tracking.

[0067] In actual application, accurate detection and continuous tracking of dynamic targets can be achieved through the coordinated cooperation of the two subtasks: detection branch and tracking branch.

[0068] S3. Based on infrared images and target tracking information, a knowledge vector library is constructed using the retrieval enhanced generation method.

[0069] Step S3 specifically includes the following steps.

[0070] S31. Based on the retrieval enhancement generation method, the infrared image is converted into an infrared feature vector using an encoding function.

[0071] S32. Construct a knowledge vector library based on the correspondence between the infrared feature vector and the target tracking information.

[0072] S4. Use the knowledge vector library to perform multimodal feature fusion and save the infrared data after multimodal feature fusion.

[0073] Step S4 specifically includes the following steps.

[0074] S41. Retrieve the most relevant data features by performing similarity matching between the infrared image as a query vector and the infrared feature vector stored in the knowledge vector library.

[0075] Furthermore, the similarity matching is cosine similarity matching.

[0076] S42. The retrieved infrared feature vector is fused with the target tracking information through the generation module of the knowledge vector library to obtain a multimodal output result.

[0077] S43. Save the multimodal output results as infrared data.

[0078] In actual application, the collected data is stored locally through the knowledge vector library retrieval-augmented generation (RAG). The infrared data R can be converted into a vector representation V using an encoding function, and the knowledge vector library is constructed in combination with relevant metadata (such as time and location tags). During retrieval, the query vector q is matched with the infrared feature vector stored in the knowledge base for similarity (such as cosine similarity) to retrieve the most relevant data features. Subsequently, the generation module fuses the retrieved infrared features with other modal information (such as text, visual features, etc.) to generate the final multimodal output results. The above method provides a unified and flexible solution for the efficient preservation and utilization of infrared data, which is particularly suitable for dynamic and complex multimodal scenes.

[0079] The first step is infrared data encoding: convert infrared image data Convert to feature vector representation , through the encoding function The conversion formula is as follows.

[0080] .

[0081] Step 2: Knowledge vector library construction :The feature vector of infrared data It is combined with its corresponding metadata m (such as timestamp, location tag, etc.) to form an entry in the knowledge vector library. The formula of the knowledge vector library is as follows.

[0082] .

[0083] The third step is the multimodal retrieval mechanism: the query vector q (which may contain text, image or infrared features) is compared with the feature vector in the knowledge vector library by calculating the similarity (such as using cosine similarity). , to retrieve the most relevant infrared data.

[0084] .

[0085] Step 4: Generate module integration: integrate the retrieved infrared data features and multimodal features (other modal features) Fusion is performed and the generating function is Output target generation result, the formula is as follows.

[0086] .

[0087] The technical effects of this application are as follows.

[0088] This application optimizes the YOLOv8 model to construct a target tracking model, and uses the target tracking model to obtain target tracking information, thereby improving the accuracy of target tracking model detection and tracking; constructs and uses a knowledge vector library for multimodal feature fusion, and saves infrared data, thereby realizing the localization and efficient preservation of infrared data in dynamic and complex multimodal scenes. Specifically, this application realizes the effective fusion of multi-scale features through the PANet structure. By introducing the variable kernel convolution AKConv module and the dynamic adjustment of the sampling point position of the convolution kernel, the network can flexibly capture detail information according to the geometric shape of the input feature. Significant improvements are achieved in the detection of low-contrast and long-distance small targets, and the characterization ability of key feature areas is enhanced. The introduction of the DASI dimensional perception selective integration module effectively solves the problem of feature information loss and background clutter in infrared small target detection. The introduction of the adaptive feature fusion module dynamically adjusts the feature fusion strategy according to the size, shape, motion state and other information of the target, thereby maintaining a high detection accuracy under various target scales and shapes, and enhancing the infrared small target detection capability. The dual-branch network is used to achieve accurate detection and stable tracking of dynamic targets, and the real-time reasoning verification of edge devices ensures the practical application performance of the model. By using a parallel convolutional network, the target can be effectively detected and tracked in real time in a dark environment with complex foreground and background, unclear contours of positive and negative samples, and missing texture information; in a dim environment, even if the target is obstructed, the target area cannot be continuously and accurately identified in real time for dynamic obstacles, and the trajectory tracking does not deviate; by associating the feature information of the previous and next frames, a position-dependent relationship is established for the non-local domain features, and the spatial channel features are rearranged to map the deep feature similar pixel representations.

[0089] The target tracking model in this application is a target detection and tracking algorithm applied to indoor and outdoor night infrared environments. In order to achieve effective detection and tracking of fuzzy contour features in dark environments, a real-time detection algorithm for infrared targets based on the spatiotemporal association mechanism of a deep learning network SLD-Yolov8 is proposed. The repeated cross attention module RCCA and the changeable kernel convolution AKConv module are designed to optimize the bottleneck layer Bottleneck CSP of the original Yolov8 network. This application constructs an infrared multi-target tracking and detection network configuration with a spatiotemporal association attention mechanism in dim environments. By associating the feature information of the previous and next frames, a position-dependent connection is established for the non-local domain features, and the spatial channel features are rearranged to represent and map similar pixels of deep features. Based on the above configuration, the feature information of dynamic obstacles in unknown dark environments can be effectively identified, and the tracking trajectory is more stable.

[0090] Embodiment 2, the present application also provides a computer system, which may be a server or a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer system includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer system is used to provide computing and control capabilities. The memory of the computer system includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer system is used to store video tag processing data. The input / output interface of the computer system is used to exchange information between the processor and an external device. The communication interface of the computer system is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment is implemented.

[0091] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer system to which the solution of the present application is applied. The specific computer system may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0092] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment, characterized in that: The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment includes: Acquire an infrared image in a dim environment; the dim environment is an environment under the condition that visibility is lower than a preset value; Input the infrared image into the target tracking model and output the target tracking information; the target tracking model is optimized based on the YOLOv8 model; the target tracking information includes: the location, category and trajectory information of all detected targets; Based on infrared images and target tracking information, a knowledge vector library is constructed using the retrieval enhancement generation method; The knowledge vector library is used to fuse multimodal features and save the infrared data after multimodal feature fusion.

2. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 1, characterized in that: The target tracking model includes a backbone network, a neck network and a head network connected in sequence; the structure of the backbone network is a CSPDarknet53 structure; the backbone network includes an offset learning module, a BottleneckCSP structure and a DASI module connected in sequence; wherein the BottleneckCSP structure includes a convolution operation module, an AKConv module and an RCCA module connected in sequence; the neck network is a PANet structure; and the head network includes multiple multi-scale dynamic detection heads.

3. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 2, characterized in that: Input the infrared image into the target tracking model and output the target tracking information, including: By using the offset learning module and the convolution operation module, the multi-scale initial feature map of the infrared image is extracted according to the offset of the convolution kernel sampling point; The neck network based on the PANet structure uses a path aggregation mechanism to fuse features at different levels in the multi-scale initial feature map to obtain a fused multi-scale feature map. According to the fused multi-scale feature map, the variable and convolution AKConv module is used to extract geometric shape features through convolution kernels of variable shapes and sizes to obtain variable kernel enhanced feature maps; The RCCA module is used to calculate the relationship between the target feature pixel point in the variable kernel enhanced feature map and all other points in the feature map, and the features of the target pixel point are weighted according to the relationship to obtain the weighted variable kernel enhanced feature map; The DASI module is used to perform feature fusion on the weighted variable kernel enhanced feature map in an adaptive screening manner to obtain a dimension-aware fused feature map. Multiple multi-scale motion detection heads are used to extract multi-scale semantic information from the dimension-aware fusion feature map to obtain a multi-scale motion detection feature map. The target detection and target tracking are performed on the multi-scale dynamic detection feature map obtained by the dual-branch network to obtain the target tracking information; the dual-branch network includes a detection branch and a tracking branch; wherein the detection branch is used for the classification and positioning of the target, and the tracking branch is used for the trajectory tracking of the target.

4. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 3 is characterized in that: The multi-scale dynamic detection head is used to process the multi-scale semantic information of the dimension-aware fusion feature map to obtain the multi-scale dynamic detection feature map, which specifically includes: Using a five-frame difference model combined with dynamic region and depth estimation, the pixel intensity differences of the dimension-aware fusion feature maps of consecutive frames are calculated to generate a difference image. The target area of ​​the differential image is determined by binarization to obtain a binary image; A calibration method is used based on the camera intrinsic parameter matrix to map the pixel coordinates in the binary image to the normalized camera coordinate system; The binary image in the camera coordinate system is mapped to the world coordinate system to obtain a multi-scale dynamic detection feature map.

5. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 4, characterized in that: The formula of the five-frame difference model is as follows: ； in, The difference image after five frames of difference, For the Frame image, For the Frame image, For the Frame image.

6. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 1, characterized in that: Based on infrared images and target tracking information, a knowledge vector library is constructed using the retrieval enhancement generation method, which includes: Based on the retrieval enhancement generation method, the infrared image is converted into an infrared feature vector using a coding function; A knowledge vector library is constructed based on the correspondence between infrared feature vectors and target tracking information.

7. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 1, characterized in that: Use the knowledge vector library to perform multimodal feature fusion and save the infrared data after multimodal feature fusion, including: The most relevant data features are retrieved by performing similarity matching between the infrared image as a query vector and the infrared feature vectors stored in the knowledge vector library; The retrieved infrared feature vector is fused with the target tracking information through the generation module of the knowledge vector library to obtain a multi-modal output result; Save the multimodal output results as infrared data.

8. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 7, characterized in that: The similarity matching is cosine similarity matching.

9. The method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment according to claim 1, characterized in that: The infrared image is an image captured by a binocular infrared camera; the model of the binocular infrared camera module is InfiSenseS2640-P20004S00.

10. A computer system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing a knowledge base for infrared target tracking and retrieval in a dim environment as described in any one of claims 1 to 9.

Images