Track station target tracking method and device based on visible light-infrared image

Abstract

The present application relates to the field of track perception, and provides a track platform target tracking method and device based on a visible light-infrared image, the method comprising: based on a zero sample image recovery strategy, pre-processing a first visible light image to obtain a second visible light image; the zero sample image recovery strategy comprises input initialization, back projection guidance and noise injection mechanism; inputting the second visible light image and an infrared image into a cross-modal regulation attention model to obtain a predicted target position; the cross-modal regulation attention model comprises a visible light-infrared tracker based on a Transformer, a Transformer block, a cross-modal regulation attention unit and a tracking prediction head. The present application solves the problem of low target tracking accuracy of visible light images in complex environments in the prior art, and improves the robustness and accuracy of target tracking.

Description

Technical Field

[0001] This invention relates to the field of track sensing technology, and in particular to a method and apparatus for tracking track platform targets based on visible light-infrared images. Background Technology

[0002] With the rapid development of rail transit, safety monitoring and target tracking at rail stations have become crucial for ensuring passenger safety and operational efficiency. Traditional rail station monitoring systems primarily rely on visible light cameras. However, visible light images are easily interfered with in complex environments (such as insufficient light, fog, and nighttime), leading to decreased image quality and affecting the accuracy of target tracking. To address this issue, infrared imaging technology has been gradually introduced into rail station monitoring systems. Infrared images can provide clear target information in low light and adverse weather conditions, but their resolution is low and they lack color information.

[0003] In image restoration, traditional methods primarily rely on signal processing and optimization techniques, such as model-based methods (e.g., Wiener filtering, inverse filtering), variational methods, and sparse representations. These methods model image degradation using mathematical models and restore the image through optimization techniques. With the development of deep learning, deep learning-based image restoration methods have gradually become mainstream. These methods are mainly divided into task-specific deep neural networks and zero-shot image restoration. Task-specific deep neural networks are trained specifically for each image restoration task, but these networks typically perform well on specific tasks, are highly sensitive to changes in the observed model during testing, and have limited generalization ability. Summary of the Invention

[0004] This invention provides a method and apparatus for tracking track platform targets based on visible light-infrared images, which solves the problems of visible light images being easily interfered with in complex environments and low target tracking accuracy in the prior art, and improves the robustness and accuracy of target tracking.

[0005] This invention provides a method for tracking targets on a railway station based on visible light-infrared images, comprising the following steps:

[0006] Based on a zero-shot image restoration strategy, a first visible light image is preprocessed to obtain a second visible light image; the zero-shot image restoration strategy includes input initialization, back-projection guidance, and noise injection mechanisms.

[0007] The second visible light image and infrared image are input into the cross-modal control attention model to obtain the predicted target position; the cross-modal control attention model includes a Transformer-based visible light-infrared tracker, a Transformer block, a cross-modal control attention unit, and a tracking prediction head.

[0008] According to the present invention, a target tracking method for railway stations based on visible light-infrared images is provided. Based on a zero-sample image restoration strategy, a first visible light image is preprocessed to obtain a second visible light image. Specifically, the method includes: obtaining the pseudo-inverse of the first visible light image; initializing the image based on the pseudo-inverse to generate a target image starting point; performing image restoration based on the target image starting point to obtain the second visible light image; accelerating the image restoration process through back-projection guidance; and reducing the number of iterations in the image restoration process through a noise injection mechanism.

[0009] According to the present invention, a method for tracking a target on a railway station based on visible light-infrared images is provided. The step of inputting the second visible light image and the infrared image into a cross-modal attention model to obtain the predicted target position specifically includes: inputting the second visible light image and the infrared image into a Transformer-based visible light-infrared tracker to obtain a marker sequence; the second visible light image corresponds to the infrared image; inputting the marker sequence into a Transformer block for processing to obtain a feature sequence; inputting the feature sequence into a cross-modal attention unit to obtain an updated feature sequence; and merging the updated feature sequence along the channel dimension and inputting it into a tracking prediction head to output the predicted target position.

[0010] According to the present invention, a method for tracking track platform targets based on visible light-infrared images is provided. The step of inputting a second visible light image and an infrared image into a Transformer-based visible light-infrared tracker to obtain a marker sequence specifically includes: inputting the second visible light image and the infrared image into the Transformer-based visible light-infrared tracker and dividing them into image blocks of the same size; flattening the divided image blocks to obtain an image block sequence; determining template features and search area features based on the image block sequence; and concatenating the template features and search area features to obtain a marker sequence.

[0011] According to the present invention, a method for tracking track station targets based on visible light-infrared images is provided. The step of inputting the feature sequence into a cross-modal modulation attention unit to obtain an updated feature sequence specifically includes: inputting the feature sequence into the cross-modal modulation attention unit to obtain a query matrix and keyword matrix for the visible light modality, and a query matrix and keyword matrix for the infrared modality; determining an original visible light correlation map based on the query matrix and keyword matrix for the visible light modality; determining an original infrared correlation map based on the query matrix and keyword matrix for the infrared modality; determining aggregation information based on the original visible light correlation map and the original infrared correlation map; performing an attention operation on the aggregation information to obtain an initial visible light modulation correlation map and an initial infrared modulation correlation map; determining a final visible light image attention map based on the original visible light correlation map and the initial visible light modulation correlation map; determining a final infrared image attention map based on the original infrared correlation map and the initial infrared modulation correlation map; and determining the corresponding updated feature sequence based on the final visible light image attention map and the final infrared image attention map.

[0012] According to the present invention, a track station target tracking method based on visible light-infrared images is provided. In the process of inputting the feature sequence into a cross-modal modulation attention unit to obtain an updated feature sequence, a cooperative label elimination strategy is used to eliminate labels irrelevant to the target.

[0013] According to the present invention, a method for tracking railway station targets based on visible light-infrared images, wherein the method for eliminating targets-irrelevant markers through a cooperative marker elimination strategy specifically includes: determining a defined search region in the current frame image; extracting marker features from the search region, the marker features including markers from visible light and infrared modes; calculating the attention weight between a marker in each search region and a template marker; obtaining a fused weight based on the attention weights of the visible light and infrared modes; sorting all markers in the search region according to the fused weight, selecting the top k markers with the highest weights as target candidates, and deleting markers other than the top k markers.

[0014] The present invention also provides a track station target tracking device based on visible light-infrared images, comprising the following modules:

[0015] The preprocessing module is used to preprocess the first visible light image based on a zero-sample image restoration strategy to obtain a second visible light image; the zero-sample image restoration strategy includes input initialization, back-projection guidance, and noise injection mechanisms.

[0016] The target tracking module is used to input the second visible light image and infrared image into the cross-modal control attention model to obtain the predicted target position; the cross-modal control attention model includes a Transformer-based visible light-infrared tracker, a Transformer block, a cross-modal control attention unit, and a tracking prediction head.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the track station target tracking method based on visible light-infrared images as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the track station target tracking method based on visible light-infrared images as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the track station target tracking method based on visible light-infrared images as described above.

[0020] This invention provides a method and apparatus for tracking railway station targets based on visible light-infrared images, which offers the following advantages: A zero-sample image restoration strategy is used to preprocess a first visible light image to obtain a high-quality second visible light image, effectively solving the problem of visible light images being susceptible to interference and quality degradation in complex environments. Simultaneously, a Transformer-based visible light-infrared tracker and a cross-modal attention unit are used to fuse the second visible light image with the infrared image for multimodal features, enhancing the robustness and accuracy of target tracking. Therefore, this invention achieves efficient image restoration with fewer iterations and effectively fuses multimodal information from visible light and infrared images, significantly improving the accuracy and efficiency of railway station target tracking. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is one of the flowcharts of the track station target tracking method based on visible light-infrared images provided by the present invention.

[0023] Figure 2This is the second flowchart of the track station target tracking method based on visible light-infrared images provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the cross-modal attention control model architecture provided by the present invention.

[0025] Figure 4 This is a schematic diagram of the track platform target tracking device based on visible light-infrared images provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Before the advent of deep learning, image restoration primarily relied on signal processing and optimization techniques. These methods included model-based approaches, variational methods, and sparse representations. Model-based methods, such as Wiener filtering and inverse filtering, model image degradation using mathematical models and restore the image through optimization techniques. Variational methods optimize the image restoration problem by defining energy functions (such as total variational regularization). Sparse representations utilize the sparsity of images (such as wavelet transform) to restore images, assuming that the image is sparse in a certain transform domain. With the development of deep learning, many new technologies have emerged in the field of image restoration. Deep learning-based methods are mainly divided into task-specific deep neural networks and zero-shot image restoration. Task-specific deep neural networks train a dedicated deep neural network for each image restoration task (such as denoising, super-resolution, deblurring, etc.), but these networks typically perform well on specific tasks, are very sensitive to changes in the observed model during testing, and have limited generalization ability. Zero-shot image restoration methods do not rely on task-specific trained networks. Instead, they use pre-trained generative models (such as diffusion models, generative adversarial networks, etc.) as signal priors. By adapting the observation model during testing, they avoid the limitations of task-specific networks.

[0029] In target tracking, existing Transformer-based visible-infrared tracking methods extract unimodal features through self-attention and enhance multimodal feature interaction through cross-modal attention. However, this method independently calculates correlations in self-attention, making it susceptible to low-quality data and resulting in inaccurate correlation weights, thus limiting tracking performance.

[0030] It is evident that there is an urgent need to provide a robust target tracking method that can effectively address various image degradation tasks and integrate multimodal information.

[0031] The present invention proposes a visible light-infrared target tracking method for rail stations that employs a zero-sample image restoration strategy. This method can be applied to the processing of various image degradation tasks and effectively utilizes multimodal information.

[0032] This invention provides a visible-infrared target tracking method for railway stations employing a zero-sample image restoration strategy. This method can be applied to various image degradation tasks and effectively utilizes multimodal information. For example... Figure 1 The diagram shows the overall flowchart of a visible light-infrared track station target tracking method using a zero-shot image restoration strategy provided by the present invention. The method includes the following steps: The input visible light image is first processed by the zero-shot image restoration strategy to obtain a visible light image without degradation effect. Then, it is input together with the infrared image into a cross-modal modulation attention model. After processing by the Transformer block, a feature sequence is obtained. Then, it is processed by the cross-modal modulation attention unit. At the same time, a cooperative label elimination strategy is applied in a specific layer to obtain an updated feature sequence. Then, it is input into the tracking prediction head to output the predicted position of the target.

[0033] The advantages of this invention compared to previous technologies can be summarized as follows:

[0034] (1) The present invention adopts a zero-sample image restoration strategy, which can efficiently complete tasks such as image super-resolution, deblurring and restoration with fewer iterations, significantly reducing computational complexity and improving restoration quality.

[0035] (2) By using a cross-modal attention control mechanism, this invention integrates visible light and infrared multimodal information, enhances feature interaction and correlation calculation, and improves the robustness and accuracy of target tracking, especially in complex scenarios.

[0036] The following is combined with Figures 2-5 The embodiments of the present invention are described in detail.

[0037] Figure 2 This is a flowchart illustrating the target tracking method for railway stations based on visible light-infrared images provided by the present invention, as shown below. Figure 2 As shown, the method includes the following steps:

[0038] S210. Based on the zero-shot image restoration strategy, the first visible light image is preprocessed to obtain the second visible light image. The zero-shot image restoration strategy includes input initialization, backprojection guidance, and noise injection mechanisms.

[0039] According to the present invention, a target tracking method for railway stations based on visible light-infrared images is provided. Based on a zero-sample image restoration strategy, a first visible light image is preprocessed to obtain a second visible light image. Specifically, the method includes: obtaining the pseudo-inverse of the first visible light image; initializing the image based on the pseudo-inverse to generate a target image starting point; performing image restoration based on the target image starting point to obtain the second visible light image; accelerating the image restoration process through back-projection guidance; and reducing the number of iterations in the image restoration process through a noise injection mechanism.

[0040] Specifically, this invention designs a zero-shot image restoration strategy based on a consistency model, which can effectively complete tasks such as image super-resolution, deblurring, and restoration with fewer iterations. This strategy includes input initialization, back-projection guidance, and a noise injection mechanism. Input initialization is used to preprocess the first visible light image, providing a suitable starting point for subsequent image restoration; back-projection guidance is used to adjust the intermediate results of the image restoration process; and the noise injection mechanism can effectively utilize noise to reduce the number of iterations.

[0041] Specifically, zero-sample image restoration strategies based on the consistency model include: input initialization, backprojection guidance, and noise injection mechanism.

[0042] The goal of an image restoration task is to recover from a degraded first visible light image. Recovering high-quality images ,in and The relationship between them can usually be represented by a linear model:

[0043]

[0044] in It is a measurement matrix. It is additive noise.

[0045] Initialization refers to the initialization of the first visible light image before the recovery algorithm begins. This is a preprocessing procedure. Its purpose is to provide a suitable starting point for subsequent recovery operations. This improves the efficiency and recovery quality of the algorithm.

[0046] Traditional diffusion-based image restoration methods typically use pure noise initialization. Specifically, the initialization input... It is usually a Gaussian noise vector, i.e. ,in Represents a Gaussian distribution. Represents the time step. This represents the identity matrix. This initialization method ignores the first visible light image. The target image contained The information depends solely on the model's generative capabilities. To ensure the generated image matches the first visible light image... To maintain consistency, the diffusion model introduces data fidelity guidance during the iteration process; however, this guidance mechanism usually requires multiple iterations (i.e., numerous neural function evaluations) to achieve good results.

[0047] The consistency model attempts to utilize the first visible light image more directly during initialization. Information. Specifically, initializing input. Through the first visible light image pseudo-reversal Initialization is performed, that is ,in For measurement matrix The pseudo-inverse is defined as , It is the initial noise level, measurement matrix This method represents the process of image degradation (such as blurring, downsampling, etc.). It utilizes the first visible light image. The structural information within the image provides a more accurate representation of the target image for the restoration process. The starting point. A certain level of noise will be injected during this process. However, noise injection not only preserves the model's generative ability but also provides the necessary "degrees of freedom" for the subsequent restoration process, enabling the model to better adapt to different restoration tasks. This initialization method reduces the burden on subsequent iterations, allowing the consistency model to achieve efficient image restoration with fewer neural function evaluations.

[0048] Back-projection guidance is used to accelerate the image restoration process; its core is to utilize the first visible light image. and measurement matrix pseudo-reversal The intermediate results during the recovery process are adjusted through backprojection. Specifically, the update formula guided by backprojection is:

[0049]

[0050] in, Indicates the time step Image recovery at that time This represents the initial estimated image at time step τn. It is a scaling factor used to control the weight of data fidelity terms. Represents noise level; It is the data fidelity term guided by back projection, (AA) T ) -1 / 2 Let ||·||2 denote the square root of the pseudo-inverse of the measurement matrix A, and let ||·||2 denote the L2 norm, used to calculate the Euclidean distance between vectors. It is the gradient of the data fidelity term. Ax represents the pseudo-inverse of measurement matrix A, Ax represents the degradation operation performed on the restored image x through measurement matrix A, and y represents the observation data, i.e., the first visible light image.

[0051] The noise injection mechanism includes noise level decoupling and noise injection segmentation, which allows the image to utilize noise more effectively during the restoration process and reduces the number of iterations.

[0052] Noise level decoupling refers to separating the noise level used in the denoising operation from the noise level in the noise injection step, so that the two processes can be optimized and adjusted independently.

[0053] Noise injection segmentation involves dividing injected noise into random noise and estimated noise. Random noise This refers to unpredictable noise added during image restoration or generation. It helps the model explore different parts of the data distribution, increases sample diversity, and prevents the model from prematurely converging to a local optimum. It follows a Gaussian distribution. Noise estimation It is calculated based on the difference between the currently estimated signal and the true signal. It can help the model converge to the true signal more quickly, and is expressed as...

[0054]

[0055] Then noise injection can be represented as

[0056]

[0057] in It is a hyperparameter used to weigh the random noise and the estimated noise.

[0058] Visible light images are processed using a zero-sample image restoration strategy to obtain images without degradation effects, which can be used in conjunction with infrared images to predict the location of targets.

[0059] S220. Input the second visible light image and infrared image into the cross-modal control attention model to obtain the predicted target position. The cross-modal control attention model includes a Transformer-based visible light-infrared tracker, a Transformer block, a cross-modal control attention unit, and a tracking prediction head.

[0060] Specifically, this invention designs a visible light-infrared orbital platform target tracking method, which is implemented through a cross-modal attention model. For example... Figure 3 As shown, the model mainly includes a Transformer-based visible-infrared tracker and a cross-modal attention unit. It also proposes a cross-modal attention mechanism and a cooperative label removal strategy. By using a unified attention model, it simultaneously performs single-modal autocorrelation, cross-modal feature interaction and search-template correlation calculation to improve tracking performance.

[0061] According to the present invention, a track station target tracking method based on visible light-infrared images is provided. The method inputs a second visible light image and an infrared image into a cross-modal attention model to obtain the predicted target position. Specifically, the method includes: inputting the second visible light image and the infrared image into a Transformer-based visible light-infrared tracker to obtain a marker sequence; corresponding the second visible light image and the infrared image; inputting the marker sequence into a Transformer block for processing to obtain a feature sequence; inputting the feature sequence into a cross-modal attention unit to obtain an updated feature sequence; and merging the updated feature sequence along the channel dimension and inputting it into a tracking prediction head to output the predicted target position.

[0062] According to the present invention, a track station target tracking method based on visible light-infrared images is provided. The method inputs a second visible light image and an infrared image into a Transformer-based visible light-infrared tracker to obtain a marker sequence. Specifically, the method includes: inputting the second visible light image and the infrared image into the Transformer-based visible light-infrared tracker and dividing them into image blocks of the same size; flattening the divided image blocks to obtain an image block sequence; determining template features and search area features based on the image block sequence; and concatenating the template features and search area features to obtain a marker sequence.

[0063] Specifically, the model mainly includes a Transformer-based visible-infrared tracker, which mainly includes a Transformer block, a cross-modal controlled attention unit, and a tracking prediction head.

[0064] The tracker consists of two branches: visible light and infrared. These two branches share parameters but handle different modes independently.

[0065] Given input visible light and infrared template images and search area image pairs ,in , Represents the image height. , This represents the image height, and 3 represents 3 channels.

[0066] First, divide these images into groups of size [size missing]. The image patches are then flattened to obtain an image patch sequence. and ,in , representing the number of blocks in the template and the search box, respectively. Using parameters... and learnable positional encoding and Block embedding layer to obtain template features , and search region features , As shown in the following formula:

[0067]

[0068] These features are then concatenated to obtain the label sequence. , .

[0069] According to the present invention, a method for tracking track station targets based on visible light-infrared images is provided. The method inputs a feature sequence into a cross-modal modulation attention unit to obtain an updated feature sequence. Specifically, the method includes: inputting the feature sequence into the cross-modal modulation attention unit to obtain a query matrix and keyword matrix for the visible light mode, and a query matrix and keyword matrix for the infrared mode; determining an original visible light correlation map based on the query matrix and keyword matrix for the visible light mode; determining an original infrared correlation map based on the query matrix and keyword matrix for the infrared mode; determining aggregation information based on the original visible light correlation map and the original infrared correlation map; performing an attention operation on the aggregation information to obtain an initial visible light modulation correlation map and an initial infrared modulation correlation map; determining a final visible light image attention map based on the original visible light correlation map and the initial visible light modulation correlation map; determining a final infrared image attention map based on the original infrared correlation map and the initial infrared modulation correlation map; and determining the corresponding updated feature sequence based on the final visible light image attention map and the final infrared image attention map.

[0070] Specifically, the flag sequence Delivered to Transformer block of the layer The forward propagation process is processed as follows:

[0071] , This is the feature sequence output by the last Transformer block. , Input is fed into the cross-modal modulated attention unit.

[0072] The query and keyword matrix for visible light modes can be represented as:

[0073]

[0074] in , These represent the linear projection weights of the query and the key, respectively. For the visible light branch, its features generate a correlation graph. The process can be represented as:

[0075]

[0076] For the infrared branch, the same processing is used to obtain... . and They are all divided into four parts and These components play different roles in tracking prediction. Each component is simplified and named TT, TS, ST, and SS based on the query key pairs used to calculate relevance. ST is a special component that controls the information flow from the template to the search box and has a significant impact on the tracking results. Due to the spatiotemporally aligned multimodal image pairs, STs within different branches exhibit significant correlations.

[0077] To achieve adaptive relevance modulation, a cross-modal attention modulation mechanism is designed, utilizing the relevance graphs of two modalities to enhance the interaction within the cross-modal attention modulation unit. The purpose of this mechanism is to modulate the ST (Speed ​​Target) while also considering the SS (Speed ​​Target) to adjust the final attention graph. Specifically, the aggregated information of the two branches is as follows:

[0078]

[0079] Where LN represents the normalization layer. It is a learnable linear projection weight used to embed the correlation between the two branches. Then we... Perform attention-based operations to obtain regulatory correlation graphs. .

[0080]

[0081] in It is the number of template tags. and This represents the linear projection weights of the query and key in the cross-modal modulation attention unit. Next, we will examine the initial modulation correlation map... and Separate from and .

[0082]

[0083] in These are learnable linear projection weights.

[0084] Finally, what will be obtained Correlation plot with the original The summations yield the final modulation correlation map. This generates the final visible light image attention map A. rgb The process can be described as follows:

[0085] ,

[0086] Where C represents the dimension of the flag.

[0087] The cross-modal attention unit has a symmetric structure, meaning that parameters at corresponding positions in the two branches are shared. The updated feature sequence is obtained after processing by the cross-modal attention unit. , .

[0088] These features are merged along the channel dimension and fed into the tracking prediction head. The predicted bounding box is obtained. The bounding box position is the location of the target.

[0089] According to the present invention, a target tracking method for railway stations based on visible light-infrared images eliminates targets irrelevant to the target by means of a cooperative label elimination strategy during the process of inputting the feature sequence into a cross-modal modulation attention unit to obtain an updated feature sequence.

[0090] According to the present invention, a target tracking method for railway stations based on visible light-infrared images is provided. This method eliminates targets-irrelevant markers through a cooperative marker elimination strategy. Specifically, the method includes: determining a defined search region in the current frame image; extracting marker features from the search region, including markers from visible light and infrared modes; calculating the attention weight between each marker in the search region and a template marker; obtaining a fused weight based on the attention weights of the visible light and infrared modes; and sorting all markers in the search region according to the fused weights, selecting the top k markers with the highest weights as target candidates, and deleting all markers except the top k markers.

[0091] Specifically, a collaborative label removal strategy is proposed and applied to a specific layer of the cross-modal attention unit to improve tracking efficiency and accuracy. The core idea of ​​this strategy is to combine the attention weights of both visible and infrared modes to make judgments and eliminate non-target labels.

[0092] First, features are extracted from the search region, including labels from both visible and infrared modalities. For each search region label, its relevance, or attention weight, to a template label is calculated. These weights reflect the similarity between each label and the target. Since the visible and infrared modalities provide complementary information, a cooperative label elimination strategy combines the attention weights of the two modalities to achieve a more accurate distinction between targets and non-targets. The search region labels are then ranked using the fused weights, and the top k labels with the highest weights are selected as target candidates. Labels with lower weights are excluded, as these are considered less relevant to the target.

[0093] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows:

[0094] (1) Improved efficiency of image restoration: The present invention adopts a zero-sample image restoration strategy, which significantly reduces the number of iterations for tasks such as image super-resolution, deblurring and restoration, improves computational efficiency, and improves image restoration quality.

[0095] (2) Enhanced robustness of target tracking: By using a cross-modal attention mechanism, this invention enhances the interaction and correlation calculation of visible light and infrared image features, thereby improving the accuracy and robustness of target tracking.

[0096] The track station target tracking device based on visible light-infrared images provided by the present invention will be described below. The track station target tracking device based on visible light-infrared images described below can be referred to in correspondence with the track station target tracking method based on visible light-infrared images described above.

[0097] like Figure 4 The image shown is a track platform target tracking device based on visible light-infrared images provided by the present invention, comprising:

[0098] Preprocessing module 410 is used to preprocess the first visible light image based on a zero-shot image restoration strategy to obtain a second visible light image; the zero-shot image restoration strategy includes input initialization, back-projection guidance and noise injection mechanism;

[0099] The target tracking module 420 is used to input the second visible light image and infrared image into the cross-modal modulation attention model to obtain the predicted target position; the cross-modal modulation attention model includes a Transformer-based visible light-infrared tracker, a Transformer block, a cross-modal modulation attention unit, and a tracking prediction head.

[0100] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a track station target tracking method based on visible light-infrared images. The method includes: preprocessing a first visible light image based on a zero-sample image restoration strategy to obtain a second visible light image; the zero-sample image restoration strategy includes input initialization, back-projection guidance, and noise injection mechanisms; inputting the second visible light image and the infrared image into a cross-modal control attention model to obtain the predicted target position; the cross-modal control attention model includes a Transformer-based visible light-infrared tracker, a Transformer block, a cross-modal control attention unit, and a tracking prediction head.

[0101] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the track station target tracking method based on visible light-infrared images provided by the above methods. The method includes: preprocessing a first visible light image based on a zero-sample image restoration strategy to obtain a second visible light image; the zero-sample image restoration strategy includes input initialization, back-projection guidance, and noise injection mechanisms; inputting the second visible light image and the infrared image into a cross-modal modulation attention model to obtain the predicted target position; the cross-modal modulation attention model includes a Transformer-based visible light-infrared tracker, a Transformer block, a cross-modal modulation attention unit, and a tracking prediction head.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the track station target tracking method based on visible light-infrared images provided by the above methods. The method includes: preprocessing a first visible light image based on a zero-sample image restoration strategy to obtain a second visible light image; the zero-sample image restoration strategy includes input initialization, back-projection guidance, and noise injection mechanisms; inputting the second visible light image and the infrared image into a cross-modal modulation attention model to obtain the predicted target position; the cross-modal modulation attention model includes a Transformer-based visible light-infrared tracker, a Transformer block, a cross-modal modulation attention unit, and a tracking prediction head.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of tracking a platform target based on visible-infrared images, characterized in that, The method comprises the following steps: based on the zero sample image restoration strategy, the first visible light image is preprocessed to obtain the second visible light image; The zero sample image restoration strategy includes input initialization, back projection guidance and noise injection mechanism; The second visible light image and the infrared image are input into the cross-modal regulation attention model to obtain the predicted target position; the cross-modal regulation attention model includes a visible light-infrared tracker based on Transformer, a Transformer block, a cross-modal regulation attention unit and a tracking prediction head; The second visible light image and the infrared image are input into the cross-modal regulation attention model to obtain the predicted target position, specifically including: inputting the second visible light image and the infrared image into the visible light-infrared tracker based on Transformer to obtain a mark sequence; the second visible light image corresponds to the infrared image; inputting the mark sequence into the Transformer block for processing to obtain a feature sequence; inputting the feature sequence into the cross-modal regulation attention unit to obtain an updated feature sequence; after merging the updated feature sequence along the channel dimension, inputting it into the tracking prediction head to output the position of the predicted target; The feature sequence is input into the cross-modal regulation attention unit to obtain the updated feature sequence, specifically including: inputting the feature sequence into the cross-modal regulation attention unit to obtain the query matrix and the keyword matrix of the visible light mode, and the query matrix and the keyword matrix of the infrared mode; determine the original visible light correlation graph according to the query matrix and the keyword matrix of the visible light mode; determine the original infrared correlation graph according to the query matrix and the keyword matrix of the infrared mode; determine the corresponding updated feature sequence according to the original visible light correlation graph and the original infrared correlation graph; Determine the corresponding updated feature sequence according to the original visible light correlation graph and the original infrared correlation graph, specifically including: determining the aggregation information according to the original visible light correlation graph and the original infrared correlation graph; performing attention operation on the aggregation information to obtain an initial visible light regulation correlation graph and an initial infrared regulation correlation graph; determine the final visible light image attention map according to the original visible light correlation graph and the initial visible light regulation correlation graph; determine the final infrared image attention map according to the original infrared correlation graph and the initial infrared regulation correlation graph; determine the corresponding updated feature sequence according to the final visible light image attention map and the final infrared image attention map.

2. The visible-infrared image-based tracking method of a docking station target according to claim 1, characterized in that, Based on the zero sample image restoration strategy, the first visible light image is preprocessed to obtain the second visible light image, specifically including: Obtain the pseudo-inverse of the first visible light image, initialize the target image starting point according to the pseudo-inverse of the first visible light image; Based on the target image starting point, the image is restored to obtain the second visible light image; Accelerate the image restoration process through back projection guidance; Reduce the number of iterations in the image restoration process through the noise injection mechanism. 3.The visible-infrared image-based TP tracking method according to claim 1, wherein, The second visible light image and the infrared image are input into a Transformer-based visible-infrared tracker to obtain a mark sequence, and the method specifically comprises the following steps: The second visible light image and the infrared image are input into a Transformer-based visible-infrared tracker and are respectively divided into image blocks of the same size; The divided image blocks are flattened to obtain an image block sequence; Template features and search area features are determined according to the image block sequence; The template features and the search area features are connected to obtain a mark sequence. 4.The visible-infrared image-based TP tracking method according to claim 1, wherein, In the process of inputting the feature sequence into the cross-modal regulation attention unit to obtain an updated feature sequence, irrelevant marks are eliminated through a cooperative mark elimination strategy.

5. The visible-infrared image-based tracking method of a platform target according to claim 4, characterized in that, The irrelevant marks are eliminated through the cooperative mark elimination strategy, and the method specifically comprises the following steps: A search area is determined in a current frame image; Mark features are extracted from the search area, and the mark features include marks from visible light and infrared modalities; For each mark of the search area, an attention weight between the mark and a template mark is calculated; According to the attention weights of the visible light and infrared modalities, a fused weight is obtained; According to the fused weight, all marks in the search area are sorted, and the top k marks with the highest weights are selected as target candidates, and marks other than the top k marks are deleted.

6. A track platform target tracking apparatus based on a visible-infrared image, characterized by, The method comprises the following steps: A preprocessing module is configured to perform preprocessing on a first visible light image based on a zero-sample image recovery strategy to obtain a second visible light image; The zero-sample image recovery strategy comprises an input initialization, a back-projection guidance, and a noise injection mechanism; A target tracking module is configured to input the second visible light image and an infrared image into a cross-modal regulation attention model to obtain a predicted target position; the cross-modal regulation attention model comprises a Transformer-based visible-infrared tracker, a Transformer block, a cross-modal regulation attention unit, and a tracking prediction head; The second visible light image and the infrared image are input into the cross-modal regulation attention model to obtain the predicted target position, and the method specifically comprises the following steps: the second visible light image and the infrared image are input into the Transformer-based visible-infrared tracker to obtain a mark sequence; the second visible light image corresponds to the infrared image; the mark sequence is input into the Transformer block for processing to obtain a feature sequence; the feature sequence is input into the cross-modal regulation attention unit to obtain an updated feature sequence; and the updated feature sequence is merged along a channel dimension and then input into the tracking prediction head to output a position of a predicted target. The feature sequence is input into the cross-modal regulation attention unit to obtain an updated feature sequence, specifically comprising: inputting the feature sequence into the cross-modal regulation attention unit to obtain a query matrix and a keyword matrix of the visible light modality, and a query matrix and a keyword matrix of the infrared modality; determining an original visible light correlation graph according to the query matrix and the keyword matrix of the visible light modality; determining an original infrared correlation graph according to the query matrix and the keyword matrix of the infrared modality; determining a corresponding updated feature sequence according to the original visible light correlation graph and the original infrared correlation graph; The corresponding updated feature sequence is determined according to the original visible light correlation graph and the original infrared correlation graph, specifically comprising: determining aggregation information according to the original visible light correlation graph and the original infrared correlation graph; performing attention operation on the aggregation information to obtain an initial visible light regulation correlation graph and an initial infrared regulation correlation graph; determining a final visible light image attention graph according to the original visible light correlation graph and the initial visible light regulation correlation graph; determining a final infrared image attention graph according to the original infrared correlation graph and the initial infrared regulation correlation graph; determining a corresponding updated feature sequence according to the final visible light image attention graph and the final infrared image attention graph.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the track station target tracking method based on visible light-infrared images according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the track station target tracking method based on visible light-infrared images according to any one of claims 1 to 5.

Images