An infrared pedestrian tracking method

Abstract

This invention discloses an infrared pedestrian tracking method, belonging to the field of computer vision. The method includes a main network design, a convolutional module to build an encoder-decoder network, a Transformer auxiliary network, an encoder-decoder, and a tracker. It combines Transformer multi-head self-attention computation with convolutional computation, enhancing the network's feature extraction capabilities to improve the extraction of pedestrian feature information from infrared images. The Transformer and convolutional modules are designed to be lightweight to ensure model size and computational speed. The method captures pedestrian images at night using an infrared imager, then obtains the pedestrian's movement trajectory through a tracking algorithm, and determines the pedestrian's intention based on the trajectory.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to an infrared pedestrian tracking method. Background Technology

[0002] With societal progress, technological innovation, and improved living standards, computer vision technology has become increasingly popular. In recent years, the rapid advancement of deep learning algorithms has propelled computer vision technology forward. Computer vision is a technology that uses machines to observe objects, replacing human observation. It is applied to object tracking, target classification, and other areas. Pedestrian tracking technology, in particular, is finding increasing applications in daily life, with broad prospects in intelligent surveillance, driver assistance systems, and human behavior analysis. Pedestrian tracking technology is used to study and determine whether a pedestrian exists in a given image or video sequence, accurately and quickly locating the target's specific position. The development of pedestrian tracking technology provides effective technical support for reducing traffic accidents to a certain extent.

[0003] Existing pedestrian tracking technologies can be categorized into two types: one utilizes traditional camera imaging technology, and the other is based on infrared thermal imaging technology. For example... Figures 1-2 As shown. Traditional visible light equipment cannot be used in nighttime driving or autonomous driving, especially in adverse weather conditions such as fog, rain, and snow. It is difficult to accurately track pedestrians using traditional visible light equipment. In contrast, infrared thermal imaging is a technology that images the infrared radiation emitted by living organisms based on the relative temperature information of objects. The resulting images are less affected by various natural phenomena. Infrared thermal imaging is non-contact, highly sensitive, and highly available, and can be used in densely populated areas such as stations, hospitals, squares, and shopping malls. It is also used in power system failures, oil exploration, security checks, pedestrian tracking, ID card recognition, and medical applications. However, pedestrians captured by infrared images lack color, making them susceptible to interference from similar infrared radiating objects, sometimes resulting in lower accuracy. Currently, pedestrian tracking speeds are far greater than those achieved by human tracking. Machine tracking accuracy is even higher than human tracking accuracy because humans are easily affected by various environmental and physical conditions when tracking, such as fatigue and adverse weather conditions like rain and snow, which affect tracking accuracy.

[0004] Therefore, this application proposes an infrared pedestrian tracking method. Summary of the Invention

[0005] 1. Technical problems to be solved

[0006] To address the problems existing in the prior art, the purpose of this invention is to provide an infrared pedestrian tracking method that can capture pedestrian images at night using an infrared imager, then obtain the pedestrian's movement trajectory through a tracking algorithm, and determine the pedestrian's intention based on the trajectory.

[0007] 2. Technical Solution

[0008] To solve the above problems, the present invention adopts the following technical solution.

[0009] An infrared pedestrian tracking method includes a main network design, a convolutional module to build an encoding / decoding network, a Transformer auxiliary network, an encoding / decoding network, and a tracker. It combines Transformer multi-head self-attention computation with convolutional computation, and improves the extraction of pedestrian feature information from infrared images by enhancing the network's feature extraction capabilities. The Transformer module and convolutional module are designed to be lightweight to ensure the model's size and computational speed.

[0010] Furthermore, the main network design uses the convolution module to build an encoding and decoding network, and utilizes convolution calculation to fully extract local features, ensuring that the network extracts the target features. At the same time, the structure of the encoding and decoding also ensures that the target will not gradually become smaller as the feature map is downsampled and passed.

[0011] Furthermore, the main network design uses the convolution module to build an encoding and decoding network, and utilizes convolution calculation to fully extract local features, ensuring that the network extracts the target features. At the same time, the structure of the encoding and decoding also ensures that the target will not gradually become smaller as the feature map is downsampled and passed.

[0012] Furthermore, the convolutional module in the encoding / decoding network uses Tied Block Convolution (TBC) instead of standard convolution. The calculation process of TBC convolution is similar to that of grouped convolution; it uses the same set of convolutions when calculating the grouped feature maps, meaning that parameters are shared among the groups. The formula for calculating TBC convolution can be expressed as:

[0013]

[0014] In the formula above, X1, X2, ..., X G This represents a grouping operation on the feature map, dividing the feature map... Divide into G groups, and represent the feature map of each group as X. i =c i / G×h i ×w i W′ represents the key difference between TBC convolution and grouped convolution, where W′ is the shared convolution kernel for each group convolution, which can be represented as: (where the number of B's ​​is the same as the number of feature map groups G), This represents the concat operation, which restores the dimensions of the feature map. The total number of computational parameters for TBC convolution is... The number of parameters c for standard convolution computation with the same input and kernel size o ×c i Compared to ×k×k, B is reduced. 2 This significantly reduces the number of network parameters. Based on TBC convolution, the construction of the convolution module firstly divides the network into different groups based on the required parameter reduction factor, and secondly, it uses an identity mapping method to construct the "input-output" residual channel, avoiding feature gaps and disconnections caused by too many groups.

[0015] Furthermore, the Transformer auxiliary network progressively enlarges the segmented regions from the center point of the feature map, with each region containing the upper-layer region, until the entire feature map is covered. This nested approach allows information interaction between each feature block. After the nested Transformer module completes the extraction of nested blocks from the feature map, the convolutional layer integrates each feature block, regularizing the size of the feature blocks to be the same, so that the subsequently converted tokens are of the same size and can be used for matching degree calculation. Convolutional calculation can perform local aggregation of features of each feature block. After that, the aligned feature blocks are transformed into vectors to obtain one-dimensional vector tokens. Global attention relationships are constructed through multi-head self-attention calculation.

[0016] Furthermore, the tracker employs a uniform linear Kalman filter to predict the target's trajectory. When the detector detects the presence of a target, this application uses the spatial motion information of the target in the preceding and following frames. Let (u, v) represent the center point of the current predicted bounding box, h be the height of the current predicted bounding box, and γ be the aspect ratio of the current predicted bounding box; while This is represented by the velocity components of the four elements mentioned above. A Kalman filter using both the uniform velocity model and the linear observation model is used to predict the next state (u, v, γ, h) of the target. Mahalanobis distance is used for motion matching of the predicted trajectory information to quickly determine the predicted movement coordinates of each target.

[0017]

[0018] The above formula d j This represents the target location information detected by the detector, y i S is the position information of the next frame predicted by the Kalman filter. i Calculate the covariance matrix between the target's current location information and its predicted location information.

[0019] 3. Beneficial effects

[0020] Compared with the prior art, the advantages of this invention are:

[0021] 1. Obtain the pedestrian's movement trajectory through tracking algorithms, and determine whether the pedestrian's intention is suspicious based on the trajectory.

[0022] 2. Improve the extraction of pedestrian features from infrared images by enhancing the network's feature extraction capabilities; and design lightweight versions of the Transformer and convolution modules to ensure model size and computational speed.

[0023] 3. The number of groups can be divided by reducing the parameter multiple as needed. Secondly, the identity mapping method is used to construct the "input-output" residual channel to avoid feature discontinuity caused by too many groups.

[0024] 4. For the feature extraction network of the detection module, the convolution module and the transformer module are effectively combined to improve the network's ability to perceive local and global information. Attached Figure Description

[0025] Figure 1 The images are those captured by conventional cameras in the background art of this invention;

[0026] Figure 2 The image is taken using infrared thermal imaging technology in the background art of this invention; Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] Example 1:

[0029] An infrared pedestrian tracking method includes a main network design, a convolutional module to build an encoding / decoding network, a Transformer auxiliary network, an encoding / decoding network, and a tracker. It combines Transformer multi-head self-attention computation with convolutional computation, and improves the extraction of pedestrian feature information from infrared images by enhancing the network's feature extraction capabilities. The Transformer module and convolutional module are designed to be lightweight to ensure the model's size and computational speed.

[0030] This task involves capturing pedestrian images at night using an infrared imager, then tracking the pedestrians' trajectories using a tracking algorithm to determine their intentions. The main challenge lies in pedestrian tracking. Due to the characteristics of infrared imaging, the captured images contain limited texture and color, and pedestrians occupy a small pixel area in the images captured from the monitoring perspective. Therefore, the tracking difficulty in this scenario can be attributed to the limited target feature information and small area. Furthermore, considering embedded device integration and portability, there are also certain requirements for the size and computational speed of the network model. To address these tracking challenges, this application combines Transformer multi-head self-attention computation with convolution computation, enhancing the network's feature extraction capabilities to improve the extraction of pedestrian feature information from infrared images. Additionally, the Transformer and convolution modules are designed to be lightweight to ensure the model's size and computational speed.

[0031] Specifically, the design of the main convolutional network takes into account the small area of ​​the target in the infrared image. Therefore, this application uses convolutional modules to build the encoding and decoding network in the main network design. Convolutional computation is mainly used to fully extract local features, ensuring that the network extracts the target features. At the same time, the structure of the encoding and decoding also ensures that the target does not gradually shrink as the feature map is downsampled and passed.

[0032] To achieve a lightweight model, this application uses Tied Block Convolution (TBC) convolution instead of standard convolution. The calculation process of TBC convolution is similar to that of grouped convolution, but it uses the same group of convolutions when calculating the grouped feature maps, meaning that parameters are shared among the groups. The formula for calculating TBC convolution can be expressed as:

[0033]

[0034] In the formula above, X1, X2, ..., X G This represents a grouping operation on the feature map, dividing the feature map... Divide into G groups, and represent the feature map of each group as X. i =c i / G×h i ×w i W′ represents the key difference between TBC convolution and grouped convolution, where W′ is the shared convolution kernel for each group convolution, which can be represented as: (where the number of B's ​​is the same as the number of feature map groups G), This represents the concat operation, which restores the dimensions of the feature map. The total number of computational parameters for TBC convolution is... The number of parameters c for standard convolution computation with the same input and kernel size o ×ci Compared to ×k×k, B is reduced. 2 This significantly reduces the number of network parameters. Based on TBC convolution, the construction of the convolution module firstly divides the network into different groups based on the required parameter reduction factor, and secondly, it uses an identity mapping method to construct the "input-output" residual channel, avoiding feature gaps and disconnections caused by too many groups.

[0035] Example 2:

[0036] In this application, a Transformer auxiliary network is constructed. Convolutional modules are used in the main encoder-decoder network to enhance the model's ability to capture small target features, leveraging their sensitivity to local features. The auxiliary network is primarily built between the encoder and decoder, transferring feature information from the encoder to the decoder to supplement the semantic level of the feature information. Furthermore, the auxiliary network incorporates additional global information from the feature map via a Transformer module to supplement the local information of the convolutional modules. While the Transformer can construct long-distance mappings in the feature map, it requires calculating the matching degree for each token, resulting in a large computational parameter. Currently, vision transformers divide the feature map into equally sized feature blocks and then convert them into tokens for computation. This method lacks correlation between feature blocks, and the smaller the granularity of the division, the greater the computational cost. Therefore, from the perspective of lightweight and fast computation, this application designs a nested Transformer module.

[0037] The transformer module designed in this application optimizes the feature block partitioning method, changing it to a segmentation region that expands layer by layer from the center point of the feature map. Each region contains the upper-layer region until the entire feature map is covered. Taking a 9x9 feature map as an example, assuming nested segmentation from its center point in a 3x3 pattern, only 4 feature blocks are needed to encompass the entire feature map, while the standard vision transformer requires 9 feature blocks. This difference becomes more pronounced for larger feature maps. Furthermore, this nesting method allows information interaction between each feature block, resulting in a more compact global feature structure. After the nested transformer module extracts the nested blocks of the feature map, a convolutional layer integrates each feature block, regularizing the size of the feature blocks to be the same, ensuring that the subsequently transformed tokens are of the same size for matching degree calculation. For feature blocks of different sizes, this application uses downsampling and zero-padding operations for size alignment (the alignment size is set to half the original feature map size), while convolutional calculations enable local aggregation of features for each feature block. The aligned feature blocks are then transformed into vectors to obtain one-dimensional vector tokens, and global attention relationships are constructed through multi-head self-attention computation.

[0038] In the encoding / decoding structure of this application, since the feature map output by the nested transformer module is half the original size, a staggered connection method is adopted for feature transfer between the encoder and decryptor. In summary, for the feature extraction network of the detection module, the effective combination of the convolutional module and the transformer module enhances the network's ability to perceive both local and global information.

[0039] Example 3:

[0040] In designing the tracker, this paper prioritizes tracking response speed and selects a uniform linear Kalman filter to predict the target's trajectory. When the detector detects the target's presence, this paper uses the spatial motion information of the target in the preceding and following frames. Let (u, v) represent the center point of the current predicted bounding box, h be the height of the current predicted bounding box, and γ be the aspect ratio of the current predicted bounding box; while This is represented by the velocity components of the four elements mentioned above. A Kalman filter using both the uniform velocity model and the linear observation model is used to predict the next state (u, v, γ, h) of the target. Mahalanobis distance is used for motion matching of the predicted trajectory information to quickly determine the predicted movement coordinates of each target.

[0041]

[0042] The above formula d j This represents the target location information detected by the detector, y i S is the position information of the next frame predicted by the Kalman filter. i Calculate the covariance matrix between the target's current location information and its predicted location information.

[0043] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.

Claims

1. An infrared pedestrian tracking method, characterized by: This method includes main network design, construction of an encoding / decoding network using convolutional modules, a Transformer auxiliary network, encoding / decoding, and a tracker. It combines Transformer multi-head self-attention computation with convolutional computation, enhances the network's feature extraction capabilities to improve the extraction of pedestrian feature information from infrared images, and performs lightweight design of the Transformer module and convolutional module to ensure the model's size and computational speed. The main network design uses the convolution module to build an encoding and decoding network. Convolution calculation is used to fully extract local features to ensure that the network extracts the target features. At the same time, the structure of the encoding and decoding also ensures that the target will not gradually become smaller as the feature map is downsampled and passed. The convolutional module in the encoding / decoding network uses Tied Block Convolution (TBC) instead of standard convolution. The calculation process of TBC convolution is similar to that of grouped convolution; it uses the same set of convolutions when calculating the grouped feature maps, meaning that parameters are shared among the groups. The formula for calculating TBC convolution can be expressed as: In the above formula This represents a grouping operation on the feature map, dividing the feature map... Divided into There are groups, and the feature map of each group is represented as follows: ,and The main difference between TBC convolution and grouped convolution lies in the shared convolution kernel for each group convolution, which can be represented as: ,in The number of features and the number of feature map groups same," "" represents the concat operation, which restores the dimensions of the feature map; the total number of computational parameters for TBC convolution is The number of parameters required for standard convolution with the same input and kernel size Compared to smaller Based on TBC convolution, in the construction of the convolution module, firstly, different numbers of groups can be divided according to the required reduction factor of the parameters; secondly, the identity mapping method is used to construct the "input-output" residual channel to avoid feature discontinuity due to too many groups. The Transformer auxiliary network progressively enlarges the segmented regions from the center point of the feature map, with each region containing the upper-layer region, until the entire feature map is covered. This nested approach allows information interaction between each feature block. After the nested Transformer module completes the extraction of nested blocks from the feature map, the convolutional layer integrates each feature block, regularizing the size of the feature blocks to be the same, so that the subsequently converted tokens are of the same size and can be used for matching degree calculation. Convolutional calculation can perform local aggregation of features of each feature block. After that, the aligned feature blocks are transformed into vectors to obtain one-dimensional vector tokens. Global attention relationships are constructed through multi-head self-attention calculation. The tracker uses a uniform linear Kalman filter to predict the target's trajectory; when the detector detects the presence of a target, it uses the spatial motion information of the target in the preceding and following frames. Indicated, where This indicates the center point of the current predicted bounding box for the target. The height of the current prediction box. This indicates the aspect ratio of the current prediction box; while The velocity components of the above four elements are then used to determine the next state of the target using Kalman filters of the uniform velocity model and the linear observation model. Prediction is performed; and Mahalanobis distance is used for motion matching of the predicted trajectory information to quickly determine the moving coordinates of each predicted target. The above formula This indicates the target location information detected by the detector. The position information for the next frame predicted by the Kalman filter. Calculate the covariance matrix between the target's current location information and its predicted location information.

Images