A pedestrian detection and tracking method based on visual attention mechanism

Abstract

The application discloses a pedestrian detection and tracking method based on a visual attention mechanism, wherein for an input continuous video stream, a backbone network of a YOLOv7 target pedestrian detection network is used to perform feature extraction on an input image to obtain three feature maps of different sizes; a neck network of the YOLOv7 adopts an FPN feature pyramid network and a PAN path aggregation network structure to perform feature fusion, thereby obtaining three reinforced feature maps of different sizes; a ViT feature extraction network is used to perform feature fusion on the three reinforced feature maps of different sizes to obtain a final feature map; a detection head of the YOLOv7 performs pedestrian detection based on the final feature map, and the detection head can output a bounding box, a class label and a confidence score for each detected pedestrian; and a DeepSORT multi-target pedestrian tracking algorithm is used to track the pedestrians based on the bounding box of the pedestrian in a current frame, thereby obtaining the motion trajectory of each pedestrian.

Description

TECHNICAL FIELD

[0001] The present application relates to a computer vision-based pedestrian detection and tracking method, belonging to the field of machine learning, pedestrian detection and pedestrian tracking, and particularly relates to a pedestrian detection and tracking method based on a visual attention mechanism. BACKGROUND

[0002] With the acceleration of urbanization, the demand for public safety and traffic management is growing, which makes the technology that can assist in ensuring public safety and traffic order more and more valued, and pedestrian detection is one of them. Pedestrian detection has become a research hotspot in the field of computer vision in recent years, and is also a big difficulty in the field of target detection. The core task of pedestrian detection is to identify and locate pedestrians in images, and this technology has wide application value in many fields. For example, in the field of traffic safety, self-driving cars can avoid traffic accidents by detecting pedestrians in advance and avoiding them in time; in security protection, pedestrian detection can be used to monitor and prevent suspicious personnel from entering; in public place management, by counting the flow of people, resource allocation can be optimized and management efficiency can be improved.

[0003] However, the traditional pedestrian detection and tracking system often relies on inefficient algorithms, and it is difficult to meet the requirements of real-time and accuracy. In addition, the current mainstream pedestrian detection technology mainly focuses on low-density environments, and there are relatively few studies in medium and high-density scenes. In the face of medium and high-density scenes, the detection accuracy is usually low, and it is difficult, especially in continuous frames, to accurately identify pedestrians. SUMMARY

[0004] In order to solve the problems existing in the prior art, the present application provides a pedestrian detection and tracking method based on a visual attention mechanism, which aims to solve the problems of poor pedestrian detection accuracy in existing dense scenes and real-time and stable tracking of pedestrians.

[0005] In order to achieve the purpose of the present application, the following is an embodiment of the present application:

[0006] A pedestrian detection and tracking method based on a visual attention mechanism includes the following steps:

[0007] Step S1, acquire a continuous video stream and perform preprocessing

[0008] In step S2, the pre-processed continuous video stream is input, and a target pedestrian detection frame of a current frame is obtained through a YOLO-ViT detection model; the YOLO-ViT detection model comprises a YOLOv7 target pedestrian detection network and a ViT feature extraction network; a backbone network of the YOLOv7 target pedestrian detection network performs feature extraction on an input image to obtain three feature maps of different sizes; a neck network of the YOLOv7 target pedestrian detection network adopts an FPN feature pyramid network and a PAN path aggregation network structure to perform feature fusion, and obtains three reinforced feature maps of different sizes; the ViT feature extraction network performs feature fusion on the three reinforced feature maps of different sizes to obtain a final feature map; and a detection head of the YOLOv7 target pedestrian detection network performs pedestrian detection based on the final feature map, and the detection head can output a bounding box, a category label and a confidence score for each detected pedestrian; in step S3, the DeepSORT multi-target pedestrian tracking algorithm is adopted to track the pedestrians based on the pedestrian output bounding box of the current frame obtained in step S2, and a motion trajectory of each pedestrian is obtained.

[0009] Further, the pre-processing in step S1 comprises image size and pixel normalization processing.

[0010] Further, the backbone network of the YOLOv7 target pedestrian detection network performs feature extraction on the input image as follows:

[0011] In step S2.1.1, an initial convolutional layer is used for initializing feature extraction; the initial convolutional layer applies a 3×3 convolutional kernel to process the input image, and performs down-sampling with a 2-step size, and the calculation formula is as follows:

[0012] F stem =Conv(I,K stem ,stride=2)

[0013] Wherein, I is the normalized input image, the size is 224×224×3, K stem is the convolutional kernel, the size is 3×3×3×32, and the output feature map size is 112×112×32;

[0014] The feature map is processed through batch normalization and Swish activation function, and the standardization layer output reduces the internal offset, and the calculation formula is as follows:

[0015] F stem =Swish(BatchNorm(Conv(I,K stem ,stride=2))

[0016] Wherein, F stem is the feature map processed through batch normalization and Swish activation function.

[0017] Step S2.1.2, more complex features are extracted using MBConv modules, each MBConv module including a depthwise convolution and a 1x1 pointwise convolution for extracting different levels of features;

[0018] The convolution is applied independently to each input channel by depthwise separable convolution, and the calculation formula is as follows:

[0019] F dw =DepthwiseConv(F in ,K dw )

[0020] Where F in is the output feature map of the previous layer, K dw is the depthwise convolution kernel, and the stride is 2;

[0021] The output of the depthwise convolution is mapped to a new channel space using a 1x1 convolution to enhance the extracted features, and the calculation formula is as follows:

[0022] F pw =PointwiseConv(F dw ,K pw )

[0023] Where K pw is a 1x1 convolution kernel used to change the number of channels;

[0024] After the dilated convolution and pointwise convolution, the feature map is processed using batch normalization and Swish activation function, and the output of the standardization layer is calculated as follows:

[0025] F MBConv =Swish(BatchNorm(PointwiseConv(F dw ,K pw )))

[0026] Where F MBConv is the feature map processed by batch normalization and Swish activation function;

[0027] Step S2.1.3, extract feature maps from different levels of the network to capture information at different scales and levels; extract feature maps from the third, fifth and seventh layers of the network, as follows:

[0028] Step S2.1.3.1, extract the third layer feature map, which contains low-level features such as edges and textures, and the calculation formula is as follows:

[0029] F3=Conv(F2,K3,stride=2)

[0030] wherein F2 is the output feature map of the second layer, and F3 is the output feature map of the third layer;

[0031] Step S2.1.3.2, the fifth layer feature map is extracted, and the calculation formula is as follows:

[0032] F5 = Conv (F4, K5, stride = 2)

[0033] wherein F4 is the output feature map of the fourth layer, and F5 is the output feature map of the fifth layer;

[0034] Step S2.1.3.3, the seventh layer feature map is extracted, and the calculation formula is as follows:

[0035] F7 = Conv (F6, K7, stride = 2)

[0036] wherein F6 is the output feature map of the sixth layer, and F7 is the output feature map of the seventh layer.

[0037] Further, the neck network of the YOLOv7 target pedestrian detection network adopts the steps of feature fusion of FPN feature pyramid network and PAN path aggregation network structure as follows:

[0038] Step S2.2.1, the FPN structure is adopted to perform feature fusion through the top-down path, the high-level semantic feature map of the backbone network is gradually reduced in dimension and passed down, the bottom layer feature map is given richer semantic information, and the blurred pedestrian target is helped to be recognized; including the following steps:

[0039] Step S2.2.1.1, the up-sampling operation is used to enlarge the high layer feature map F7 to the same size as the middle layer feature map F5, and the feature map F 7↑ is obtained;

[0040] Step S2.2.1.2, the feature map F 7↑ obtained in step S2.2.1.1 is element-wise added with the middle layer fusion feature map F5, to obtain the feature map F5' containing high layer details and middle layer semantic information;

[0041] Step S2.2.1.3, the feature map obtained in step S2.2.1.2 is further up-sampled to expand to the same size as the low layer feature map, to obtain the feature map F5' ↑ ;

[0042] Step S2.2.1.4, the feature map F5' ↑ obtained in step S2.2.1.3 is element-wise added with the low layer feature map F3, to obtain the feature map F3' fused with high layer, middle layer and bottom layer information;

[0043] Step S2.2.2, feature enhancement by bottom-up path using PAN structure, includes the following steps:

[0044] Step S2.2.2.1, down-sampling the low-level fusion feature map F3' using max-pooling to extract local features from the low-level fusion feature map and extract the most significant features;

[0045] Step S2.2.2.2, element-wise addition of the feature map obtained in step S2.2.2.1 and the middle-level fusion feature map F5' to obtain a feature map F5" containing bottom-level details and middle-level semantic information;

[0046] Step S2.2.2.3, further down-sampling the feature map obtained in step S2.2.2.2 to reduce the size of the high-level feature map;

[0047] Step S2.2.2.4, element-wise addition of the feature map F5" obtained in step S2.2.2.3 and the high-level fusion feature map F7 to obtain a feature map F7' containing high-level details and bottom-level semantic information;

[0048] Further, the step of using the ViT feature extraction network to fuse the three different size enhanced feature maps to obtain the final feature map in step S2.3 is as follows:

[0049] Step S2.3.1, first pre-process the feature map, divide the input image into fixed size blocks, stretch each small block into a vector, and map it through a linear layer to obtain the embedding representation of the image block, the calculation formula is as follows:

[0050] z i =x i E

[0051] Where x i represents the flattened vector of the i-th image block, i = 1,...,N, and E is a linear mapping matrix;

[0052] Step S2.3.2, import the embedding representation of the image block obtained in step S2.3.1 into the position encoding, add position encoding to provide the position information of each image block, denoted as:

[0053]

[0054] Where z0 is the vector after adding position encoding, and E pos represents the position encoding vector;

[0055] Step S2.3.3, the position encoding vector obtained in step S2.3.2 is input into the Transformer encoder, the multi-head self-attention mechanism is used to capture global features, and the feedforward neural network is used to process the embedding, so as to further analyze and extract the features of the image block, integrate the feature relationship of different subspaces, and obtain a global feature map;

[0056] Step S2.3.4, the feature map obtained in step S2.3.3 is subjected to global average pooling to average the feature map of each channel in the spatial dimension, and the high-dimensional feature map processed by the multiple Transformer encoders is converted into a low-dimensional representation that can be used for classification, thereby generating a global feature vector containing multi-scale spatial information and global semantic information.

[0057] Further, in step S2.3.4, the global feature vector is generated.

[0058] Step S2.3.4.1, the feature map processed by the ViT module is subjected to global average pooling, and the calculation formula is as follows:

[0059]

[0060] Wherein, g represents the global feature vector; N represents the number of blocks, z i represents the feature vector of each block;

[0061] Step S2.3.4.2, the global feature vectors of the multi-scale features F3', F5" and F7' obtained in step S2.3.4.1 are fused to obtain the final global feature vector, and the calculation formula is as follows:

[0062] F final =Concat(g3,g5,g7)

[0063] Wherein, g3, g5, g7 are global feature vectors of feature maps of different scales.

[0064] Further, step S3 uses the DeepSORT algorithm to obtain the specific steps of the pedestrian trajectory:

[0065] Step S3.1, the video current frame detection result extracted in step S2.3, including pedestrian detection box information, DeepSORT receives these detection boxes, and tracks the target pedestrian, and the target pedestrian detection box of the current frame is:

[0066] B t ={b1,b2,…,b n}

[0067] Wherein, b i= [x, y, w, h] represents the i-th detected pedestrian bounding box, x, y is the center coordinate of the bounding box, w, h is the width and height of the bounding box;

[0068] Step S3.2, for each target pedestrian detection box b i , DeepSORT uses a deep feature extraction network to extract the appearance feature vector f t of the target pedestrian;

[0069] Step S3.3, based on the current position of the target pedestrian, a Kalman filter is used to estimate the position and speed of each detected pedestrian target, and the position of each pedestrian target in the next frame in the current frame is predicted;

[0070] Step S3.4, the Hungarian algorithm is used to associate the detection box of the current frame with the target tracked in the previous frame, and the trajectory of the target pedestrian is obtained;

[0071] Step S3.5, based on the matching result obtained in step S3.4, the Kalman filter state is updated, new targets are created or lost targets are deleted, and the motion state of each tracked target is output, including position, speed and trajectory.

[0072] Further, the step of extracting the appearance feature vector f t of the target pedestrian is as follows:

[0073] Step S3.2.1, the target pedestrian region is cropped from the input image I t using the target pedestrian detection box:

[0074]

[0075] wherein, is the target pedestrian region cropped from the input image I t ;

[0076] Step S3.2.2, the target pedestrian region obtained in step S3.2.1 is adjusted to a fixed size;

[0077] Step S3.2.3, the target pedestrian image adjusted in step S3.2.2 is input into ResNet for feature extraction, and the appearance feature vector f t is extracted;

[0078] Step S3.2.4, the target pedestrian region of the current frame obtained in step S3.2.1 and the feature vector f t obtained in step S3.2.3 are input for feature extraction, and global average pooling is used to convert the feature map into a fixed length vector as the appearance feature f t of the target pedestrian.

[0079] Further, the method for predicting the position of each pedestrian target in the next frame in S3.3 is:

[0080] In step S3.3.1, the state of the target pedestrian in the current frame is calculated by using the Kalman filter, and the motion trajectory of the target pedestrian in the next frame is predicted. The state transition equation of the target pedestrian is as follows:

[0081] x t =F·x t-1 +B·u t +ω t

[0082] wherein x t =[x,y,v x ,v y ] T is the state of the target pedestrian, x and y represent the two-dimensional position of the target pedestrian, v x and v y represent the speed of the target pedestrian in the x and y directions respectively; F is the state transition matrix, which describes the dynamic change of the target pedestrian from the last frame to the current frame, x t-1 is the state of the target pedestrian in the last frame, B·u t is the control input term, and ω t is the process noise, which represents the random disturbance caused by model errors and external factors;

[0083] In step S3.3.2, the state of the target pedestrian in the next frame is predicted by the state transition equation obtained in step S3.3.1. The predicted state of the target pedestrian at the next frame can be represented as:

[0084]

[0085] The position and speed of each pedestrian target are predicted, and the predicted state

[0086] Further, the method for obtaining the trajectory of the target pedestrian by using the Hungarian algorithm in step S3.4 is:

[0087] In step S3.4.1, based on the appearance feature f t extracted in step S3.2 in the current frame and the appearance feature f t-1 of the pedestrian in the last frame, the appearance similarity between the target pedestrian in the current frame and the target pedestrian in the last frame is calculated by using the Euclidean distance. The Euclidean distance is calculated as follows:

[0088] d appearance =||f t -f t-1 ||2 ​

[0089] wherein f t and f t-1 are the target pedestrian appearance feature vectors of the current frame and the previous frame, and ||·|| represents the norm of the vector. The larger the Euclidean distance, the more similar the pedestrian appearance is;

[0090] Step S3.4.2, the motion feature similarity between the target pedestrians is evaluated by calculating the motion feature difference between the target pedestrian positions and speeds through the Euclidean distance, and the similarity is calculated as follows:

[0091]

[0092] wherein (x t ,y t ) and (x t-1 ,y t-1 ) are the center positions of the target pedestrians in the current frame and the previous frame, i.e. the center position coordinates of the detection frame;

[0093] Step S3.4.3, the Hungarian algorithm is used to associate the detection frame of the current frame with the target tracked in the previous frame to determine the pedestrian target corresponding to each detection frame; and a cost matrix of the target pedestrian matching is generated according to the weighted similarity of the motion feature and the appearance feature, and the total similarity is calculated as follows:

[0094] d total = alpha * d motion + beta * d appearance

[0095] wherein alpha and beta are weight coefficients for adjusting the proportion of the motion information and the appearance information in the matching process.

[0096] Compared with the prior art, the beneficial results of the present application are:

[0097] (1) Improved detection accuracy

[0098] The prior art has low pedestrian detection accuracy in medium and high density scenes, and the present application introduces ViT into the YOLO framework, takes advantage of the ViT processing global information, especially the advantage of processing long-distance dependence between pixels in high-density scenes, and significantly improves the detection accuracy of the model in high-density scenes.

[0099] (2) Enhanced real-time performance

[0100] The present application adopts a single-stage detection method (YOLO), which avoids the high time and hardware cost problems caused by two-step execution in two-stage detection methods (such as R-CNN series), and improves the real-time detection performance. Using the DeepSORT algorithm, the speed is improved by about 20 times compared with the traditional algorithm in tracking, which can quickly process the target pedestrian detection result and ensure the real-time performance of the entire system in pedestrian tracking.

[0101] (3) Adaptability to complex scenes is improved

[0102] The ViT adopted by the application can better capture the subtle differences between pedestrians and backgrounds, which is particularly important for complex scenes (such as multiple people gathering or partial occlusion), and improves the detection and tracking capabilities of the system in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0103] Figure 1 A schematic diagram of a pedestrian detection and tracking method based on a visual attention mechanism according to the application.

[0104] Figure 2 A schematic diagram of pedestrian detection based on a YOLO and ViT model according to the application.

[0105] Figure 3 A schematic diagram of pedestrian trajectory tracking based on a DeepSORT model according to the application. DETAILED DESCRIPTION

[0106] In order to make the purpose, technical solutions and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and are not intended to limit the application, i.e., the described examples are only a part of the examples of the application, not all examples. Therefore, the detailed description of the examples provided in the drawings below is not intended to limit the scope of the claimed application, but only to represent selected embodiments of the application. Based on the examples of the application, all other examples obtained by those skilled in the art without making creative efforts fall within the scope of the application.

[0107] The technical solutions of the application will be further described in detail below in combination with the drawings:

[0108] As shown in Figure 1 , Figure 2 and Figure 3 , a pedestrian detection and tracking method based on a visual attention mechanism includes the following steps:

[0109] Step S1, acquire a continuous video stream and pre-process the image size and pixels.

[0110] In this embodiment, the picture size is adjusted to 224x224.

[0111] More specifically, the method for pixel normalization processing of the input image is:

[0112] The pixel value of the image is scaled to between 0 and 1 to be compatible with the weights in the neural network, and the pixel value calculation formula is as follows:

[0113]

[0114] where I is the input image, μ is the mean of the image, σ is the standard deviation of the image, is the normalized image;

[0115] Step S2, input the pre-processed continuous video stream, and obtain the target pedestrian detection frame of the current frame through a YOLO-ViT detection model, wherein the YOLO-ViT detection model comprises a YOLOv7 target pedestrian detection network and a ViT feature extraction network. Mainly includes the following steps:

[0116] Step S2.1, input the pre-processed data into the backbone network EfficientNetB0 of YOLOv7, the backbone network EfficientNetB0 is a nine-layer network structure based on the inverted residual structure (InvertedResiduals) and depthwise separable convolution (Depthwise Separable Convolutions) of MobileNetV2, and the EfficientNetB0 is composed of multiple convolutional blocks (Convolutional Blocks), activation functions (Swish), batch normalization (BatchNormalization) and pooling layers, etc. A series of feature maps are generated through the EfficientNetB0 network, and the feature information of the third, fifth and seventh layers is extracted to obtain three feature maps of different sizes. Including the following steps:

[0117] Step S2.1.1, an initial convolutional layer is used for initializing feature extraction. The input image is preliminarily processed to extract the basic features of the image and reduce the spatial resolution of the input image;

[0118] The initial convolutional layer applies a 3x3 convolutional kernel to process the input image, and uses a 2-step down-sampling, and the calculation formula is as follows:

[0119] F stem =Conv(I,K stem ,stride=2)

[0120] I: the normalized input image, the size is 224x224x3, K stem : the convolutional kernel, the size is 3x3x3x32 (the input channel number is 3, and the output channel number is 32), and the output feature map size is: 112x112x32.

[0121] The feature map is processed through batch normalization and Swish activation function, and the standardization layer output reduces the internal offset, and the calculation formula is as follows:

[0122] F stem =Swish(BatchNorm(Conv(I,K)) stem stride=2)))

[0123] Among them, F stem This is the feature map after batch normalization and Swish activation function processing.

[0124] Step S2.1.2 uses the MBConv (Mobile Inverted Bottleneck Convolution) module to extract more complex features. Each MBConv module includes depthwise convolutions and 1×1 point convolutions to extract features at different levels.

[0125] Each input channel is independently convolved using depthwise separable convolution, calculated as follows:

[0126] F dw =DepthwiseConv(F in ,K dw )

[0127] Among them, F in It is the output feature map of the previous layer, K dw It is a depthwise convolution kernel (3×3×32×32), with a stride of 2.

[0128] A 1×1 convolution is used to map the output of the depthwise convolution to a new channel space, thereby enhancing the extracted features. The calculation formula is as follows:

[0129] F pw =PointwiseConv(F dw ,K pw )

[0130] Among them, K pw It is a 1×1 convolution kernel used to change the number of channels.

[0131] After performing extended convolution and pointwise convolution, batch normalization and the Swish activation function are used to process the feature map, and the output of the normalized layer is calculated using the following formula:

[0132] F MBConv =Swish(BatchNorm(PointwiseConv(F dw ,K pw )))

[0133] Among them, F MBConv This is the feature map after batch normalization and Swish activation function processing.

[0134] Step S2.1.3. Extract feature maps from different levels of the network to capture information at different scales and levels. These feature maps will be used for subsequent detection tasks to help the network detect pedestrians at different levels of abstraction space. Feature maps are extracted from the third, fifth and seventh layers of the network, as follows:

[0135] Step S2.1.3.1, extract the third layer feature map, which contains low-level features such as edges, textures, etc. The calculation formula is as follows:

[0136] F3 = Conv(F2, K3, stride = 2)

[0137] Where F2 is the output feature map of the second layer, F3 is the output feature map of the third layer, and K3 is the convolution kernel of the third layer convolution operation.

[0138] Step S2.1.3.2, extract the fifth layer feature map, the calculation formula is as follows:

[0139] F5 = Conv(F4, K5, stride = 2)

[0140] Where F4 is the output feature map of the fourth layer, F5 is the output feature map of the fifth layer, and K5 is the convolution kernel of the fifth layer convolution operation.

[0141] Step S2.1.3.3, extract the seventh layer feature map, the calculation formula is as follows:

[0142] F7 = Conv(F6, K7, stride = 2)

[0143] Where F6 is the output feature map of the sixth layer, F7 is the output feature map of the seventh layer, and K7 is the convolution kernel of the seventh layer convolution operation.

[0144] Step S2.2, transmit the three different size feature maps obtained in step S2.1 to the neck network of YOLOv7 to obtain three different size enhanced feature maps. The neck network uses FPN feature pyramid network and PAN path aggregation network structure for feature fusion, which effectively integrates high-level semantic information and low-level detail information through top-down and bottom-up paths, enhancing the detection ability of pedestrians at different scales. In the FPN and PAN structure, the top-down path reduces the dimension of the high-level semantic feature map and propagates it to the bottom layer, giving the bottom layer semantic information and passing it to the bottom layer to locate the fuzzy pedestrian; the bottom-up path integrates the rich details of the bottom layer to enhance the high layer, and refines the edge features of the pedestrian target. After bidirectional fusion, small-scale pedestrians are guided by high-level semantics for accurate detection, and large-scale pedestrians are outlined with bottom-level details to improve the detection efficiency of the model for pedestrians of different distances and sizes. The specific steps are as follows:

[0145] Step S2.2.1, feature fusion is performed by a top-down path using the FPN structure, gradually reducing the dimension of the high-level semantic feature map of the backbone network and passing it down, giving the bottom layer feature map more rich semantic information, helping the bottom layer feature map to identify ambiguous pedestrian targets. It includes the following steps:

[0146] Step S2.2.1.1, using the up-sampling operation to enlarge the high-level feature map F7 to the same size as the middle layer feature map F5, to obtain the feature map F 7↑ .

[0147] Step S2.2.1.2, element-wise addition is performed between the feature map F 7↑ obtained in step S2.2.1.1 and the middle layer fusion feature map F5, to obtain the feature map F5' containing high-level details and middle layer semantic information;

[0148] Step S2.2.1.3, further up-sampling is performed on the feature map obtained in step S2.2.1.2 to expand it to the same size as the low layer feature map, to obtain the feature map F5' ↑ ;

[0149] Step S2.2.1.4, element-wise addition is performed between the feature map F5' ↑ obtained in step S2.2.1.3 and the low layer feature map F3, to obtain the feature map F3' which fuses high-level, middle layer and bottom layer information.

[0150] Step S2.2.2, feature enhancement is performed by a bottom-up path using the PAN structure, including the following steps:

[0151] Step S2.2.2.1, using maximum pooling to down-sample the low layer fusion feature map F3', to extract local features of the bottom layer fusion feature map and extract the most significant features;

[0152] Step S2.2.2.2, element-wise addition is performed between the feature map obtained in step S2.2.2.1 and the middle layer fusion feature map F5', to obtain the feature map F5" containing bottom layer details and middle layer semantic information;

[0153] Step S2.2.2.3, further down-sampling is performed on the feature map obtained in step S2.2.2.2 to reduce it to the size of the high layer feature map;

[0154] Step S2.2.2.4, element-wise addition is performed between the feature map F5" obtained in step S2.2.2.3 and the high layer fusion feature map F7, to obtain the feature map F7' containing high-level details and bottom layer semantic information.

[0155] Step S2.3, input the feature maps F3', F5" and F7' output by step S2.2 into a ViT network for further processing, and then perform global pooling and global feature fusion to obtain a final feature map. The specific steps are as follows:

[0156] Step S2.3.1, before the feature map is input into the ViT network, the feature map is preprocessed. The input image is segmented according to fixed size blocks (patches), each small block is stretched into a vector, and a linear layer is used for mapping to obtain the embedding representation of the image block. The calculation formula is as follows:

[0157] z i =x i E

[0158] Where x i represents the flattened vector of the i-th image block (i = 1,...,N), and E is a linear mapping matrix.

[0159] Step S2.3.2, the embedding representation of the image block obtained in step S2.3.1 is imported into the position encoding, and the position encoding is added to provide the position information of each image block.

[0160] z i =[z 1 ;z 2 ;...;z N ]+E pos

[0161] Where z i represents the feature vector of the i-th image block after adding the position encoding, and E pos represents the position encoding vector.

[0162] Step S2.3.3, the position encoding vector obtained in step S2.3.2 is input into the Transformer encoder, and the multi-head self-attention mechanism is used to capture the global features, and the feedforward neural network is used to process the embedding, so as to analyze and extract the features of the image block more deeply, and integrate the feature relationship of different subspaces to obtain a global feature map.

[0163] Step S2.3.4, use the global average pooling on the global feature map obtained in step S2.3.3 to average the feature map of each channel in the spatial dimension (height and width), and convert the high-dimensional feature map processed by multiple Transformer encoders into a low-dimensional representation that can be used for classification, to generate a global feature vector containing multi-scale spatial information and global semantic information. The specific steps are as follows:

[0164] Step S2.3.4.1, perform global average pooling on the feature map processed by the ViT network. The calculation formula is as follows:

[0165]

[0166] where g represents the global feature vector; N represents the number of blocks.

[0167] Step S2.3.4.2, the global feature vectors of the multi-scale features F3', F5" and F7' obtained in step S2.3.4.1 are fused to obtain the final global feature vector. The calculation formula is as follows:

[0168] F final =Concat(g3,g5,g7)

[0169] where g3, g5, g7 are global feature vectors of feature maps of different scales, respectively.

[0170] Step S2.4, the feature map obtained in step S2.3 is passed to the detection network of YOLOv7 for pedestrian detection. The detection network can output a bounding box, a class label and a confidence score for each detected pedestrian.

[0171] Step S3, based on the target pedestrian detection box of the current frame obtained in step S2, the DeepSORT multi-target pedestrian tracking algorithm is used to track the pedestrians to obtain the motion trajectory of each pedestrian. The specific steps are as follows:

[0172] Step S3.1, the current frame detection result extracted from step S2.3, including pedestrian detection box information, DeepSORT receives these detection boxes to track the target pedestrians. The target pedestrian detection box of the current frame is:

[0173] B t ={b1,b2,…,b n}

[0174] where b i =[x,y,w,h] represents the i-th detected pedestrian bounding box, x, y is the center coordinate of the bounding box, w, h is the width and height of the bounding box.

[0175] Step S3.2, for each target pedestrian detection box b i , DeepSORT uses a deep feature extraction network to extract the appearance feature vector f t of the target pedestrian. The appearance feature vector is used to match between target pedestrians. The steps of using ResNet as an appearance feature extractor are as follows:

[0176] Step S3.2.1, use the target pedestrian detection box to crop the target pedestrian region from the input image I t :

[0177]

[0178] wherein, is the target pedestrian region cropped from the input image I t , with size w t ×h t .

[0179] Step S3.2.2, adjust the target pedestrian region obtained in step S3.2.1 to a fixed image size, and fix the image size to 224×224.

[0180] Step S3.2.3, input the target pedestrian image adjusted in step S3.2.2 into ResNet for feature extraction to obtain an appearance feature vector g.

[0181] Step S3.2.4, combine the current frame target pedestrian appearance feature vector g obtained in step S3.2.3 and the target pedestrian bounding box information obtained in step S3.1 to generate a final appearance feature vector f t of the target pedestrian through global average pooling.

[0182] Step S3.3, based on the current position of the target pedestrian, a Kalman filter is used to estimate the position and speed of each detected pedestrian target, and the position of each pedestrian target in the next frame is predicted. The steps are as follows:

[0183] Step S3.3.1, the state of the target pedestrian in the current frame is calculated using the Kalman filter, and the motion trajectory of the target pedestrian in the next frame is predicted. The state transition equation of the target pedestrian is as follows:

[0184] x t =F·x t-1 +B·u t +ω t

[0185] wherein, x t =[x,y,v x ,v y ] T is the state of the target pedestrian, x and y represent the two-dimensional position of the target pedestrian, v x and v y represent the speed of the target pedestrian in the x and y directions respectively. F is the state transition matrix, which describes the dynamic change of the target pedestrian from the last frame to the current frame, x t-1 is the state of the target pedestrian in the last frame, B·u t is the control input term, ω t is the process noise, which represents the random disturbance caused by model error and external factors.

[0186] Step S3.3.2, the predicted state of the target pedestrian at the next frame It can be represented as:

[0187]

[0188] Predict the position and velocity of each pedestrian target and output the predicted state.

[0189] Step S.3.4: The Hungarian algorithm is used to associate the detection box of the current frame with the target tracked in the previous frame to obtain the trajectory of the target pedestrian. The specific steps are as follows:

[0190] Step S3.4.1, based on the appearance features f of the current frame extracted in step S3.2 t The pedestrian appearance features f from the previous frame t-1 The appearance similarity between the target pedestrian in the current frame and the target pedestrian in the previous frame is calculated using Euclidean distance. The Euclidean distance is calculated as follows:

[0191] d appearance =||f t -f t-1 ||2

[0192] Among them, f t and f t-1 It represents the appearance feature vectors of the target pedestrians in the current frame and the previous frame, where ||·|| denotes the norm of the vectors. A larger Euclidean distance indicates that the pedestrians are more similar in appearance.

[0193] Step S3.4.2: The similarity of motion characteristics between target pedestrians is evaluated by calculating the difference in motion characteristics between their positions and velocities using Euclidean distance. The similarity calculation is as follows:

[0194]

[0195] Among them, (x t ,y t ) and (x t-1 ,y t-1 These are the center positions of the target pedestrian in the current frame and the previous frame, respectively, which are the coordinates of the center position of the detection box.

[0196] Step S3.4.3: The Hungarian algorithm is used to associate the detection boxes in the current frame with the tracked targets in the previous frame to determine the pedestrian target corresponding to each detection box. A cost matrix for target pedestrian matching is generated based on the weighted similarity of motion and appearance features. The total similarity is calculated as follows:

[0197] d total =α·d motion +β·d appearance

[0198] Wherein, a and β are weight coefficients, used to adjust the proportion of motion information and appearance information in the matching process.

[0199] Step 3.5, based on the matching results obtained in step S3.4, update the Kalman filter state, create a new target or delete a lost target, output the motion state of each tracking target, including position, velocity and trajectory.

[0200] A pedestrian detection and tracking system based on visual attention mechanism, the system comprises a YOLO-ViT pedestrian detection module and a DeepSORT pedestrian tracking module, the YOLO-ViT pedestrian detection module is composed of a YOLOv7 target pedestrian detection network and a ViT feature extraction network, and is used for detecting pedestrians; the DeepSORT pedestrian tracking module is built-in DeepSORT multi-target pedestrian tracking algorithm, and is used for tracking pedestrians.

[0201] More specifically, in the present embodiment, the preprocessed data is divided into training set and test set according to the ratio of 6:4 using MOT20 dataset, and is used for training and testing YOLO-ViT detection model, YOLOv7 target pedestrian detection network, ViT feature extraction network and DeepSORT multi-target pedestrian tracking algorithm in DeepSORT pedestrian tracking module.

[0202] The above embodiments are only used to illustrate the design idea and characteristics of the present application, and its purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and the protection scope of the present application is not limited to the above embodiments. Therefore, any equivalent changes or modifications made according to the principles and design ideas disclosed by the present application are within the protection scope of the present application.

Claims

1. A pedestrian detection and tracking method based on visual attention mechanism, characterized in that, The method comprises the following steps: Step S1, acquiring a continuous video stream and performing preprocessing Step S2, inputting the preprocessed continuous video stream, and obtaining a target pedestrian detection frame of a current frame through a YOLO-ViT detection model; the YOLO-ViT detection model comprises a YOLOv7 target pedestrian detection network and a ViT feature extraction network; a backbone network of the YOLOv7 target pedestrian detection network performs feature extraction on an input image to obtain three feature maps of different sizes; a neck network of the YOLOv7 target pedestrian detection network adopts an FPN feature pyramid network and a PAN path aggregation network structure to perform feature fusion, and obtains three reinforced feature maps of different sizes; the ViT feature extraction network performs feature fusion on the three reinforced feature maps of different sizes to obtain a final feature map; and a detection head of the YOLOv7 target pedestrian detection network performs pedestrian detection based on the final feature map, and the detection head can output a bounding box, a category label and a confidence score for each detected pedestrian; Step S3, tracking pedestrians based on the pedestrian output bounding box of the current frame obtained in step S2 by using a DeepSORT multi-target pedestrian tracking algorithm to obtain a motion trajectory of each pedestrian. 2.The pedestrian detection and tracking method based on visual attention mechanism according to claim 1, wherein, The preprocessing in step S1 comprises image size normalization and pixel normalization. 3.The pedestrian detection and tracking method based on visual attention mechanism according to claim 1, characterized in that, The steps of feature extraction performed by the backbone network of the YOLOv7 target pedestrian detection network are as follows: Step S2.1.1, an initial convolutional layer is used for initializing feature extraction; the initial convolutional layer applies a 3x3 convolutional kernel to process the input image, and performs down-sampling with a 2-step length, and the calculation formula is as follows: F stem = Conv(I, K stem , stride = 2) wherein I is a normalized input image, the size of which is 224x224x3, K stem is a convolution kernel, the size of which is 3x3x3x32, and the size of an output feature map is 112x112x32; After batch normalization and Swish activation function processing, the normalized layer outputs, and the internal offset is reduced, and the calculation formula is as follows: F stem = Swish(BatchNorm(Conv(I, K stem , stride=2))) wherein F stem is a feature map processed by batch normalization and Swish activation function; Step S2.1.2, an MBConv module is used to extract more complex features, and each MBConv module comprises a depth convolution and a 1x1 point convolution, which are used to extract features of different levels; A depth separable convolution is applied to each input channel independently, and the calculation formula is as follows: F dw = DepthwiseConv(F in ,K dw ) wherein F in is the output feature map of the previous layer, K dw is a deep convolution kernel with a stride of 2; A 1x1 convolution is used to map the output of the depth convolution to a new channel space, and the extracted features are enhanced, and the calculation formula is as follows: F pw = PointwiseConv(F dw , K pw ) wherein K pw is a 1x1 convolution kernel for changing the number of channels; After the expansion convolution and the point-by-point convolution, the batch normalization and the Swish activation function are used to process the feature map, and the normalized layer outputs, and the calculation formula is as follows: F MBConv = Swish(BatchNorm(PointwiseConv(F dw , K pw ))) wherein F MBConv is a feature map processed by batch normalization and Swish activation function; Step S2.1.3, feature maps are extracted from different levels of the network to capture information of different scales and levels; the feature maps are extracted from the third, fifth and seventh layers of the network, and the steps are as follows: Step S2.1.3.1, the third layer feature map is extracted, and the third layer feature map contains low-level features of edges and textures, and the calculation formula is as follows: F3=Conv(F2,K3,stride=2) Wherein, F2 is the output feature map of the second layer, and F3 is the output feature map of the third layer; Step S2.1.3.2, the fifth layer feature map is extracted, and the calculation formula is as follows: F5=Conv(F4,K5,stride=2) Wherein, F4 is the output feature map of the fourth layer, and F5 is the output feature map of the fifth layer; Step S2.1.3.3, extract the seventh layer feature map, and the calculation formula is as follows: F7=Conv(F6, K7, stride=2) Wherein, F6 is the output feature map of the sixth layer, and F7 is the output feature map of the seventh layer.

4. The method of claim 3, wherein the method further comprises: The neck network of the YOLOv7 target pedestrian detection network adopts the FPN feature pyramid network and the PAN path aggregation network structure to perform the steps of feature fusion as follows: Step S2.2.1, using the FPN structure, the features are fused through the top-down path, the high-level strong semantic feature maps of the backbone network are gradually reduced in dimension and passed down, the bottom layer feature maps are given more rich semantic information, and the fuzzy pedestrian target is helped to be recognized by the bottom layer feature map; including the following steps: Step S2.2.1.1, upsample the output feature map F7 of the seventh layer to the same size as the output feature map F5 of the fifth layer using an upsampling operation, to obtain a feature map F 7↑ ; Step S2.2.1.2, element-wise addition is performed on the output feature map F5 of the fifth layer to obtain a feature map F5' containing high-level details and middle-level semantic information. 7↑ Step S2.2.1.2, element-wise addition is performed on the output feature map F5 of the fifth layer to obtain a feature map F5' containing high-level details and middle-level semantic information. Step S2.2.1.

3. Further up-sampling the feature map obtained in step S2.2.1.2 to the same size as the low-level feature map to obtain a feature map F 5↑ ′; Step S2.2.1.4, element-wise addition is performed between the feature map F3 obtained in step S2.2.1.3 and the low-level feature map F3, to obtain a feature map F3' that integrates high-level, middle-level and low-level information. 5↑ ′ and the low-level feature map F3, to obtain a feature map F3' that integrates high-level, middle-level and low-level information. Step S2.2.2, using the PAN structure, the features are enhanced through the bottom-up path, including the following steps: Step S2.2.2.1, using maximum pooling to down-sample the low-layer fusion feature map F3', local feature extraction is performed on the bottom layer fusion feature map, and the most significant features are extracted; Step S2.2.2.2, the feature map obtained in step S2.2.2.1 is added to the middle layer fusion feature map F5' element by element, and a feature map F5'' containing bottom layer details and middle layer semantic information is obtained; Step S2.2.2.3, the feature map obtained in step S2.2.2.2 is further down-sampled to the size of the high layer feature map; Step S2.2.2.4, the feature map F5'' obtained in step S2.2.2.3 and the high layer fusion feature map F7 are added element by element, and a feature map F7' containing high layer details and bottom layer semantic information is obtained.

5. The method of claim 1, wherein the method further comprises: The steps of using the ViT feature extraction network to perform feature fusion on the three different sizes of enhanced feature maps to obtain the final feature map in step S2.3 are as follows: Step S2.3.1, the feature map is preprocessed first, the input image is segmented according to a fixed size block, each small block is stretched into a vector, and a linear layer is used for mapping to obtain the embedding representation of the image block, and the calculation formula is as follows: z i = x i E where x i represents the flattened vector of the i-th image block, i = 1,..., N, E is a linear mapping matrix; Step S2.3.2, the embedding representation of the image block obtained in step S2.3.1 is imported into the position coding, and the position coding is added to provide the position information of each image block, which is represented as: wherein z i adding the position encoded feature vector, E pos denotes the position encoding vector; Step S2.3.3, the position coding vector obtained in step S2.3.2 is input into the Transformer encoder, the multi-head self-attention mechanism is used to capture the global features, and the feedforward neural network is used to process the embedding, so as to analyze and extract the features of the image block more deeply, and the feature relationship in different subspaces is combined to obtain a global feature map; Step S2.3.4, the feature map obtained in step S2.3.3 is used to perform global average pooling on each channel feature map in the spatial dimension, and the high-dimensional feature map processed by the multiple Transformer encoders is converted into a low-dimensional representation that can be used for classification, and a global feature vector containing multi-scale spatial information and global semantic information is generated.

6. The pedestrian detection and tracking method based on visual attention mechanism according to claim 5, characterized in that, Step S2.3.4, generate a global feature vector; Step S2.3.4.1, global average pooling is performed on the feature map processed by the ViT module, and the calculation formula is as follows: where g denotes the global feature vector; N denotes the number of blocks, z i denotes the feature vector of each block; Step S2.3.4.2, the global feature vectors of the multi-scale features F3', F5" and F7' obtained in step S2.3.4.1 are fused to obtain the final global feature vector, and the calculation formula is as follows: F final = Concat(g3, g5, g7) Wherein, g3, g5, g7 are global feature vectors of feature maps of different scales.

7. The method of claim 1, wherein the visual attention mechanism is based on a convolutional neural network. The specific steps of step S3 for obtaining the pedestrian trajectory by using the DeepSORT algorithm are as follows: Step S3.1, the detection result of the current frame of the video extracted in step S2.3 includes pedestrian detection box information, and DeepSORT receives these detection boxes to track the target pedestrians, and the target pedestrian detection box in the current frame is as follows: B t = {b1, b2,..., b n} wherein b i = [x, y, w, h] represents the detected i-th pedestrian bounding box, x, y are the center coordinates of the bounding box, w, h are the width and height of the bounding box; Step S3.2, for each target pedestrian detection box b i DeepSORT uses a deep feature extraction network to extract the appearance feature vector f t of the target pedestrian Step S3.3, based on the current position of the target pedestrian, a Kalman filter is used to estimate the position and speed of each detected pedestrian target, and the position of each pedestrian target in the next frame is predicted in the current frame; Step S3.4, the Hungarian algorithm is used to associate the detection boxes in the current frame with the targets tracked in the previous frame to obtain the trajectory of the target pedestrian; Step S3.5, based on the matching result obtained in step S3.4, the Kalman filter state is updated, new targets are created or lost targets are deleted, and the motion state of each tracking target is output, including position, speed and trajectory. 8.The pedestrian detection and tracking method based on visual attention mechanism according to claim 7, characterized in that, extracting the appearance feature vector f of the target pedestrian t The steps are as follows: Step S3.2.1, cropping a target pedestrian region from the input image I using the target pedestrian detection box t Step S3.2.1, cropping a target pedestrian region from the input image I using the target pedestrian detection box wherein, is a target pedestrian region cropped out from the input image I t in the input image I Step S3.2.2, the target pedestrian region obtained in step S3.2.1 is adjusted to a fixed size; Step S3.2.3, the target pedestrian image adjusted in step S3.2.2 is input into ResNet for feature extraction to extract the appearance feature map; Step S3.2.4, taking the target pedestrian region of the current frame obtained in step S3.2.1 and the feature map obtained in step S3.2.3 as inputs to perform feature extraction, and converting the feature map into a fixed-length vector by using global average pooling as the appearance feature f of the target pedestrian t . 9.The pedestrian detection and tracking method based on visual attention mechanism according to claim 7, characterized in that, The method for predicting the position of each pedestrian target in the next frame in the current frame is as follows: Step S3.3.1, the state of the target pedestrian in the current frame is calculated by using the Kalman filter, and the motion trajectory of the target pedestrian in the next frame is predicted, and the state transition equation of the target pedestrian is as follows: x t = F · x t-1 + B · u t + ω t Where, x t =[x,y,v x ,v y ] T This represents the state of the target pedestrian, where x and y represent the two-dimensional position of the target pedestrian, and v... x and v y Let x and y represent the velocities of the target pedestrian in the x and y directions, respectively; F is the state transition matrix, describing the dynamic changes of the target pedestrian from the previous frame to the current frame, where x... t-1 It represents the state of the target pedestrian in the previous frame, B·u t It is a control input item, ω t Process noise represents random disturbances caused by model errors and external factors; Step S3.3.2, predicting the state of the target pedestrian in the next frame through the state transition equation obtained in step S3.3.1; the predicted state of the target pedestrian at the next frame may be represented as: predicting a position and a velocity of each pedestrian object, outputting a predicted state 10. The pedestrian detection and tracking method based on visual attention mechanism according to claim 7, characterized in that, The method for obtaining the trajectory of the target pedestrian by using the Hungarian algorithm in step S3.4 is as follows: Step S3.4.1, calculating the appearance similarity between the target pedestrian in the current frame and the target pedestrian in the previous frame based on the appearance feature f of the current frame extracted in step S3.2 t and the appearance feature f of the pedestrian in the previous frame t-1 , using the Euclidean distance to calculate the appearance similarity between the target pedestrian in the current frame and the target pedestrian in the previous frame; the Euclidean distance is calculated as follows: d appearance =||f t -f t-1 ||2 wherein f t and f t-1 are the target pedestrian appearance feature vectors of the current frame and the previous frame, respectively, and ||·|| denotes the norm of a vector; the greater the Euclidean distance, the more similar the appearance of the pedestrian. Step S3.4.2, the motion feature difference between the positions and speeds of the target pedestrians is calculated by using the Euclidean distance to evaluate the motion feature similarity between the target pedestrians, and the similarity calculation is as follows: wherein (x t ,y t ) and (x t-1 ,y t-1 ) are the target pedestrian center positions of the current frame and the previous frame, respectively, i.e., the center position coordinates of the detection frame; Step S3.4.3, the Hungarian algorithm is used to associate the detection boxes in the current frame with the targets tracked in the previous frame to determine the pedestrian target corresponding to each detection box; a cost matrix for matching the target pedestrians is generated according to the weighted similarity of the motion features and the appearance features, and the total similarity calculation is as follows: d total = a · d motion + β · d appearance Wherein, α and β are weight coefficients, which are used to adjust the proportion of motion information and appearance information in the matching process.

Images