Classroom scene identity recognition method based on multi-level information fusion transformer

Abstract

The application discloses a classroom scene identity recognition method based on a multi-level information fusion Transformer, and belongs to the technical field of image detection.The application comprises the following steps: sampling and processing a video, sending the video into a target detection network, obtaining a person candidate frame based on the output of the target detection network, extracting features of the person candidate frame based on a multi-level information fusion Transformer network, obtaining and saving key feature information of the person candidate frame, extracting features of the person candidate frame of a target person based on the multi-level information fusion Transformer network, and obtaining key feature information of the target person; and determining whether the target person exists in a video frame based on similarity comparison between the key feature information.The application can effectively detect and recognize specific persons, has good robustness, and has good effects in sparse crowds and dense crowds.Meanwhile, the module can provide strong assistance for the construction of a smart classroom.

Description

Technical Field

[0001] This invention belongs to the field of image detection technology, specifically relating to a classroom scene identity recognition method based on multi-level information fusion Transformer. Background Technology

[0002] With the development of computer vision, visual technology is widely used in people's lives. Among these applications, classroom scene recognition of people is a crucial and challenging topic. Classroom scenes involve numerous occlusions and overlaps, and targets move freely within the scene, exhibiting a high degree of freedom. Furthermore, due to the unique characteristics of classrooms, the videos captured by cameras typically suffer from distortion, resulting in poor video quality. By designing classroom scene detection and recognition algorithms, intelligent functions such as smart classrooms, automated attendance tracking, and identity verification can be realized. Additionally, the ability to analyze classroom quality can be beneficial for teachers in understanding their teaching styles and methods. Identity recognition is essentially a feature similarity matching problem; however, in classroom scenes, the targets are typically highly mobile, and the camera angles are not uniform, leading to significant differences in features between the same target, thus increasing the technical difficulty of identity recognition. Summary of the Invention

[0003] The present invention aims to provide a method for real-time detection and identity recognition in classroom scenarios, so as to improve the target recognition performance in classroom scenarios and better assist the quality of classes.

[0004] The technical solution adopted in this invention is as follows:

[0005] A classroom scene identity recognition method based on multi-level information fusion Transformer includes the following steps:

[0006] Step 1: The video is sampled using an equal-interval sampling method, and each sampled video frame is fed into the target detection network. The candidate bounding boxes of people are obtained based on the output of the target detection network.

[0007] Step 2: Extract features from the candidate bounding boxes of people based on a multi-level information fusion Transformer network, obtain the key feature information of the candidate bounding boxes of people, and save it.

[0008] Step 3: Extract features from the candidate bounding boxes of the target person using a multi-level information fusion Transformer network to obtain the key feature information of the target person;

[0009] The key feature information of the target person is compared with the key feature information of each candidate frame of the person saved in step 2. If the similarity between the key feature information of the current candidate frame of the person and the key feature information of the target person is greater than the similarity threshold, it means that the video frame corresponding to the current candidate frame of the person contains the target person; otherwise, the video frame corresponding to the current candidate frame of the person does not contain the target person. The target person recognition result of each video frame is obtained.

[0010] Preferably, in step 1, the target detection network uses the YOLOv5 detection network and is trained based on sample images of crowded-human populations.

[0011] Preferably, the Transformer network based on multi-level information fusion specifically includes:

[0012] The feature maps of the candidate person boxes are extracted in four stages through a four-layer ViT (Vision Transformer) encoder: feature map f1 is obtained through the first layer ViT encoder, feature map f1 is then input into the second layer ViT encoder to obtain feature map f2, feature map f2 is then input into the third layer ViT encoder to obtain feature map f3, and finally feature map f3 is input into the fourth layer ViT encoder to obtain feature map f4.

[0013] The multi-level information fusion Transformer network includes a module focusing on local region features, consisting of 6 local convolutional fusion modules, and a module focusing on global features, consisting of 3 global content information modules.

[0014] In this module, the inputs to the first local convolutional fusion module are feature maps f1 and f2; the inputs to the second local convolutional fusion module are the output of the third local convolutional fusion module and feature map f3; and the inputs to the third local convolutional fusion module are the output of the second local convolutional fusion module and feature map f4. The output of the third local convolutional fusion module is denoted as feature map f. c1 The input to the fourth local convolutional fusion module is the output of the first and second local convolutional fusion modules; the input to the fifth local convolutional fusion module is the output of the third and fourth local convolutional fusion modules, and the output of the fifth local convolutional fusion module is denoted as feature map f. c2 The input to the sixth local convolutional fusion module is the output of the fourth and fifth local convolutional fusion modules, and the output of the sixth local convolutional fusion module is denoted as feature map f. c3 ;

[0015] The input to the first global content information module is the block features output by the block segmentation module after feature maps f1 and f2 have passed through the block segmentation module, and the block features of feature map f1 after passing through the block segmentation module are used as the low-level features I of the feature map of the first global content information module. sThe block features after feature map f2 is used as the high-level feature I of the feature map of the first global content information module. s+1 The input to the second global content information module is the output of the first global content information module and the block features output by the block segmentation module after feature map f3 is processed by the block segmentation module. The output of the first global content information module is used as the low-level feature I of the feature map of the second global content information module. s The block features after feature map f3 block segmentation are used as the high-level features I of the feature map of the second global content information module. s+1 The input to the third global content information module is the output of the second global content information module and the block features output by the block segmentation module after feature map f4 is processed by the block segmentation module. The output of the second global content information module is used as the low-level feature I of the feature map of the third global content information module. s The block features after the feature map f4 block module are used as the high-level features I of the feature map of the third global content information module. s+1 And the output of the third global content information module is denoted as feature map f. t ;

[0016] splicing feature map f c1 f c2 f c3 f4 and f t The output feature vector of the multi-level information fusion Transformer network is obtained, thereby obtaining the key feature information of the candidate person box and the key feature information of the target person;

[0017] The local convolutional fusion module includes two convolutional blocks. The output feature maps of the two convolutional blocks are concatenated and then fed into another convolutional block to obtain the output feature map of the local convolutional fusion module. The convolutional block includes convolution operation, batch normalization operation and activation mapping operation in sequence.

[0018] The processing of the global content information module is specifically as follows:

[0019] For low-level feature I s Attention enhancement is performed in both the channel and spatiotemporal dimensions to obtain the enhanced feature map F. s ;

[0020] Feature map F s The feature map K is obtained by sequentially passing it through a convolutional block and a convolutional layer;

[0021] The features are normalized using the sofimax function, and the 0th dimension of the channel dimension is extracted as the main representation of the feature to obtain the main feature θ(K);

[0022] High-level feature I s+1 After one convolutional layer W qThe feature map W is obtained. q (I s+1 );

[0023] The main feature θ(K) and feature map W q (I s+1 Multiply by the product, then feed into a global pooling layer to obtain the feature map K. fuse ;

[0024] High-level feature I s+1 After one convolutional layer W q The feature map W is obtained. q (I s+1 ); By applying the Hadamard product to the feature map K fuse and W q (I s+1 The relationships are then fused to obtain the relation matrix f;

[0025] High-level feature I s+1 The feature map V is obtained by passing it through two convolutional modules. s and V c , feature map V s After taking the inner product with f, the result is fed into the structure enhancement module for structural information enhancement, resulting in the structure-enhanced feature map VF. s and the feature map V c After performing an inner product with f, the result is fed into the content activation module for content activation, yielding the content-activated feature map VF. c ;

[0026] The relation matrix f is related to the feature map VF and the feature map VF respectively. c Perform inner product operation to obtain the restored features, concatenate the two restored features, and use a convolution module to adjust the dimensions (adjust the feature dimensions to the desired dimension value) to obtain the output feature map of the global content information module.

[0027] Preferably, during training, the feature map f c1 f c2 and f c3 Then, each component passes through a GCP-NK module to obtain the first identity classification loss. Second identity classification loss and third identity classification loss The GCP-NK module consists of a global covariance pooling layer and a bottleneck layer network. The feature map f4 is passed through a GCP-NK module to obtain the fourth identity classification loss. The output of the third global content information module is passed through a GCP-NK module to obtain the fifth identity classification loss. The identity classification loss of the multi-level information fusion Transformer network is obtained by summing the five identity classification losses. The total network loss of the multi-level information fusion Transformer network during training is obtained by comparing the identity classification loss with the output feature vector of the multi-level information fusion Transformer network. This loss is used to train and update the network parameters of the multi-level information fusion Transformer network.

[0028] Preferably, in step 3, the similarity between the key feature information of the target person and the key feature information of each candidate box is calculated using the Euclidean distance between the key feature information.

[0029] The technical solution provided by this invention brings at least the following beneficial effects:

[0030] This invention can effectively detect and identify specific individuals, exhibiting good robustness and performing well in both sparse and dense crowds. Furthermore, this module can provide significant support for the construction of smart classrooms. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of the processing procedure of the classroom scene identity recognition method based on multi-level information fusion Transformer provided in the embodiments of the present invention;

[0033] Figure 2 This is a schematic diagram of the structure of a multi-level information fusion Transformer network used in an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram illustrating the processing of the global content information module based on a multi-level information fusion Transformer network in an embodiment of the present invention.

[0035] Figure 4 In an embodiment of the present invention, the feature effect diagram extracted by the method of the present invention from a public dataset is shown. Figures (4-a) and (4-b) are schematic diagrams comparing the processing results of two images of different people. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0037] like Figure 1 As shown, this embodiment of the invention provides a classroom scene identity recognition method based on a multi-level information fusion Transformer, which includes the following steps:

[0038] Step S1: The collected video of the classroom scene is sampled using an equal-interval sampling method. Then, each sampled video frame is sent into the object detection network. Based on the output of the object detection network, candidate boxes for people are obtained.

[0039] Specifically, in this embodiment of the invention, the target detection network is the YOLO V5 detection network.

[0040] Step S2: Extract features from the candidate bounding boxes of people based on the multi-level information fusion Transformer network (i.e., the key feature information extraction network) to obtain and save the key feature information of the candidate bounding boxes of people.

[0041] Specifically, such as Figure 2 As shown, the Transformer backbone network is first used to extract features from the candidate bounding boxes of people to obtain four-stage feature maps, denoted as f1, f2, f3, and f4. That is, the ViT (Vision Transformer) encoder obtains the four-stage feature maps. First, feature map f1 is obtained through the first-layer ViT encoder. Then, feature map f1 is input into the second-layer ViT encoder to obtain feature map f2. Next, feature map f2 is input into the third-layer ViT encoder to obtain feature map f3. Finally, feature map f3 is input into the fourth-layer ViT encoder to obtain feature map f4.

[0042] The upper part of the key feature information extraction network mainly consists of local convolutional fusion modules, while the lower part mainly consists of global content information fusion modules. It is worth noting that before being fed into the global content information fusion module, the image needs to be segmented into blocks to ensure stronger connections between the different blocks.

[0043] The input to the local convolutional fusion module is the feature maps (T) of two adjacent frames. s and T s+1 The two input feature maps are convolved by convolutional blocks, and the output feature maps of the convolutional blocks are concatenated and then convolved again to obtain the output feature map T of the local convolutional fusion module. c The convolutional block consists of convolution operation, batch normalization operation and activation mapping operation (RelU) in sequence.

[0044] In this embodiment of the invention, a total of 6 local convolutional fusion modules are included. The inputs of the first local convolutional fusion module are feature maps f1 and f2, the inputs of the second local convolutional fusion module are the output of the first local convolutional fusion module and feature map f3, and the inputs of the third local convolutional fusion module are the output of the second local convolutional fusion module and feature map f4. The output of the third local convolutional fusion module is denoted as feature map f. c1 The input to the fourth local convolutional fusion module is the output of the first and second local convolutional fusion modules; the input to the fifth local convolutional fusion module is the output of the third and fourth local convolutional fusion modules, and the output of the fifth local convolutional fusion module is denoted as feature map f. c2 The input to the sixth local convolutional fusion module is the output of the fourth and fifth local convolutional fusion modules, and the output of the sixth local convolutional fusion module is denoted as feature map f. c3 ; and feature map f c1 f c2 and f c3 The loss is then passed through a GCP-NK module (which includes a GCP (Global Covariance Pooling) layer and a BottleNcck (bottleneck) layer) to obtain the first identity classification loss. Second identity classification loss and third identity classification loss

[0045] Furthermore, feature map f4 also passes through a GCP-NK module to obtain the fourth identity classification loss.

[0046] In this embodiment of the invention, a total of three global content information modules are included. The first global content information module receives the block features output from feature maps f1 and f2 after passing through the block segmentation module. The second global content information module receives the output from the first global content information module and the block features output from feature map f3 after passing through the block segmentation module. The third global content information module receives the output from the second global content information module and the block features output from feature map f4 after passing through the block segmentation module. The output of the third global content information module passes through a GCP-NK module to obtain the fifth identity classification loss.

[0047] For the global content information module, the entire process is as follows: Figure 3 As shown, given feature map f s First, the block module is used to divide f s Divide into smaller sub-blocks This allows for better extraction of fine-grained features.

[0048] Among them, emb s HP represents the encoding length of this feature. S ×WPs This represents the spatial dimensions of the feature map after partitioning, and its value is set as follows:

[0049]

[0050] Among them, H s ×W s The space dimension of the feature map before partitioning is represented by l, where l represents the size of the sub-block.

[0051] Next, for the low-level feature I that enters the global content information module s In order to obtain richer features and advanced semantic information, this invention utilizes channel-dimensional and spatiotemporal-dimensional attention to enhance the features.

[0052] F s =Atn(I s )

[0053] Get again Next, it is necessary to integrate low-level feature information into high-level features. Therefore, this invention designs a relationship matrix f to represent the relationship between the main information of the two, and uses f to perform information fusion.

[0054] Specifically, the relation matrix f is constructed as follows: first, a convolution module β is used. s To adjust f s Dimensions to HP S+1 ×WP s+1 f s The adjusted space dimensions can be represented by Z. s+1 To represent this. Then, another convolutional layer W is used. k Extract its main information.

[0055] K = W k (β s (F s ))

[0056] Next, softmax is applied to the channel dimension of K, and the Lth column is extracted as its principal feature (where L is usually set to 0 empirically). This entire process is denoted by θ. Then, another convolutional layer W is used... q Extracting high-level features I s+1 The main information is then extracted. Next, the Hadamard product is used to integrate low-level feature information into high-level features to enhance the semantic information of the low-level features. However, this operation can lead to information imbalance; therefore, this invention employs Global Avgpooling to suppress information imbalance.

[0057] K fuse=GAP(W q (I s+1 )·θ(K))

[0058] Finally, the inner product is used to construct the relationship between low-level features and high-level features:

[0059]

[0060] Where σ represents the mapping of features to the (0, 1) interval using the sigmoid function.

[0061] After obtaining the relation matrix f, a deeper level of information fusion is performed using the relation matrix. First, two convolutional modules W are used. vs W vc Extracting high-level features I respectively s+1 Key information.

[0062] V s =W vs (I s+1 V c =W vc (I s+1 )

[0063] Among them, V s V c These represent the convolutional module W. vs W vc The extracted feature information.

[0064] After that, as Figure 3 As shown, the relation matrix f is compared with V respectively. s V c Perform an inner product and pass it into the structural enhancement module SE(·) and the content excitation CE(·) respectively. Then, use the relation matrix f to interact with the structurally enhanced V. s With enhanced content information V c We perform inner product calculations to restore the features.

[0065] More specifically, in this embodiment of the invention, an MLP (Multilayer Perceptron) is used to enhance the content information, and a 1D convolutional layer (a one-dimensional convolutional layer) is used to enhance the spatial domain information. Next, the restored features are concatenated, and another convolutional layer is used for dimensionality adjustment. Finally, the results are combined with the original feature I. s+1 Add the results to prevent gradient vanishing. Figure 3 In the middle, V s V c These represent the two original features I extracted through convolution. s+1 Key information, VF s VF cS1(·) and S2(·) represent the features fed into the structure enhancement module and the content enhancement module after fusion with low-level information, respectively. S1(·) and S2(·) represent the two convolutional modules, 16×Z. s+1 Z represents the dimension of the feature map, 16 represents the number of channels. s+1 The spatial dimensions of the feature map, i.e., HP S+1 ×WP s+1 .

[0066] For the local convolutional fusion module, given low-level features T s and advanced features T s+1 First, the convolution module μ1 is used to adjust the dimension and resolution of the low-level features, and then the convolution module μ2 is used to extract information from the high-level features. Finally, the two features are concatenated, and the dimensions are adjusted using the convolution module μ3. The entire process can be represented by the following formula:

[0067] T c =μ3(cat(μ1(T) s ), μ2(T s+1 )))

[0068] For training this model, this embodiment of the invention employs a classification loss L. id To identify the person and compare the loss L t Given a set of comparisons {a, p, n}, the loss function can be described by the following formula:

[0069] L = L id +L t

[0070] L id =-y i log(p i )

[0071]

[0072] Among them, y i Let p represent the true label (related to identity) of the i-th sample. i Let f represent the predicted label (identity-related) for the i-th sample. a f p and f n represents the features of a random sample, a positive example feature of the random sample, and a negative example feature of the random sample, respectively.

[0073] Step S3: Based on a multi-level information fusion Transformer network, feature extraction is performed on the candidate bounding boxes of the target person to obtain the key feature information of the target person;

[0074] The key feature information of the target person is compared with the key feature information of each candidate frame of the person saved in step 2. If the similarity between the key feature information of the current candidate frame of the person and the key feature information of the target person is greater than the similarity threshold, it means that the video frame corresponding to the current candidate frame of the person contains the target person; otherwise, the video frame corresponding to the current candidate frame of the person does not contain the target person. The target person recognition result of each video frame is obtained.

[0075] like Figure 2 As shown, after obtaining the feature maps fc1, fc2, and fc3 processed by local convolutional fusion, the fc1, fc2, and fc3 obtained directly by the backbone network are... g (i.e., feature map f4), and f obtained from the global content information fusion module t The output feature map of the third global content information fusion module is then concatenated to serve as an indicator of matching features. Simultaneously, Euclidean distance is used to measure the feature similarity between the two.

[0076] out = β × mat + α × (mat1 @ mat2)

[0077] Where mat is a matrix of all 1s, α and β represent two preset weight coefficients, preferably set to α = --2 and β = 1 in this example, @ represents matrix multiplication, and mat1 and mat2 represent the two features to be compared.

[0078] For the overall model, the model of this invention outperforms the current mainstream and state-of-the-art models when tested on public datasets (MSMT17, Market150, Occluded-Duke).

[0079] As shown in Table 1:

[0080] Table 1. Experimental comparison with mainstream methods

[0081]

[0082] In Table 1, HOREID, ISP, RN, HAT, HA-CNN, PAT, BATNet, PGFA, SNR, RGA-SC, OSNet, and TransReID are the mainstream algorithms currently used for identity re-identification. mAP represents the average accuracy, and R1 represents the probability that the first image returned is correct. The resulting images are shown below. Figure 4 As shown, Figure 4 In the figures, (4-a) and (4-b) are schematic diagrams comparing the processing results of the two images, respectively. Figure 4In the image, column (a) is the original image; column (b) is the feature map output by the second ViT; column (c) is the feature map output by the third ViT; (d) is the feature map output by the fourth ViT; and (e) the feature vector extracted by the method of this invention (i.e., the key feature information of the candidate person box extracted by the multi-level information fusion Transformer network of this invention). It can be seen that after the improvement, the model can notice a wider and more accurate region.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0084] The above descriptions are merely some embodiments of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A classroom scene identity recognition method based on multi-level information fusion Transformer, characterized in that, Includes the following steps: Step 1: The video is sampled using an equal-interval sampling method, and each sampled video frame is fed into the target detection network. The candidate bounding boxes of people are obtained based on the output of the target detection network. Step 2: Extract features from the candidate bounding boxes of people based on a multi-level information fusion Transformer network, obtain the key feature information of the candidate bounding boxes of people, and save it. Step 3: Extract features from the candidate bounding boxes of the target person using a multi-level information fusion Transformer network to obtain the key feature information of the target person; The key feature information of the target person is compared with the key feature information of each candidate frame of the person saved in step 2. The similarity is calculated. If the similarity between the key feature information of the current candidate frame of the person and the key feature information of the target person is greater than the similarity threshold, it means that the video frame corresponding to the current candidate frame of the person contains the target person. Otherwise, the video frame corresponding to the current candidate frame of the person does not contain the target person. The target person recognition result of each video frame is obtained. Specifically, the Transformer network based on multi-level information fusion is as follows: Feature maps of person candidate boxes are extracted in four stages using a four-layer ViT encoder: the feature map is obtained through the first layer of the ViT encoder. Then the feature map The feature map is obtained by inputting the second-layer ViT encoder. Next, the feature map The feature map is obtained by inputting the third-layer ViT encoder. Finally, the feature map The feature map is obtained by inputting the fourth layer ViT encoder. ; The multi-level information fusion Transformer network includes a module focusing on local region features, consisting of 6 local convolutional fusion modules; and a module focusing on global features, consisting of 3 global content information modules. The input to the first local convolutional fusion module is the feature map. and The input to the second local convolutional fusion module is the output of the first local convolutional fusion module and the feature map. The input to the third convolutional fusion module is the output of the second local convolutional fusion module and the feature map. The output of the third local convolutional fusion module is recorded as the feature map. The input to the fourth local convolutional fusion module is the output of the first and second local convolutional fusion modules; the input to the fifth local convolutional fusion module is the output of the third and fourth local convolutional fusion modules, and the output of the fifth local convolutional fusion module is denoted as the feature map. The input to the sixth local convolutional fusion module is the output of the fourth and fifth local convolutional fusion modules, and the output of the sixth local convolutional fusion module is denoted as the feature map. ; The input to the first global content information module is the feature map. and The block features output by the block segmentation module are used as inputs to the second global content information module, which consists of the output of the first global content information module and the feature map. The block features output by the block segmentation module are used as input to the third global content information module, which takes the output of the second global content information module and the feature map as input. The block features output after the block segmentation module; The output of the third global content information module is denoted as the feature map. ; splicing feature map , , , and The output feature vector of the multi-level information fusion Transformer network is obtained, thereby obtaining the key feature information of the candidate person box and the key feature information of the target person.

2. The method as described in claim 1, characterized in that, In step 1, the object detection network used is the YOLOv5 detection network, which is trained based on sample images of crowded-human populations.

3. The method as described in claim 1, characterized in that, The Transformer network based on multi-level information fusion is specifically as follows: The local convolutional fusion module includes two convolutional blocks. The output feature maps of the two convolutional blocks are concatenated and then fed into another convolutional block to obtain the output feature map of the local convolutional fusion module. The convolutional block includes convolution operation, batch normalization operation and activation mapping operation in sequence. feature map The segmented features after the segmentation module are used as low-level features of the feature map of the first global content information module, and the feature map is... The segmented features after segmentation are used as high-level features of the feature map of the first global content information module; The output of the first global content information module is used as a low-level feature of the feature map of the second global content information module. The segmented features after segmentation are used as high-level features of the feature map of the second global content information module; The output of the second global content information module is used as a low-level feature of the feature map of the third global content information module. The segmented features after segmentation are used as high-level features of the feature map of the third global content information module; The processing of the global content information module is specifically as follows: use Indicates low-level features, Indicates high-level features; For low-level features Attention enhancement is performed in both the channel and spatiotemporal dimensions to obtain enhanced feature maps. ; Feature map The feature map is obtained by passing the data through a convolutional block and a convolutional layer in sequence. ; The features are normalized using the softmax function, and the 0th dimension of the channel dimension is extracted as the principal representation of the feature to obtain the main feature. ; High-level features After one convolutional layer , to obtain feature map ; Key features and feature map Multiply the results and then feed them into a global pooling layer to obtain the feature map. ; High-level features After one convolutional layer , to obtain feature map Feature maps are paired using the Hadamard product. and By merging, a relation matrix is ​​obtained. ; High-level features The feature maps are obtained by passing the data through two convolutional modules. and , feature map and After performing the inner product, the data is fed into the structure enhancement module for structural information enhancement, resulting in a structure-enhanced feature map. and feature maps and After performing the inner product, the data is fed into the content activation module for content activation, resulting in a content-activated feature map. ; Relationship matrix respectively with feature map and feature map Perform inner product operations to obtain the restored features, concatenate the two restored features, and use a convolution module to adjust the dimensions to obtain the output feature map of the global content information module.

4. The method as described in claim 3, characterized in that, When training a multi-level information fusion Transformer network, feature maps are used... , and Then, each component passes through a GCP-NK module to obtain the first identity classification loss. Second identity classification loss and third identity classification loss The GCP-NK module consists of a global covariance pooling layer and a bottleneck layer network; it processes feature maps... After passing through a GCP-NK module, the fourth identity classification loss is obtained. The output of the third global content information module is passed through a GCP-NK module to obtain the fifth identity classification loss. The identity classification loss of the multi-level information fusion Transformer network is obtained by summing the five identity classification losses. The total network loss of the multi-level information fusion Transformer network during training is obtained by comparing the identity classification loss with the output feature vector of the multi-level information fusion Transformer network. This loss is used to train and update the network parameters of the multi-level information fusion Transformer network.

5. The method as described in claim 1, characterized in that, In step 3, the similarity between the key feature information of the target person and the key feature information of each candidate box is calculated using the Euclidean distance between the key feature information.

Images