Specified pedestrian action retrieval method based on pedestrian re-identification algorithm

Abstract

The present invention discloses a method for retrieving the actions of designated pedestrians based on a pedestrian re-identification algorithm. The method for retrieving the actions of designated pedestrians based on a pedestrian re-identification algorithm is as follows: each frame of video data is input into a feature extraction backbone network, and the frame-level backbone network feature map is extracted and then input into a pedestrian detection branch module, which outputs the final target detection bounding box of each pedestrian after processing; the re-identification branch module processes the backbone network feature map and the final target detection bounding box of each pedestrian and outputs the action feature queue of each pedestrian; the action classification module uniformly scales the action feature queue of each pedestrian to a size of 288*288, and aggregates it in the channel dimension to extract the information of each pedestrian in the time dimension, and then performs action classification and obtains the final action retrieval result. The present invention adds pedestrian re-identification features and action recognition results that can continuously track designated targets, which greatly improves the accuracy of detecting pedestrians.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for retrieving a designated pedestrian action based on a pedestrian re-identification algorithm. Background Art

[0002] With the increasing amount of video data, a large number of computer vision tasks have been proposed to analyze video data. Among them, human action recognition has important value in many aspects of real life and has received more and more attention.

[0003] The current action recognition algorithm mainly uses the motion information of the target to complete the action classification. In simple experimental scenarios, this method has achieved good results. However, the video data in real life is often more complex, with a large number of pedestrians and frequent position movement and interaction between pedestrians. In this case, continuing to use the above method is likely to cause the problem of losing pedestrians, which in turn affects the correct recognition of the pedestrian's action. This requires us to further explore the role of appearance information in action recognition.

[0004] In sparse scenes with few people and no obstructions, the appearance information at this time only needs to be accurate enough to be recognized as a human by the pedestrian detection sub-algorithm. For example, people doing the same action cannot be divided into two categories because of differences in body shape or wearing different styles and colors of clothes. In complex scenes, the extracted appearance information needs to be rich enough to distinguish different pedestrians. For example, Yang Wenhao and others proposed to use facial information as the main feature to measure the similarity between different action senders, thereby reducing the problem of action misclassification caused by incorrect recognition of pedestrian IDs. However, facial features are difficult to collect when the pedestrian is facing away, the light is dim, or the distance is far, so it is necessary to use more universal pedestrian appearance features.

[0005] On the other hand, in practical applications such as customer behavior analysis in unmanned stores, search and rescue in security scenarios, and suspect tracking, it is generally necessary to identify all actions of a specific pedestrian in discrete time in order to obtain useful information after further analysis. For example, Ketan Kotecha et al. used non-deep learning methods to measure the similarity between targets, and cut the input video into independent video clips according to different pedestrians, and finally performed action classification tasks on the video clips. However, non-deep learning methods have poor generalization and it is difficult to identify the same pedestrian over a long period of time, so a more effective similarity measurement method is needed.

[0006] It can be seen that the action recognition algorithms in the prior art generally have the disadvantage of being easily misclassified due to the lack of appearance features, and the prior art cannot continuously track the action recognition results of the specified target. Summary of the invention

[0007] In view of the above problems, the present invention proposes a specified pedestrian action retrieval method based on a pedestrian re-identification algorithm.

[0008] To achieve the purpose of the present invention, the present invention proposes a method for retrieving a specified pedestrian action based on a pedestrian re-identification algorithm, comprising the following steps:

[0009] s1: input each frame of video data collected in real time by the video acquisition device into the feature extraction backbone network, the feature extraction backbone network processes each frame and extracts the backbone network feature map;

[0010] s2: input the backbone network feature map into the pedestrian detection branch module, the pedestrian detection branch module processes the backbone network feature map and outputs the final target detection bounding box of each pedestrian;

[0011] s3: inputting the backbone network feature map and the final target detection bounding box of each pedestrian into a re-identification branch module, the re-identification branch module processes the backbone network feature map and the final target detection bounding box of each pedestrian, and outputs the action feature queue of each pedestrian;

[0012] s4: The action classification module uniformly scales the action feature queues of the pedestrians to a size of 288*288, and aggregates them in the channel dimension to extract the information of the pedestrians in the time dimension, and then performs action classification to obtain the final action retrieval result.

[0013] Furthermore, the specific process of step s1 includes:

[0014] Each frame of the video data collected by the video acquisition device is sequentially input into the feature extraction backbone network, and the feature extraction backbone network extracts and obtains the backbone network feature map corresponding to each frame image, which is denoted as f. Among them, R represents the real space, W represents the width of the backbone network feature map f, H represents the height of the backbone network feature map f, D represents the spatial downsampling rate, and B represents the number of channels of the backbone network feature map f.

[0015] Furthermore, the pedestrian detection branch module includes: a bounding box center point prediction head subnetwork, a bounding box size prediction head subnetwork and a center point offset prediction head subnetwork;

[0016] The bounding box center point prediction head subnetwork, the bounding box size prediction head subnetwork and the center point offset prediction head subnetwork are respectively obtained through actual sample training;

[0017] The specific process of step s2 includes:

[0018] The backbone network feature map f is input into the bounding box center point prediction head sub-network, and the bounding box center point prediction head sub-network predicts the backbone network feature map f and outputs the thermal map of each pedestrian

[0019] Heat map of each pedestrian Use the loss function focal loss:

[0020] Among them, x and y represent the output heat maps of each pedestrian respectively. The horizontal and vertical coordinates of each element in , α and β represent the hyperparameters that control the contribution weight of the central point, represents the probability that a pedestrian target exists at the coordinate (x, y) as the center point, L x，y Indicates the true probability that a pedestrian target exists with the coordinate (x, y) as the center point;

[0021] The backbone network feature map f is input into the bounding box size prediction head sub-network, and the bounding box size prediction head sub-network predicts the backbone network feature map f and outputs the bounding box size of each pedestrian.

[0022] The bounding box size of each pedestrian Use the minimum absolute deviation loss function l1:

[0023] Among them, i∈[1,N] represents the index of the pedestrian, s i represents the true value of the i-th pedestrian bounding box size, represents the predicted value of the i-th pedestrian bounding box size, N represents the number of pedestrians in the current frame, and l size That is, the minimum absolute deviation loss function l1, size represents the prediction used here to constrain the size of the bounding box;

[0024] The backbone network feature map f is input into the center point offset prediction head sub-network, and the center point offset prediction head sub-network predicts the backbone network feature map f and outputs the offset of the center point of each pedestrian's bounding box in the two dimensions of length and width.

[0025] The offset of the center point of the bounding box of each pedestrian in the length and width dimensions Use the minimum absolute deviation loss function l1: l off That is, the minimum absolute value deviation loss function l1, off represents the network to which it belongs, o iRepresents the actual quantized offset of the i-th pedestrian; Represents the quantized offset of the predicted i-th pedestrian;

[0026] The heat map corresponding to each pedestrian Bounding box size The offset from the center of the bounding box in both length and width dimensions The candidate target detection bounding boxes of the pedestrians are combined, and then the NMS algorithm is used to deduplicate the candidate target detection bounding boxes of the pedestrians and filter out the bounding boxes with confidence lower than a threshold of 0.8, so as to obtain the final target detection bounding boxes of the pedestrians.

[0027] Furthermore, the re-identification branch module includes: a first pre-processing module, a first convolutional layer, a global average pooling layer and a post-processing module, the convolution kernel size of the first convolutional layer is 128; the re-identification branch module is obtained through actual sample training;

[0028] The specific process of step s3 includes:

[0029] The first preprocessing module cuts out the target feature map P of each pedestrian from the backbone network feature map f corresponding to each frame of the image according to the final target detection bounding box of each pedestrian. F，j , where F represents the frame number and j represents the pedestrian number in this frame;

[0030] The first convolutional layer generates a target feature map P of each pedestrian. F，j Perform feature extraction again and obtain the target feature map P′ with 128 channels F，j The average pooling layer transforms the target feature map P′ with a channel number of 128 F，j All pixels in the target feature map of each channel are added and averaged to obtain the spatial embedding feature E of each pedestrian with a length of 128. F，j , E F，j ∈R 128 , where E F，j Represents the spatial embedding feature corresponding to the jth pedestrian in this frame image;

[0031] The post-processing module embeds the spatial features E of each pedestrian F，j and the spatial embedding features E of each pedestrian in the previous frame F-1，j Compare them one by one, select the one with the smallest metric distance as the matching target, and then convert the target feature map P of each pedestrian into F，j Store the action feature queue Q corresponding to the pedestrian ID in the matching target P，id, where E indicates that the stored data is the spatial embedding features of each pedestrian, and P indicates that the stored data is the regional image in the final target detection bounding box of each pedestrian.

[0032] Furthermore, the action classification module includes: a second preprocessing module, a second convolutional layer group and a plurality of fully connected layers; the action classification module is obtained through actual sample training;

[0033] The specific process of step s4 includes:

[0034] The second preprocessing module converts the action feature queue Q of length K into P，id The size is uniformly scaled to 288*288, and it is aggregated in the channel dimension to extract the information of each pedestrian in the time dimension; then it passes through the second convolution layer group and multiple fully connected layers to output the feature vector A, A∈[0,1] num_action , that is, the prediction vector for the category to which each pedestrian belongs, which is also the final action retrieval result, where num_action represents the number of action types in the data set.

[0035] Compared with the prior art, the present invention has the following beneficial technical effects:

[0036] The present invention integrates pedestrian re-identification embedded features into the common action recognition algorithm, adopts a multi-task architecture that can be trained and optimized end-to-end, and solves the shortcomings of common action recognition algorithms that are prone to misclassification due to the lack of appearance features. Furthermore, in order to meet the growing demand for refined tracking, retrieval and analysis of specified pedestrians in video data, the present invention can continuously track the action recognition results of the specified pedestrian after using the image of the specified pedestrian to retrieve the pedestrian in the video for further high-level semantic reference and analysis; at the same time, the strategy of first retrieving the pedestrian and then performing action recognition can greatly reduce the required amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flowchart of a method for retrieving a specified pedestrian action based on a re-identification algorithm according to an embodiment;

[0038] Figure 2 is a schematic diagram of a flow chart of specifying a single pedestrian for action retrieval according to an embodiment;

[0039] Figure 3 is a schematic diagram of a process of performing group action retrieval by selecting some features in a query image according to an embodiment;

[0040] Figure 4 The diagram is a schematic diagram of the architecture of an attribute-based person re-identification model in a group action retrieval method according to an embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0043] refer to Figure 1 As shown, Figure 1 The figure is a flowchart of a method for retrieving a specified pedestrian action based on a re-identification algorithm according to an embodiment.

[0044] In this embodiment, the method for retrieving a specified pedestrian action based on a pedestrian re-identification algorithm includes the following steps:

[0045] s1: input each frame of video data collected in real time by the video acquisition device into the feature extraction backbone network, the feature extraction backbone network processes each frame and extracts the backbone network feature map;

[0046] s2: input the backbone network feature map into the pedestrian detection branch module, the pedestrian detection branch module processes the backbone network feature map and outputs the final target detection bounding box of each pedestrian;

[0047] s3: inputting the backbone network feature map and the final target detection bounding box of each pedestrian into a re-identification branch module, the re-identification branch module processes the backbone network feature map and the final target detection bounding box of each pedestrian, and outputs the action feature queue of each pedestrian;

[0048] s4: The action classification module uniformly scales the action feature queues of the pedestrians to a size of 288*288, and aggregates them in the channel dimension to extract the information of the pedestrians in the time dimension, and then performs action classification to obtain the final action retrieval result.

[0049] In one embodiment, the specific process of step s1 includes:

[0050] Each frame of the video data collected by the video acquisition device is sequentially input into the feature extraction backbone network, and the feature extraction backbone network extracts and obtains the backbone network feature map corresponding to each frame image, which is denoted as f. Among them, R represents the real space, W represents the width of the backbone network feature map f, H represents the height of the backbone network feature map f, D represents the spatial downsampling rate, and B represents the number of channels of the backbone network feature map f.

[0051] In one embodiment, the pedestrian detection branch module includes: a bounding box center point prediction head sub-network, a bounding box size prediction head sub-network and a center point offset prediction head sub-network;

[0052] The bounding box center point prediction head subnetwork, the bounding box size prediction head subnetwork and the center point offset prediction head subnetwork are respectively obtained through actual sample training;

[0053] The specific process of step s2 includes:

[0054] The backbone network feature map f is input into the bounding box center point prediction head sub-network, and the bounding box center point prediction head sub-network predicts the backbone network feature map f and outputs the thermal map of each pedestrian

[0055] Heat map of each pedestrian Use the loss function focal loss:

[0056] Among them, x and y represent the output heat maps of each pedestrian respectively. The horizontal and vertical coordinates of each element in , α and β represent the hyperparameters that control the contribution weight of the central point, represents the probability that a pedestrian target exists at the coordinate (x, y) as the center point, L x，y Indicates the true probability that a pedestrian target exists with the coordinate (x, y) as the center point;

[0057] The backbone network feature map f is input into the bounding box size prediction head sub-network, and the bounding box size prediction head sub-network predicts the backbone network feature map f and outputs the bounding box size of each pedestrian.

[0058] The bounding box size of each pedestrian Use the minimum absolute deviation loss function l1:

[0059] Among them, i∈[1,N] represents the index of the pedestrian, s i represents the true value of the i-th pedestrian bounding box size, represents the predicted value of the i-th pedestrian bounding box size, N represents the number of pedestrians in the current frame, and l sizeThat is, the minimum absolute deviation loss function l1, size represents the prediction used here to constrain the size of the bounding box;

[0060] The backbone network feature map f is input into the center point offset prediction head sub-network, and the center point offset prediction head sub-network predicts the backbone network feature map f and outputs the offset of the center point of each pedestrian's bounding box in the two dimensions of length and width.

[0061] The offset of the center point of the bounding box of each pedestrian in the length and width dimensions Use the minimum absolute deviation loss function l1: l off That is, the minimum absolute value deviation loss function l1, off represents the network to which it belongs, o i Represents the actual quantized offset of the i-th pedestrian; Represents the quantized offset of the predicted i-th pedestrian;

[0062] The heat map corresponding to each pedestrian Bounding box size The offset from the center of the bounding box in both length and width dimensions The candidate target detection bounding boxes of the pedestrians are combined, and then the NMS algorithm is used to deduplicate the candidate target detection bounding boxes of the pedestrians and filter out the bounding boxes with confidence lower than a threshold of 0.8, so as to obtain the final target detection bounding boxes of the pedestrians.

[0063] In one embodiment, the re-identification branch module includes: a first pre-processing module, a first convolutional layer, a global average pooling layer and a post-processing module, the convolution kernel size of the first convolutional layer is 128; the re-identification branch module is obtained through actual sample training;

[0064] The specific process of step s3 includes:

[0065] The first preprocessing module cuts out the target feature map P of each pedestrian from the backbone network feature map f corresponding to each frame of the image according to the final target detection bounding box of each pedestrian. F，j , where F represents the frame number and j represents the pedestrian number in this frame;

[0066] The first convolutional layer generates a target feature map P of each pedestrian. F，j Perform feature extraction again and obtain the target feature map P′ with 128 channels F，j The average pooling layer transforms the target feature map P′ with a channel number of 128 F，jAll pixels in the target feature map of each channel are added and averaged to obtain the spatial embedding feature E of each pedestrian with a length of 128. F，j , E F，j ∈R 128 , where E F，j Represents the spatial embedding feature corresponding to the jth pedestrian in this frame image;

[0067] The post-processing module embeds the spatial features E of each pedestrian F，j and the spatial embedding features E of each pedestrian in the previous frame F-1，j Compare them one by one, select the one with the smallest metric distance as the matching target, and then convert the target feature map P of each pedestrian into F，j Store the action feature queue Q corresponding to the pedestrian ID in the matching target P，id , where E indicates that the stored data is the spatial embedding features of each pedestrian, and P indicates that the stored data is the regional image in the final target detection bounding box of each pedestrian.

[0068] In one embodiment, the action classification module includes: a second preprocessing module, a second convolutional layer group and a plurality of fully connected layers; the action classification module is obtained through actual sample training;

[0069] The specific process of step s4 includes:

[0070] The second preprocessing module converts the action feature queue Q of length K into P，id The size is uniformly scaled to 288*288, and it is aggregated in the channel dimension to extract the information of each pedestrian in the time dimension; then it passes through the second convolution layer group and multiple fully connected layers to output the feature vector A, A∈[0,1] num_action , that is, the prediction vector for the category to which each pedestrian belongs, which is also the final action retrieval result, where num_action represents the number of action types in the data set.

[0071] As a practical application of the above-mentioned method for retrieving a specified pedestrian action based on a pedestrian re-identification algorithm, the following describes an embodiment of the present invention for an application scenario of analyzing a single pedestrian action sequence. Figure 2 As shown in FIG. 1 , the steps of specifying a single pedestrian for action retrieval include:

[0072] After extracting the frame-level backbone network feature map of the determined query image, it is input into the trained pedestrian re-identification branch module. After extracting features through a convolution layer with a convolution kernel size of 128, and then through a global average pooling layer, the query embedding feature E is obtained. Query，i ∈R 128 , where i∈N q , represents multiple query images of the same pedestrian.

[0073] Specifically, the pedestrian detection branch module is used to detect all pedestrians in the video to be queried, extract the spatial embedding features of all pedestrians, and provide a candidate query library so that users can manually update the query spatial embedding features.

[0074] For each frame in the input video, the feature extraction backbone network is used to extract the frame-level backbone network feature map f, and then the pedestrian detection branch module is used to output the bounding box of each pedestrian. And according to the bounding box of each pedestrian, the corresponding area of ​​the pedestrian is cropped out from the original image, which is represented as I F，j , F represents the frame number, and j represents the pedestrian label in this frame.

[0075] According to the bounding box of each target, the target feature map is cropped from the backbone network feature map f, denoted as P F，j , F represents the frame number, j represents the target number in this frame; P F，j After further feature extraction through a convolution layer with a convolution kernel size of 128, and then through a global average pooling layer, the spatial embedding feature E of the target is obtained. F，j ∈R 128 .

[0076] Will I F，j 、E F，j Stored in the candidate query library, the storage format is (F, I F，j , E F，j ), generally only the latest 30 frames of data are stored, I F，j is a visible RGB image, which is used to directly show the user the image that may be the query target. F，j If the same query target is included and the user believes that adding this screenshot can make the query more effective, then I F，j The corresponding E F，j Join E Query The query embedding feature is used as a supplement in .

[0077] Embedding feature E according to the query Query And the spatial embedding features E of all pedestrians in this frame F，j Do a similarity comparison. If it is greater than the set threshold of 0.9, the candidate target and the query image are considered to belong to the same person and proceed to the next step; otherwise, the data corresponding to the candidate target is directly discarded.

[0078] Store the features of the new frame candidate target into the action recognition feature queue Q of length K P，id Perform feature transformation, and output the feature vector A∈[0,1] after passing through the classification network num_action That is, the prediction vector of the category to which the target belongs, and the category with the largest prediction value is taken as the action classification result, which is expressed as cls F .

[0079] The action classification result cls F and the target detection bounding box B F，id Combined into a single action retrieval result, and record the action detection results of the queried target in chronological order. Each input video frame should correspond to a retrieval record in the format of (F, cls F , B F，id ), after the whole process is completed, the search records can be merged, and the adjacent records with the same action classification results can be merged into one. The format of the merged record is (N last ,cls F , B F，id ), where N last Indicates the duration of the action in frames.

[0080] Above Figure 2 We will focus on specifying a single pedestrian for action retrieval as an example to illustrate the above Figure 2 Corresponding to the method shown in FIG. 1 , the embodiment of the present invention is introduced below by selecting some features in the query image to perform group action retrieval. Figure 3 As shown, the specific steps are as follows:

[0081] Train an attribute-based person re-identification model. The dataset uses Market-1501_Attribute, which annotates 27 attributes at the ID level, such as gender, bag, age, etc. Since only the annotation is performed at the ID level, it is only necessary to add 27 fully connected layers to the end of the original person re-identification model and perform binary classification to predict the attribute value. The model architecture diagram is shown below. Figure 4 As shown in the figure: The Market-1501_Attribute dataset contains 1501 targets, that is, the number of IDs is 1501. During training, the input image is first passed through the feature extraction backbone network to extract the frame-level backbone network feature map; then the pedestrian re-identification task is divided into two subtasks for processing. The first subtask is the pedestrian ID classification subtask. After the spatial embedding features are passed through the fully connected layer and the activation function, a pedestrian ID classification vector with a length of 1501 is obtained. The category with the largest predicted value is taken as the classification result to complete the pedestrian ID classification subtask. The second subtask is the pedestrian attribute classification subtask. For each of the 27 attributes, the spatial embedding features are passed through the fully connected layer and the activation function to obtain a pedestrian attribute classification vector with a length of 2. The two predicted values ​​in the vector represent the possibility that the pedestrian has this attribute and the possibility that the pedestrian does not have this attribute. The largest predicted value is taken as the classification result to complete the pedestrian attribute classification subtask.

[0082] Determine the query image and output its spatial embedding feature E through the pedestrian re-identification (Re-ID) model trained in the previous stepQuery，i ∈R 128 With attribute value P att Then select the required attributes from the prediction results. For example, the existing predicted attribute values ​​are: young, girl, short hair, black cloths, long-sleeve cloths, white pants, shortpants, but the group that the user needs to index only needs the attributes of girl, black cloths, and white pants. and query embedding feature E Query Together they form the target group's search basis.

[0083] Use the pedestrian detection branch module to detect all pedestrians in the query video and extract the spatial embedding features E of all pedestrians F，j With attribute value P F，j , and update the candidate query library and target group retrieval basis.

[0084] Use the dual retrieval criteria to screen candidate targets and obtain target groups that meet the criteria. First, embed the query into feature E Query And the spatial embedding features E of all pedestrians in this frame F，j Do a similarity comparison. If it is greater than the set threshold of 0.5, the candidate target is considered to basically belong to the target group and proceed to the next step; otherwise, the data corresponding to the candidate target is directly discarded. Next, determine the attribute value P of the candidate target. F，j Contains If the conditions are met, proceed to the next step; otherwise, the data corresponding to the candidate target is directly discarded.

[0085] The spatial embedding features of each target in the group are measured for similarity with the data at the end of each spatial embedding feature queue. If the similarity is greater than the set threshold of 0.9, it means that they are the same person and proceed to the next step; otherwise, it is considered that a new target has been added to the video and a new feature queue is opened.

[0086] Store the target feature map of the new frame candidate pedestrian into the action recognition feature queue Q of length K P，id Perform feature transformation, and output the feature vector A∈[0,1] after passing through the classification network num_action That is, the prediction vector of the category to which the target belongs, and the category with the largest prediction value is taken as the action classification result, which is expressed as cls F .

[0087] The action classification result cls of each person in the group F and the target detection bounding box B F，id Combine them for user statistical analysis.

[0088] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0090] The terms "including" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products, or equipment.

[0091] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A specified pedestrian action retrieval method based on a pedestrian re-identification algorithm, characterized in that: The following steps are involved: s1: input each frame of video data collected in real time by the video acquisition device into the feature extraction backbone network respectively, the feature extraction backbone network processes each frame and extracts the backbone network feature map; s2: inputting the backbone network feature map into the pedestrian detection branch module, the pedestrian detection branch module processes the backbone network feature map and outputs the final target detection bounding box of each pedestrian; s3: inputting the backbone network feature map and the final target detection bounding box of each pedestrian into a re-identification branch module, the re-identification branch module processes the backbone network feature map and the final target detection bounding box of each pedestrian, and outputs the action feature queue of each pedestrian; s4: The action classification module uniformly scales the action feature queues of each pedestrian to a size of 288*288, and aggregates them in the channel dimension to extract the information of each pedestrian in the time dimension, and then performs action classification to obtain the final action retrieval result; The pedestrian detection branch module includes: a bounding box center point prediction head subnetwork, a bounding box size prediction head subnetwork and a center point offset prediction head subnetwork; The bounding box center point prediction head subnetwork, the bounding box size prediction head subnetwork and the center point offset prediction head subnetwork are respectively obtained through actual sample training; The specific process of step s2 includes: The backbone network feature map f is input into the bounding box center point prediction head sub-network, and the bounding box center point prediction head sub-network predicts the backbone network feature map f and outputs the thermal map of each pedestrian Heat map of each pedestrian Use the loss function focal loss: Among them, x and y represent the output heat maps of each pedestrian respectively. The horizontal and vertical coordinates of each element in , α and β represent the hyperparameters that control the contribution weight of the central point, represents the probability that a pedestrian target exists with the coordinate (x, y) as the center point, L x,y Indicates the true probability that a pedestrian target exists with the coordinate (x, y) as the center point; The backbone network feature map f is input into the bounding box size prediction head sub-network, and the bounding box size prediction head sub-network predicts the backbone network feature map f and outputs the bounding box size of each pedestrian. The bounding box size of each pedestrian Use the minimum absolute deviation loss function l1: Among them, i∈[1,N] represents the index of the pedestrian, s i represents the true value of the i-th pedestrian bounding box size, represents the predicted value of the i-th pedestrian bounding box size, N represents the number of pedestrians in the current frame, and l size That is, the minimum absolute deviation loss function l1, size represents the prediction used here to constrain the size of the bounding box; The backbone network feature map f is input into the center point offset prediction head sub-network, and the center point offset prediction head sub-network predicts the backbone network feature map f and outputs the offset of the center point of each pedestrian's bounding box in the two dimensions of length and width. The offset of the center point of the bounding box of each pedestrian in the length and width dimensions Use the minimum absolute deviation loss function l1: l off That is, the minimum absolute value deviation loss function l1, off represents the network to which it belongs, o i Represents the actual quantized offset of the i-th pedestrian; Represents the quantized offset of the predicted i-th pedestrian; The heat map corresponding to each pedestrian Bounding box size The offset from the center of the bounding box in both length and width dimensions The candidate target detection bounding boxes of the pedestrians are combined, and then the NMS algorithm is used to deduplicate the candidate target detection bounding boxes of the pedestrians and filter out the bounding boxes with confidence lower than a threshold of 0.8, so as to obtain the final target detection bounding boxes of the pedestrians.

2. The method for retrieving a designated pedestrian action based on a pedestrian re-identification algorithm according to claim 1, characterized in that: The specific process of step s1 includes: Each frame of the video data collected by the video acquisition device is sequentially input into the feature extraction backbone network, and the feature extraction backbone network extracts and obtains the backbone network feature map corresponding to each frame image, which is denoted as f. Among them, R represents the real space, W represents the width of the backbone network feature map f, H represents the height of the backbone network feature map f, D represents the spatial downsampling rate, and B represents the number of channels of the backbone network feature map f.

3. The method for retrieving a designated pedestrian action based on a pedestrian re-identification algorithm according to claim 2, characterized in that: The re-identification branch module includes: a first pre-processing module, a first convolutional layer, a global average pooling layer and a post-processing module, the convolution kernel size of the first convolutional layer is 128; the re-identification branch module is obtained through actual sample training; The specific process of step s3 includes: The first preprocessing module cuts out the target feature map P of each pedestrian from the backbone network feature map f corresponding to each frame of the image according to the final target detection bounding box of each pedestrian. F,j , where F represents the frame number and j represents the pedestrian number in this frame; The first convolutional layer generates a target feature map P of each pedestrian. F,j Perform feature extraction again and obtain the target feature map P' with 128 channels F,j The average pooling layer transforms the target feature map P' with a channel number of 128 F,j All pixels in the target feature map of each channel are added and averaged to obtain the spatial embedding feature E of each pedestrian with a length of 128. F,j , E F,j ∈R 128 , where e F,j Represents the spatial embedding feature corresponding to the jth pedestrian in this frame image; The post-processing module embeds the spatial features E of each pedestrian F,j and the spatial embedding features E of each pedestrian in the previous frame F-1,j Compare them one by one, select the one with the smallest metric distance as the matching target, and then convert the target feature map P of each pedestrian into F,j Store the action feature queue Q corresponding to the pedestrian ID in the matching target P,id , where E indicates that the stored data is the spatial embedding features of each pedestrian, and P indicates that the stored data is the regional image in the final target detection bounding box of each pedestrian.

4. The method for retrieving a designated pedestrian action based on a pedestrian re-identification algorithm according to claim 3, characterized in that: The action classification module includes: a second preprocessing module, a second convolutional layer group and a plurality of fully connected layers; the action classification module is obtained through actual sample training; The specific process of step s4 includes: The second preprocessing module converts the action feature queue Q of length K into P,id The size is uniformly scaled to 288*288, and it is aggregated in the channel dimension to extract the information of each pedestrian in the time dimension; then it passes through the second convolution layer group and multiple fully connected layers to output the feature vector A, A∈[0,1] num_action , that is, the prediction vector for the category to which each pedestrian belongs, which is also the final action retrieval result, where num_action represents the number of action types in the data set.

Images