Pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding

Abstract

The present application relates to a pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding, comprising the following steps: obtaining a pedestrian search dataset, inputting a backbone network and a feature pyramid structure to obtain a feature map for subsequent modules; inputting the features into a region generation network to obtain a preliminary proposal box, extracting a region of interest from the previously obtained feature map according to the proposal box and generating a local proposal feature; inputting the local proposal feature into a multiple region convolutional neural network module to obtain a more accurate pedestrian proposal box and a pedestrian discrimination score, setting a threshold, taking out the pedestrian proposal box with a low score, and using the remaining pedestrian proposal box to extract pedestrian features in the image feature map; inputting the pedestrian features into a pedestrian re-identification module to calculate a pedestrian recognition loss; iteratively training according to specified training parameters, and finally using the pedestrian information and image scene context information to comprehensively judge the pedestrian search result; the present application can accurately and effectively perform pedestrian search.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a pedestrian search network of a pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding. Background Art

[0002] Pedestrian search technology, also known as pedestrian detection and recognition, is a key research area in computer vision. Its development has benefited from advances in computer performance, the rise of deep learning algorithms, and the widespread availability of camera hardware. The primary goal of this technology is to automatically detect pedestrians in images or videos and further identify their identities, behaviors, and other key information.

[0003] The pedestrian search task faces many challenges, some of the main ones include: 1. Viewpoint variation and occlusion: Pedestrians may appear in images or videos with different poses, angles, and degrees of occlusion, which makes detection and recognition more complicated. 2. Illumination variation: Different lighting conditions can cause changes in the appearance of pedestrians, thereby affecting the accuracy of pedestrian detection and recognition. 3. Background interference: Pedestrians often appear in complex backgrounds, and background interference may lead to misdetection or misidentification, especially in crowded or complex scenes. 4. Scale variation: The size of pedestrians may vary greatly in the image, which requires the algorithm to be able to effectively handle pedestrians of different scales. 5. Pose diversity: Pedestrians may adopt various poses, such as walking, running, sitting, etc., which increases the difficulty of Re-ID because different poses of the same person may result in different appearances. 6. Occlusion and partial visibility: Pedestrians may be occluded by other objects or objects and only partially visible, which makes pedestrian detection and recognition more challenging.

[0004] Leveraging the power of deep neural networks, we have achieved widespread success in pedestrian search tasks. Pedestrian search technology has broad applications in various fields, including traffic monitoring, intelligent driving, security surveillance, and crowd flow analysis. By monitoring and identifying pedestrians in real time, this technology helps improve urban management, enhance public safety, and optimize urban planning. Summary of the Invention

[0005] The purpose of the present invention is to propose a pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding, which can accurately and effectively perform pedestrian search.

[0006] To achieve the above object, the technical solution of the present invention is: a pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding, which specifically includes the following steps:

[0007] Step S1: Obtain a pedestrian search dataset, input it into the backbone network, and then further enhance the feature representation by combining multi-scale information with a feature pyramid structure adapted to the backbone network. After completion, obtain the feature map that is input into subsequent modules.

[0008] Step S2: Input the feature map into the region generation network to obtain a preliminary proposal box. According to the proposal box, the region of interest is extracted from the previously obtained feature map and a local proposal feature is generated. Here, the discrimination loss and the proposal box regression loss need to be calculated.

[0009] Step S3: Input the local proposal features into the multiple regional convolutional neural network module to obtain more accurate pedestrian proposal boxes and pedestrian discrimination scores. Here, two regional convolutional neural network modules are used in total, and the proposal box regression loss and discrimination loss are calculated twice. According to the pedestrian discrimination score, a threshold is set and the pedestrian proposal boxes with low scores are removed. The remaining pedestrian proposal boxes are used to extract pedestrian features in the image feature map.

[0010] Step S4: Input the pedestrian features into the pedestrian re-identification module to calculate the pedestrian recognition loss. Construct a pedestrian category hierarchy tree based on the cosine similarity calculated between the pedestrian categories in the dataset and the pedestrian re-identification features output by the pedestrian re-identification module. Then, input the image features output by the backbone network into the scene encoding module to obtain scene features. The pedestrian features with known IDs are used as query pedestrian features. The query pedestrian features are combined with the scene features to obtain scene features. Then, the positive and negative pairs of pedestrian scene features are sampled through the pedestrian category hierarchy tree to calculate the pedestrian scene structure modeling loss.

[0011] Step S5: Perform iterative training according to the specified training parameters, update the model parameters by optimizing the combined loss, and continuously save the optimal model based on the verification accuracy. The final model uses pedestrian information and image scene context information to comprehensively judge and obtain the pedestrian search results.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] Step S11: Using a public pedestrian search dataset, perform data preprocessing to complete label extraction and convert it into a usable format;

[0014] Step S12: Input the entire scene image and first pass it through a backbone network to obtain the basic feature map. It contains L stages. The intermediate feature map output by the feature extractor is represented as Where size_H s_l ,size_W s_l ,size_C s_lThey represent the height, width, and number of channels of the output feature map of the s_l stage of the backbone network, respectively, where s_l∈[1,L], the feature map of the last stage is used as the input of the feature pyramid, and the feature maps of other stages are input into the feature pyramid in a horizontal connection manner.

[0015] Step S13: Then the activation map is input into a feature pyramid structure adapted to the backbone network. The feature pyramid is also composed of a convolutional neural network. Each layer has a horizontal connection, a bilinear interpolation upsampling operation, and a 1x1 convolution layer to ensure that the size of each layer corresponds to the size of the feature map in the backbone network. The number of layers included is the same as the number of stages of the backbone network, with a total of L layers. The intermediate feature map output by each layer is Where size_H f_l , size_W f_l They represent the height and width of the feature map output by the f_l layer of the feature pyramid. The number of channels of all feature maps is unified to 256. Among them, f_l∈[1,L], for each stage s_l in the backbone network, f_l=L+1-s_l, and for the feature size of each layer of the feature pyramid output, size_H f_l =size_H L+1-s_l , size-W f_l =size_W L+1-s_l ;

[0016] First, FM s_l Horizontal connection LC in the f_l stage of the input feature pyramid f_l The horizontal feature representation LF generated in the f_l stage is obtained f-l , LF f_l =LC f_l (FM s_l ), where LC f-l (·) contains a 1×1 convolutional layer and a BatchNorm layer. When f_l=1, the input is FM s_l At this time, the intermediate feature map AM of the feature pyramid of this layer is obtained through a 1x1 convolution layer transformation channel f_l When f_l≠1, the input is AM f_l-1 , at this time it is necessary to pass through an upsampling layer UP f_l Get the intermediate feature map AM generated by the f_l stage of the feature pyramid f_l =UP f_l (LF f_l +AM f_l-1 ), where the upsampling layer UP f_l Contains bilinear interpolation upsampling operations and 1x1 convolutional layers.

[0017] The output AM of the last layer of the feature pyramid module L As the next input.

[0018] Furthermore, step S2 specifically includes the following steps:

[0019] Step S21: Construct a region generation network based on the activation map AM obtained in step S13 L As input, for the activation map AM L For each feature pixel in the image, k proposal boxes of different sizes are selected. Because the input region generation network has a 256-channel feature map, a 3×3×256×256 convolution kernel is used to integrate the information around each pixel and generate a new 256-dimensional vector of the same size. Overall, the entire feature map is directly passed through a 3×3 convolutional layer IL, which outputs a new feature map of the same size as the original.

[0020] Step S22: Each 256-dimensional vector is followed by two branches, one for the target and the other for the background. 2k ​​scores are obtained through a 1×1×256×2k judgment convolution kernel, where k is equal to the number of proposal boxes, indicating the score of the k proposal boxes being the background and the score of the target. Overall, this is done through a 1x1 convolution layer CLS. rpn . Get the corresponding category score

[0021] SCORE rpn_cls =CLS rpn (IL(AM L )),in

[0022] If the proposed box is the target area, we will determine the location of the proposed box of the target area. At this time, the other branch will obtain 4k coordinates through the convolution kernel of 1×1×256×4k. Each box contains 4 area coordinates, (x rpn ,y rpn , w rpn , h rpn ), where x rpn ,y rpn is the coordinate of the upper left corner of the proposal box, w rpn , h rpn is the height and width, which is the specific position where the boxes corresponding to the k candidate regions should be offset. Overall, it is done through a 1x1 convolution layer REG rpn . Get the corresponding regression value

[0023] SCORE rpn_reg＝ REG rpn (IL(AML )),in

[0024] Step S23: A series of rectangular proposal box lists {RP1, RP2, ..., RP win_K}, where win_K is the number of windows, win_K=size_H L ×size_W L ×k, each candidate proposal box contains four proposal box offset values ​​and two target judgment scores; some proposal boxes are discarded through the non-maximum suppression algorithm NMS.

[0025] Then, we take out the corresponding marked box list from the dataset by calculating the IOU (overlap) between the proposed box and each real marked box in the dataset. Then the loss is calculated for each box left after NMS screening and training is performed.

[0026] Step S24: Introducing the discriminant loss Loss rpn_cls , and regression loss Loss rpn_reg , where Loss rpn_cls is the cross entropy loss with softmax. First, take out the predicted score of the corresponding category of the proposal box, and then scale it to [0, 1] through softmax, denoted as p rpn Then take out the true value of the marked box category and record it as y rpn

[0027] Loss rpn_cls =L(p rpn ,y rpn )=-y rpn log(p rpn )-(1-y rpn )log(1-p rpn )

[0028] Loss rpn_reg is the smooth L1 loss. x' rpn It is the absolute value of the difference between the regression value of the proposed box and the labeled box.

[0029]

[0030] Finally, the total loss can be expressed as

[0031] Loss rpn =Loss rpn_cls +Loss rpn_reg

[0032] Use the area coordinates of each proposal box after filtering from the original feature map AM LGet the local proposal feature and adjust it to 14×14×256 size as the next input local feature map PM p_l , p_l∈[1,m], m is the number of proposal boxes after nms screening.

[0033] Furthermore, step S3 specifically includes the following steps:

[0034] Step S31: Pool the local feature map, obtain the pooled features, and then send them to a multi-layer perceptron consisting of two fully connected layers. The layers are activated using the ReLU function to further extract features. After the features are extracted, the dimension remains unchanged. Then enter the prediction stage. The multi-layer perceptron here is denoted as MLP2, and the pooling layer is denoted as POOL.

[0035] Step S32: In the prediction stage, it is responsible for predicting the local feature map. Two parallel fully connected layers are introduced, one fully connected layer is responsible for predicting the score of the category of the proposal box corresponding to the local feature map. The fully connected layer is denoted as MLP cls The corresponding formula is SCORE det_cls =MLP cls (MLP2(POOL(PM p_l ))), another fully connected layer is responsible for predicting the coordinate regression value of the proposal box corresponding to the local feature map, and the fully connected layer is recorded as MLP reg , the corresponding formula is SCORE det_reg =MLP reg (MLP2(POOL(PM p_l )))

[0036] According to the score matrix, a series of proposal box lists {R'P1, R'P2, ..., R'P anchor_k}, where anchor_K is the number of windows, and each candidate proposal box contains four proposal box offset values ​​and two target judgment scores. A part of the proposal box is discarded by the non-maximum suppression algorithm (NMS), and then the corresponding marker box list is retrieved from the dataset by calculating the IOU (overlap) between the proposal box and each real marker box in the dataset. Then the loss is calculated for each box left after NMS screening and training is performed.

[0037] Step S33: Introducing the discriminant loss Loss det_cls , and regression loss Loss det_reg , where Loss det_cls is the focal loss with softmax. First, take out the predicted score of the corresponding category of the proposal box, and then scale it to [0, 1] through softmax, denoted as p det Then take out the true value of the marked box category and record it as y det,γ is a hyperparameter and needs to be set manually.

[0038]

[0039] Loss det_reg is the focal L1 loss. det It is the absolute value of the difference between the proposed box regression value and the marked box true value, where β is the hyperparameter that needs to be set, α and C are intermediate hyperparameters calculated by β, and α = eβ., C = (2αlnβ+α) / 4,

[0040]

[0041] Finally, the total loss can be expressed as

[0042] Loss det =Loss det_cls +Loss det_reg

[0043] The area coordinates of each proposal box after filtering are used to get the original feature map AM again L The part is marked out as the local pedestrian feature and resized to 14×14×256 as the new input, and then step S3 is repeated again to obtain a more accurate proposal frame. Then the proposal frame obtained is obtained from the original feature map AM L The part is marked as the pedestrian feature and adjusted to 14×14×256 size. Finally, it is transformed into a 2048-dimensional pedestrian feature vector through a global average pooling (CM). c_l , _c_l∈[1,n], n is the number of proposed boxes after two rounds of screening and detection. For each pedestrian feature vector, its true category can be obtained from the dataset according to the corresponding marked box n class is the total number of pedestrian label categories in the dataset, when When , it means that the pedestrian has no label in the dataset.

[0044] Furthermore, step S4 specifically includes the following steps:

[0045] Step S41: After obtaining the feature vector of each pedestrian in the image, define a lookup table LUT to store the prototype vector of each marked pedestrian class, with a length of n class , the prototype vector stored in it is defined as v i ,i∈[1,n class ]. Define a circular queue CQ to store unmarked pedestrian vectors with a length of n queue, is a manually set hyperparameter, and the unlabeled pedestrian vector stored in it is called u i’ ,i'∈[1,n queue ]. In each iteration of training the network:

[0046] First obtain the discriminant score s of each feature vector, and then find the prototype vector v corresponding to its true category in the LUT t And update:

[0047]

[0048] Among them, γ is a hyperparameter and needs to be set manually.

[0049] If the feature vector has no category label, it is directly stored in the circular queue CQ. If the circular queue CQ is full, the earliest unlabeled feature vector is taken out.

[0050] Step S42: Based on the two special data structure types of LUT and CQ mentioned above, a feature vector CM is defined by the softmax function c_l The probability of being considered as a pedestrian of type i is:

[0051]

[0052] The prototype vector corresponding to the real category in LUT is transposed as follows

[0053] The prototype vector corresponding to the j1th category in the LUT is transposed as follows:

[0054] The prototype vector of CQ corresponding to the j2th unknown pedestrian is transposed as follows: τ is the temperature parameter that controls the similarity space and needs to be set manually;

[0055] The probability of being considered as an unmarked pedestrian is:

[0056]

[0057] The prototype vector of CQ corresponding to the i'th unknown pedestrian is transposed to

[0058] The prototype vector corresponding to the j1th category in the LUT is transposed as follows:

[0059] The prototype vector of CQ corresponding to the j2th unknown pedestrian is transposed as follows:

[0060] The final loss function is Loss reid =Ex [log p t ], where p t is the probability of the true category corresponding to the feature vector, where Ex[] represents the cross entropy loss maximization p t , p t It is expressed as the probability that a feature vector is considered as a pedestrian of the tth category, where t is the true category of the feature vector in the dataset.

[0061] Step S43: First, calculate the similarity matrix of each class in the entire training set; the cosine similarity between the p-th class and the q-th class is calculated as: The prototype vector corresponding to the p-th category in the LUT is v p , the prototype vector corresponding to the qth category in LUT is v q , then obtain the similarity matrix between all classes and calculate its average value as d0, where each class represents a leaf node in layer 0; then, based on the calculated similarity matrix, create a hierarchical structure by recursively merging different levels;

[0062] Set the hierarchical tree to L tree Level, and use the average similarity as the threshold for merging level 0 nodes. Two classes with a distance less than d0 are merged into nodes of the lth layer. Repeat the same operation for nodes of the lth layer to obtain nodes of the second layer, and so on. Nodes are from the 0th layer to the Lth layer. tree The layers are merged, and finally, a hierarchical tree H is generated, which starts from the leaf node of the original image class to the final top node.

[0063] Distance less than d L’-1 The two classes are merged into nodes of the L'th layer. In the L'th layer of the hierarchical tree (the number of layers except the 1st layer), the merging threshold for merging the L'-1th layer is set to d L’-1 =L'(4-d L’-2 ) / L'+d L’-2 .

[0064] Step S44: For each pedestrian feature vector CM c_l , take out the corresponding scene feature map AM L , through the full connection layer to a 2048-dimensional vector and normalized to obtain a 2048-dimensional scene feature vector GM L ,The pedestrian scene feature vector CGM is obtained by directly connecting the scene feature vector and the pedestrian feature vector c_l In the early stage of training, pedestrian scene vectors in this batch are randomly sampled through the high-level class relationship of the hierarchical tree and positive and negative pairs are constructed. In the late stage of training, pedestrian scene vectors in this batch are randomly sampled through the low-level class relationship of the hierarchical tree and positive and negative pairs are constructed. The random sampling of the hierarchical tree is reduced by one level each time a batch is passed.

[0065] Define a total of N samples sample A pair and M sample negative pairs, and then calculate the pedestrian scene structure modeling loss:

[0066]

[0067] For any positive pair sampled, the corresponding category index is represented as a, b,

[0068] represents the set of all categories contained in all pairs of categories a or b. CGM represents the pedestrian scene feature vector.

[0069] The total loss is the sum of all the pedestrian structure losses calculated for the scene:

[0070] Loss gallery =∑l a，b ;

[0071] Furthermore, step S5 specifically includes the following steps:

[0072] Step S51: Calculate the overall network loss function Loss, the calculation formula is as follows

[0073] Loss=Loss rpn +Loss det +Loss reid +Loss gallery

[0074] The overall optimization goal consists of four parts: Loss rpn is the proposal box detection loss, Loss det Loss for pedestrian detection reid Loss is the pedestrian re-identification loss. gallery It is the pedestrian scene structure loss.

[0075] Step S52: Perform iterative training according to the specified parameters, and continuously update the gradient according to the overall loss Loss calculated in step S51 to perform iterative training;

[0076] Step S53: During the inference process, by combining the scene information predictions and making full use of the complementarity of information, the pedestrian search box and pedestrian category information are finally obtained.

[0077] Step S54: During the training process, the model is verified at a certain iteration interval according to the verification interval flag, and the optimal model is continuously saved. When the number of iterations reaches the preset maximum iteration threshold, the training process ends and the optimal fine-grained prediction accuracy of the currently specified data set is returned.

[0078] Compared with the prior art, the present application has the following beneficial effects:

[0079] 1. The present application constructs a feature pyramid module used in combination with a pedestrian search network, which can well improve the ability of multi-granularity feature information extraction of the network, and through the use of multi-granularity feature information, the target detection performance and the pedestrian re-identification performance can be improved, thereby improving the overall performance of the pedestrian search.

[0080] 2. For the target detection part of the overall network, the present application uses a focal loss method different from the traditional loss to optimize the performance of target detection, and alleviates the problem of imbalance between positive and negative samples in the data set.

[0081] 3. By using the pedestrian prototype features obtained by the pedestrian search network to construct a hierarchical tree, the positive and negative samples in the pedestrian features are resampled through the hierarchical tree, which better helps the network learn the structure of the entire feature space to achieve better loss.

[0082] 4. A scene feature embedding module is constructed, which combines pedestrian features with scene features, and then uses the positive and negative feature sampling results of the hierarchical tree to train the structure loss, so that the network can reorder the pedestrian features in the prediction stage, which can improve the network's processing of similar pedestrian images and improve the network's performance on occluded and deformed pedestrian images. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 The present application is a schematic diagram of the principle. DETAILED DESCRIPTION

[0084] The technical solutions of the present application will be specifically described below in combination with the drawings.

[0085] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0086] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they mean the presence of a feature, step, operation, device, component and / or combination thereof.

[0087] As Figure 1As shown, the present invention proposes a pedestrian search network based on similarity hierarchical tree reordering and scene feature embedding. Step S1: obtain the pedestrian search dataset, input it into the backbone network, and then further enhance the feature representation by combining the feature pyramid structure adapted to the backbone network with multi-scale information. After completion, obtain the feature map input into the subsequent module.

[0088] Step S2: Input the feature map into the region generation network to obtain a preliminary proposal box. According to the proposal box, the region of interest is extracted from the previously obtained feature map and a local proposal feature is generated. Here, the discrimination loss and the proposal box regression loss need to be calculated.

[0089] Step S3: Input the local proposal features into the multiple regional convolutional neural network module to obtain more accurate pedestrian proposal boxes and pedestrian discrimination scores. Here, two regional convolutional neural network modules are used in total, and the proposal box regression loss and discrimination loss are calculated twice. According to the pedestrian discrimination score, a threshold is set and the pedestrian proposal boxes with low scores are removed. The remaining pedestrian proposal boxes are used to extract pedestrian features in the image feature map.

[0090] Step S4: Input the pedestrian features into the pedestrian re-identification module to calculate the pedestrian recognition loss. Construct a pedestrian category hierarchy tree based on the cosine similarity calculated between the pedestrian categories in the dataset and the pedestrian re-identification features output by the pedestrian re-identification module. Then, input the image features output by the backbone network into the scene encoding module to obtain scene features. The pedestrian features with known IDs are used as query pedestrian features. The query pedestrian features are combined with the scene features to obtain scene features. Then, the positive and negative pairs of pedestrian scene features are sampled through the pedestrian category hierarchy tree to calculate the pedestrian scene structure modeling loss.

[0091] Step S5: Perform iterative training according to the specified training parameters, update the model parameters by optimizing the combined loss, and continuously save the optimal model based on the verification accuracy. The final model uses pedestrian information and image scene context information to comprehensively judge and obtain the pedestrian search results.

[0092] Furthermore, step S1 specifically includes the following steps:

[0093] Step S11: Using a public pedestrian search dataset, perform data preprocessing to complete label extraction and convert it into a usable format;

[0094] Step S12: Input the entire scene image and first pass it through a backbone network to obtain the basic feature map. It contains L stages. The intermediate feature map output by the feature extractor is represented as Where size_H s_l ,size_W s_l ,size_C s_lThey represent the height, width, and number of channels of the output feature map of the s_l stage of the backbone network, respectively, where s_l∈[1,L], the feature map of the last stage is used as the input of the feature pyramid, and the feature maps of other stages are input into the feature pyramid in a horizontal connection manner.

[0095] Step S13: Then the activation map is input into a feature pyramid structure adapted to the backbone network. The feature pyramid is also composed of a convolutional neural network. Each layer has a horizontal connection, a bilinear interpolation upsampling operation, and a 1x1 convolution layer to ensure that the size of each layer corresponds to the size of the feature map in the backbone network. The number of layers included is the same as the number of stages of the backbone network, with a total of L layers. The intermediate feature map output by each layer is Where size_H f_l , size_W f_l They represent the height and width of the feature map output by the f_l layer of the feature pyramid. The number of channels of all feature maps is unified to 256. Among them, f_l∈[1,L], for each stage s_l in the backbone network, f_l=L+1-s_l, and for the feature size of each layer of the feature pyramid output,

[0096] First, FM s_l Horizontal connection LC in the f_l stage of the input feature pyramid f_l The horizontal feature representation LF generated in the f_l stage is obtained f_l , LF f_l =LC f_l (FM s_l ), where LC f_l (·) contains a 1×1 convolutional layer and a BatchNorm layer. When f_l=1, the input is FM s_l At this time, the intermediate feature map AM of the feature pyramid of this layer is obtained through a 1x1 convolution layer transformation channel f_l When f_l≠1, the input is AM f_l-1 , at this time it is necessary to pass through an upsampling layer UP f_l Get the intermediate feature map AM generated by the f_l stage of the feature pyramid f_l =UP f_l (LF f_l +AM f_l-1 ), where the upsampling layer UP f_l Contains bilinear interpolation upsampling operations and 1x1 convolutional layers.

[0097] The output AM of the last layer of the feature pyramid module L As the next input.

[0098] Furthermore, step S2 specifically includes the following steps:

[0099] Step S21: Construct a region generation network based on the activation map AM obtained in step S13 L As input, for the activation map AM L For each feature pixel in the image, k proposal boxes of different sizes are selected. Because the input region generation network has a 256-channel feature map, a 3×3×256×256 convolution kernel is used to integrate the information around each pixel and generate a new 256-dimensional vector of the same size. Overall, the entire feature map is directly passed through a 3×3 convolutional layer IL, which outputs a new feature map of the same size as the original.

[0100] Step S22: Each 256-dimensional vector is followed by two branches, one for the target and the other for the background. 2k ​​scores are obtained through a 1×1×256×2k judgment convolution kernel, where k is equal to the number of proposal boxes, indicating the score of the k proposal boxes being the background and the score of the target. Overall, this is done through a 1x1 convolution layer CLS. rpn . Get the corresponding category score

[0101] SCORE rpn_cls =CLS rpn (IL(AM L )), beside

[0102] If the proposed box is the target area, we will determine the location of the proposed box of the target area. At this time, the other branch will obtain 4k coordinates through the convolution kernel of 1×1×256×4k. Each box contains 4 area coordinates, (x rpn ,y rpn , W rpn , h rpn ), where x rpn ,y rpn is the coordinate of the upper left corner of the proposal box, W rpn , h rpn is the height and width, which is the specific position where the boxes corresponding to the k candidate regions should be offset. Overall, it is done through a 1x1 convolution layer REG rpn . Get the corresponding regression value

[0103] SCORE rpn_reg =REG rpn (IL(AM L )),in

[0104] Step S23: A series of rectangular proposal box lists {RP1, RP2, ..., RP win_K}, where win_K is the number of windows, win_K=size_H L ×size_W L ×k, each candidate proposal box contains four proposal box offset values ​​and two target judgment scores; some proposal boxes are discarded through the non-maximum suppression algorithm NMS.

[0105] Then, we take out the corresponding marked box list from the dataset by calculating the IOU (overlap) between the proposed box and each real marked box in the dataset. Then the loss is calculated for each box left after NMS screening and training is performed.

[0106] Step S24: Introducing the discriminant loss Loss rpn_cls , and regression loss Loss rpn_reg , where Loss rpn_cls is the cross entropy loss with softmax. First, take out the predicted score of the corresponding category of the proposal box, and then scale it to [0, 1] through softmax, denoted as p rpn Then take out the true value of the marked box category and record it as y rpn

[0107] Loss rpn_cls =L(p rpn ,y rpn )=-y rpn log(p rpn )-(1-y rpn )log(1-p rpn )

[0108] Loss rpn_reg is the smooth L1 loss. x' rpn It is the absolute value of the difference between the regression value of the proposed box and the labeled box.

[0109]

[0110] Finally, the total loss can be expressed as

[0111] Loss rpn =Loss rpn_cls +Loss rpn_reg

[0112] Use the area coordinates of each proposal box after filtering from the original feature map AM L Get the local proposal feature and adjust it to 14×14×256 size as the next input local feature map PM p_l, p_l∈[1,m], m is the number of proposal boxes after nms screening.

[0113] Furthermore, step S3 specifically includes the following steps:

[0114] Step S31: Pool the local feature map, obtain the pooled features, and then send them to a multi-layer perceptron consisting of two fully connected layers. The layers are activated using the ReLU function to further extract features. After the features are extracted, the dimension remains unchanged. Then enter the prediction stage. The multi-layer perceptron here is denoted as MLP2, and the pooling layer is denoted as POOL.

[0115] Step S32: In the prediction stage, it is responsible for predicting the local feature map. Two parallel fully connected layers are introduced, one fully connected layer is responsible for predicting the score of the category of the proposal box corresponding to the local feature map. The fully connected layer is denoted as MLP cls The corresponding formula is SCORE det_cls =MLP cls (MLP2(POOL(PM p_l ))), another fully connected layer is responsible for predicting the coordinate regression value of the proposal box corresponding to the local feature map, and the fully connected layer is recorded as MLP reg , the corresponding formula is SCORE det_reg =MLP reg (MLP2(POOL(PM p_l )))

[0116] According to the score matrix, a series of proposal box lists {R'P1, R'P2, ..., R'P anchor_k}, where anchor_K is the number of windows, and each candidate proposal box contains four proposal box offset values ​​and two target judgment scores. A part of the proposal box is discarded by the non-maximum suppression algorithm (NMS), and then the corresponding marker box list is retrieved from the dataset by calculating the IOU (overlap) between the proposal box and each real marker box in the dataset. Then the loss is calculated for each box left after NMS screening and training is performed.

[0117] Step S33: Introducing the discriminant loss Loss det_cls , and regression loss Loss det_reg , where Loss det_cls is the focal loss with softmax. First, take out the predicted score of the corresponding category of the proposal box, and then scale it to [0, 1] through softmax, denoted as p det Then take out the true value of the marked box category and record it as y det ,γ is a hyperparameter and needs to be set manually.

[0118]

[0119] Loss det_reg is the focal L1 loss. det It is the absolute value of the difference between the proposed box regression value and the marked box true value, where β is the hyperparameter that needs to be set, α and C are intermediate hyperparameters calculated by β, and α = eβ., C = (2αlnβ+α) / 4,

[0120]

[0121] Finally, the total loss can be expressed as

[0122] Loss det =Loss det_cls +Loss det_reg

[0123] The area coordinates of each proposal box after filtering are used to get the original feature map AM again L The part is marked out as the local pedestrian feature and resized to 14×14×256 as the new input, and then step S3 is repeated again to obtain a more accurate proposal frame. Then the proposal frame obtained is obtained from the original feature map AM L The part is marked as the pedestrian feature and adjusted to 14×14×256 size. Finally, it is transformed into a 2048-dimensional pedestrian feature vector through a global average pooling (CM). c_l , _c_l∈[1,n], n is the number of proposed boxes after two rounds of screening and detection. For each pedestrian feature vector, its true category can be obtained from the dataset according to the corresponding marked box n class is the total number of pedestrian label categories in the dataset, when When , it means that the pedestrian has no label in the dataset.

[0124] Furthermore, step S4 specifically includes the following steps:

[0125] Step S41: After obtaining the feature vector of each pedestrian in the image, define a lookup table LUT to store the prototype vector of each marked pedestrian class, with a length of n class , the prototype vector stored in it is defined as v i ,i∈[1,n class ]. Define a circular queue CQ to store unmarked pedestrian vectors with a length of n queue , is a manually set hyperparameter, and the unlabeled pedestrian vector stored in it is called u i’ ,i'∈[1,n queue]. In each iteration of training the network:

[0126] First obtain the discriminant score s of each feature vector, and then find the prototype vector v corresponding to its true category in the LUT t And update:

[0127]

[0128] Among them, γ is a hyperparameter and needs to be set manually.

[0129] If the feature vector has no category label, it is directly stored in the circular queue CQ. If the circular queue CQ is full, the earliest unlabeled feature vector is taken out.

[0130] Step S42: Based on the two special data structure types of LUT and CQ mentioned above, a feature vector CM is defined by the softmax function c_l The probability of being considered as a pedestrian of type i is:

[0131]

[0132] The prototype vector corresponding to the real category in LUT is transposed as follows

[0133] The prototype vector corresponding to the j1th category in the LUT is transposed as follows:

[0134] The prototype vector of CQ corresponding to the j2th unknown pedestrian is transposed as follows: τ is the temperature parameter that controls the similarity space and needs to be set manually;

[0135] The probability of being considered as an unmarked pedestrian is:

[0136]

[0137] The prototype vector of CQ corresponding to the i'th unknown pedestrian is transposed to

[0138] The prototype vector corresponding to the j1th category in the LUT is transposed as follows:

[0139] The prototype vector of CQ corresponding to the j2th unknown pedestrian is transposed as follows:

[0140] The final loss function is Loss reid =E x [log p t ], where p tis the probability of the true category corresponding to the feature vector, where Ex[] represents the cross entropy loss maximization p t , p t It is expressed as the probability that a feature vector is considered as a pedestrian of the tth category, where t is the true category of the feature vector in the dataset.

[0141] Step S43: First, calculate the similarity matrix of each class in the entire training set; the cosine similarity between the p-th class and the q-th class is calculated as: The prototype vector corresponding to the p-th category in the LUT is v p , the prototype vector corresponding to the qth category in LUT is v q , then obtain the similarity matrix between all classes, and calculate its average value as d0, where each class represents a leaf node in layer 0; then, based on the calculated similarity matrix, create a hierarchical structure by recursively merging different levels;

[0142] Set the hierarchical tree to L tree The average similarity is used as the threshold for merging the nodes of level 0. The two classes with a distance less than d0 are merged into the nodes of the first layer. The same operation is repeated for the nodes of the lth layer to obtain the nodes of the second layer, and so on. tree The layers are merged, and finally, a hierarchical tree H is generated, which starts from the leaf node of the original image class to the final top node.

[0143] Distance less than d L’-1 The two classes are merged into nodes of the L'th layer. In the L'th layer of the hierarchical tree (the number of layers except the 1st layer), the merging threshold for merging the L'-1th layer is set to d L’-1 =L'(4-d L’-2 ) / L'+d L，-2 .

[0144] Step S44: For each pedestrian feature vector CM c_l , take out the corresponding scene feature map AM L , through the full connection layer to a 2048-dimensional vector and normalized to obtain a 2048-dimensional scene feature vector GM L , directly connect the scene feature vector and the pedestrian feature vector to obtain the pedestrian scene feature vector CGM c_l In the early stage of training, pedestrian scene vectors in this batch are randomly sampled through the high-level class relationships of the hierarchical tree and positive and negative pairs are constructed. In the late stage of training, pedestrian scene vectors in this batch are randomly sampled through the low-level class relationships of the hierarchical tree and positive and negative pairs are constructed. The random sampling of the hierarchical tree is reduced by one level with each batch.

[0145] Define a total of N samples sampleA pair and M sample negative pairs, and then calculate the pedestrian scene structure modeling loss:

[0146]

[0147] For any positive pair sampled, the corresponding category index is represented as a, b,

[0148] represents the set of all categories contained in all pairs of categories a or b. CGM represents the pedestrian scene feature vector.

[0149] The total loss is the sum of all the pedestrian structure losses calculated for the scene:

[0150] Loss gallery =∑l a，b

[0151] Furthermore, step S5 specifically includes the following steps:

[0152] Step S51: Calculate the overall network loss function Loss, the calculation formula is as follows

[0153] Loss=Loss rpn +Loss det +Loss reid +Loss gallery

[0154] The overall optimization goal consists of four parts: Loss rpn is the proposal box detection loss, Loss det Loss for pedestrian detection reid Loss is the pedestrian re-identification loss. gallery It is the pedestrian scene structure loss.

[0155] Step S52: Perform iterative training according to the specified parameters, and continuously update the gradient according to the overall loss Loss calculated in step S51 to perform iterative training;

[0156] Step S53: During the inference process, by combining the scene information predictions and making full use of the complementarity of information, the pedestrian search box and pedestrian category information are finally obtained.

[0157] Step S54: During the training process, the model is verified at a certain iteration interval according to the verification interval flag, and the optimal model is continuously saved. When the number of iterations reaches the preset maximum iteration threshold, the training process ends and the optimal fine-grained prediction accuracy of the currently specified data set is returned.

[0158] The present invention effectively improves the network's ability to extract multi-granular feature information by constructing a feature pyramid module that cooperates with the pedestrian search network. In order to optimize target detection performance and solve the problem of imbalance between positive and negative samples in the dataset, the present invention adopts a focal loss method that is different from traditional loss. At the same time, a hierarchical tree is constructed using the pedestrian prototype features obtained by the pedestrian search network. Through this hierarchical tree, positive and negative samples in the pedestrian features are resampled. Objects of the same category often have similar structures, and the structural semantics are contained in the contextual associations of the discriminative local regions, thereby better helping the network learn the structure of the entire feature space and achieving better loss effects. In addition, we also construct a scene feature embedding module, which combines pedestrian features with scene features, fully considering accurate and diverse pedestrian positioning while extracting structural semantic information between scene images as a supplement, and then uses the positive and negative feature sampling results of the hierarchical tree to train the structural loss. This enables the network to reorder pedestrian features during the prediction stage, thereby improving the processing of similar pedestrian images and improving the performance on occluded and deformed pedestrian images.

[0159] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding, characterized by: The following steps are involved: Step S1: Obtain a pedestrian search dataset, input it into the backbone network, and then further enhance the feature representation by combining multi-scale information with a feature pyramid structure adapted to the backbone network. After completion, obtain a feature map that is input into subsequent modules; Step S2: Input the feature map into the region generation network to obtain a preliminary proposal box. According to the proposal box, the region of interest is extracted from the previously obtained feature map and a local proposal feature is generated. The discrimination loss and the proposal box regression loss are calculated. Step S3: The local proposal features are input into the multi-region convolutional neural network module to obtain more accurate pedestrian proposal boxes and pedestrian discrimination scores. The two regional convolutional neural network modules calculate the proposal box regression loss and discrimination loss twice. According to the pedestrian discrimination score, a threshold is set and the pedestrian proposal boxes with low scores are removed. The remaining pedestrian proposal boxes are used to extract pedestrian features in the image feature map. Step S4: Pedestrian features are input into the pedestrian re-identification module to calculate the pedestrian recognition loss. A pedestrian category hierarchy tree is constructed based on the cosine similarity calculated between the pedestrian categories in the dataset and the pedestrian re-identification features output by the pedestrian re-identification module. The image features output by the backbone network are then input into the scene encoding module to obtain scene features. Pedestrian features with known IDs are used as query pedestrian features. The query pedestrian features are combined with the scene features to obtain scene features. Positive and negative pairs of pedestrian scene features are then sampled through the pedestrian category hierarchy tree, and the pedestrian scene structure modeling loss is calculated. Step S5: Perform iterative training according to the specified training parameters, update the model parameters by optimizing the combined loss, and continuously save the optimal model based on the verification accuracy. The final model uses pedestrian information and image scene context information to comprehensively judge and obtain the pedestrian search results.

2. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 1 is characterized by: Step S1 specifically includes the following steps: Step S11: Using a public pedestrian search dataset, perform data preprocessing to complete label extraction and convert it into a usable format; Step S12: Input the entire scene image and first obtain the basic feature map through a backbone network, which contains L stages. The intermediate feature map output by the feature extractor is represented as Where size_H s_l ,size_W s_l ,size_C s_l They represent the height, width, and number of channels of the output feature map of the s_l stage of the backbone network, respectively, where s_l∈[1,L], and the feature map of the last stage of the network is used as the input of the feature pyramid. The feature maps of other stages are input into the feature pyramid in a horizontal connection manner. Step S13: The activation map is then input into a feature pyramid structure adapted to the backbone network. The feature pyramid consists of a convolutional neural network. Each layer has a horizontal connection, a bilinear interpolation upsampling operation, and a 1x1 convolution layer to ensure that the size of each layer corresponds to the size of the feature map in the backbone network. The number of layers included is the same as the number of stages of the backbone network, with a total of L layers. The intermediate feature map output by each layer is Where size_H f_l , size_W f_l They represent the height and width of the feature map output by the f_l layer of the feature pyramid, and the number of channels of all feature maps is unified to 256; where f_l∈[1,L], for each stage s_l in the backbone network, f_l=L+1-s_l, and the feature size of each layer of the feature pyramid output is size_H f_l =size_H L+1-s_l , size_W f_l =size_W L+1-s_l ; The output AM of the last layer of the feature pyramid module L As the next input.

3. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 2 is characterized by: The intermediate feature map output by each layer is AM f_l The specific acquisition is: First, FM s_l Horizontal connection LC in the f_l stage of the input feature pyramid f_l The horizontal feature representation LF generated in the f_l stage is obtained f_l , LF f_l =LC f_l (FM s_l ), where LC f_l (·) contains a 1×1 convolutional layer and a BatchNorm layer; when f_l=1, the input is FM s_l At this time, the intermediate feature map AM of the feature pyramid of this layer is obtained through a 1x1 convolution layer transformation channel f_l ; When f_l≠1, the input is AM f_l-1 , at this time it is necessary to pass through an upsampling layer UP f_l Get the intermediate feature map AM generated by the f_l stage of the feature pyramid f_l =UP f_l (LF f_l +AM f_l-1 ), where the upsampling layer UP f_l Contains bilinear interpolation upsampling operations and 1x1 convolutional layers.

4. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 3 is characterized by: Step S2 specifically includes the following steps: Step S21: Construct a region generation network based on the activation map AM obtained in step S13 L As input, for the activation map AM L For each feature pixel on the image, k proposal boxes of different sizes are selected. Specifically, a feature map with 256 channels in the input region generation network is input. For each pixel, a 3×3×256×256 convolution kernel is used to integrate the information near the pixel and generate a new vector of the same size with 256 dimensions. The whole feature map is passed through a 3x3 convolution layer IL, which outputs a new feature map of the same size as the original feature map. Step S22: Each 256-dimensional vector corresponds to two branches, one for the target and the other for the background. 2k ​​scores are obtained through a 1×1×256×2k judgment convolution kernel, where k is equal to the number of proposal boxes, indicating the score of the k proposal boxes as background and the score of the target. The whole is done through a 1x1 convolution layer CLS. rpn Get the corresponding category score: SCORE rpn_cls =CLS rpn (IL(AM L )),in If the proposed box is the target area, the position of the proposed box of the target area is determined. At this time, the other branch obtains 4k coordinates through the convolution kernel of 1×1×256×4k. Each box contains 4 area coordinates (x rpn ,y rpn ,w rpn ,h rpn ), where x rpn ,y rpn is the coordinate of the upper left corner of the proposal box, w rpn ,h rpn is the height and width, that is, the specific position where the boxes corresponding to the k candidate regions should be offset; the whole is through a 1x1 convolution layer REG rpn Get the corresponding regression value: SCORE rpn_reg =REG rpn (IL(AM L )),in Step S23: A series of rectangular proposal box lists {RP1, RP2, ..., RP wi n _K }, where win_K is the number of windows, win_K=size_H L ×size_W L ×k, each candidate proposal box contains four proposal box offset values ​​and two target judgment scores; some proposal boxes are discarded through the non-maximum suppression algorithm NMS; Then, we take out the corresponding marked box list from the dataset by calculating the overlap IOU between the proposed box and each real marked box in the dataset. Then calculate the loss for each box left after NMS screening and train it; Step S24: Introducing the discriminant loss Loss rpn_cls , and regression loss Loss rpn_reg , and the final total loss can be expressed as Loss rpn =Loss rpn_cls +Los Use the area coordinates of each proposal box after filtering from the original feature map AM L Get the local proposal feature and adjust it to 14×14×256 size as the next input local feature map PM p_l ,p_l∈[1,m], m is the number of proposal boxes after nms screening.

5. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 4 is characterized by: Introducing discriminant loss Loss rpn_cls and regression loss Loss rpn_reg Specifically, Loss rpn_cls For the cross entropy loss with softmax, first take out the predicted score of the corresponding category of the candidate proposal box after screening, and then scale it to [0,1] through softmax, denoted as p rpn Then take out the true value of the corresponding mark box category and record it as y rpn Loss rpn_cls =L(p rpn ,y rpn )=-y rpn log(p rpn )-(1-y rpn )log(1-p rpn ) Loss rpn_reg is the smooth L1 loss, x' rpn It is the absolute value of the difference between the proposed box regression value and the marked box.

6. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 5 is characterized by: Step S3 specifically includes the following steps: Step S31: Pooling the local feature map to obtain the pooled features is then fed into a multilayer perceptron consisting of two fully connected layers. ReLU functions are used between layers for further feature extraction. After feature extraction, the dimension remains unchanged, and the prediction phase is entered. The multilayer perceptron is denoted as MLP2, and the pooling layer is denoted as POOL. Step S32: In the prediction stage, it is responsible for predicting the local feature map, and then introduces two parallel fully connected layers; one fully connected layer is responsible for predicting the score of the category of the proposal box corresponding to the local feature map, and the fully connected layer is recorded as MLP cls ; The corresponding formula is SCORE det_cls =MLP cls (MLP2(POOL(PM p_l ))); Another fully connected layer is responsible for predicting the coordinate regression value of the proposal box corresponding to the local feature map, and this fully connected layer is recorded as MLP reg , the corresponding formula is SCORE det_reg =MLP reg (MLP2(POOL(PM p_l ))) According to the score matrix, a series of proposal box lists {R'P1, R'P2, ..., R'P anchor_k }, where anchor_K is the number of windows, each candidate proposal box contains four proposal box offset values ​​and two target judgment scores, and a part of the proposal box is discarded by the non-maximum suppression algorithm NMS, and then the corresponding marker box list is taken from the dataset by calculating the overlap IOU between the proposal box and each real marker box in the dataset. Then calculate the loss for each box left after NMS screening and train it; Step S33: Introducing the discriminant loss Loss det_cls , and regression loss Loss det_reg ; Finally, the total loss can be expressed as Loss det =Loss det_cls +Loss det_reg The area coordinates of each proposal box after filtering are used to retrieve the original feature map AM L The part is marked as the local pedestrian feature and adjusted to 14×14×256 size as the new input, and then step S3 is repeated again to further obtain a more accurate proposal frame, and then the final proposal frame is obtained from the original feature map AM. L The part is marked as the pedestrian feature and adjusted to 14×14×256 size, and finally transformed into a 2048-dimensional pedestrian feature vector through a global average pooling, denoted as CM c_l , c_l∈[1,n], n is the number of proposed boxes after two rounds of screening and detection; for each pedestrian feature vector, its true category can be obtained from the dataset according to the corresponding marked box n class is the total number of pedestrian label categories in the dataset, when When , it means that the pedestrian has no label in the dataset.

7. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 6 is characterized by: Introducing discriminant loss Loss det_cls and regression loss Loss det_reg Specifically: Loss det_cls For the focal loss with softmax, first take out the predicted score of the corresponding category of the proposal box, and then scale it to [0,1] through softmax, denoted as p det Then take out the true value of the corresponding mark box category and record it as y det ,γ is a hyperparameter and needs to be set manually; Loss det_reg is the focal L1 loss, x det It is the absolute value of the difference between the regression value of the proposed box and the true value of the marked box, where β is the hyperparameter that needs to be set, α and C are the intermediate hyperparameters calculated by β, and α = eβ and C = (2αlnβ+α) / 4.

8. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 7 is characterized by: Step S4 specifically includes the following steps: Step S41: After obtaining the feature vector of each pedestrian in the image, define a lookup table LUT to store the prototype vector of each marked pedestrian class, with a length of n class , the prototype vector stored in it is defined as v i ,i∈[1,n class ], and then define a circular queue CQ to store unmarked pedestrian vectors, with a length of n queue , is a manually set hyperparameter, and the unlabeled pedestrian vector stored in it is called u i ',i'∈[1,n queue ], in each iteration of training the network: First obtain the discriminant score s of each feature vector, and then find the prototype vector v corresponding to its true category in the LUT t And update: Among them, γ is a hyperparameter and needs to be set manually If the feature vector has no category label, it is directly stored in the circular queue CQ. If the circular queue CQ is full, the earliest unlabeled feature vector is taken out; Step S42: Based on the two special data structure types of LUT and CQ mentioned above, a feature vector CM is defined by the softmax function c_l The probability of being considered as a pedestrian of type i is: The prototype vector corresponding to the real category in LUT is transposed as follows The prototype vector corresponding to the j1th category in the LUT is transposed as follows: The prototype vector of CQ corresponding to the j2th unknown pedestrian is transposed as follows: τ is the temperature parameter that controls the similarity space and needs to be set manually; The probability of being considered as an unmarked pedestrian is: The prototype vector of CQ corresponding to the i'th unknown pedestrian is transposed to The final loss function is Loss reid =E x [log p t ], where p t is the probability of the true category corresponding to the feature vector, Ex[] represents the cross entropy loss maximization p t , p t Expressed as the probability that a feature vector is considered as a pedestrian of the tth category, where t is the true category of the feature vector in the dataset; Step S43: First, calculate the similarity matrix of each class in the entire training set; the cosine similarity between the p-th class and the q-th class is calculated as: The prototype vector corresponding to the p-th category in the LUT is v p , the prototype vector corresponding to the qth category in LUT is v q , then obtain the similarity matrix between all classes and calculate its average value as d0, where each class represents a leaf node in layer 0; then, based on the calculated similarity matrix, create a hierarchical structure by recursively merging different levels; Set the hierarchical tree to L tree Level, and use the average similarity as the threshold for merging level 0 nodes. Two classes with a distance less than d0 are merged into nodes of the lth layer. Repeat the same operation for nodes of the lth layer to obtain nodes of the second layer, and so on. Nodes are from the 0th layer to the Lth layer. tree The layers are merged, and finally, a hierarchical tree H is generated, which starts from the leaf node of the original image class to the final top node. Distance less than d L’-1 The two classes are merged into nodes of the L'th layer. In the L'th layer of the hierarchical tree except the 1st layer, the merging threshold for merging the L'-1th layer is set to d L’-1 =L'(4-d L’-2 ) / L'+d L’-2 ; Step S44: For each pedestrian feature vector CM c_l , take out the corresponding scene feature map AM L , through the full connection layer to a 2048-dimensional vector and normalized to obtain a 2048-dimensional scene feature vector GM L , the pedestrian scene feature vector CGM is obtained by directly connecting the scene feature vector and the pedestrian feature vector c_l In the early stage of training, pedestrian scene vectors in this batch are randomly sampled through the high-level class relationship of the hierarchical tree and positive and negative pairs are constructed; in the late stage of training, pedestrian scene vectors in this batch are randomly sampled through the low-level class relationship of the hierarchical tree and positive and negative pairs are constructed. The random sampling of the hierarchical tree is reduced by one level each time a batch is passed; Define a total of N samples sample A pair and M sample negative pairs, and then calculate the pedestrian scene structure modeling loss.

9. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 8, characterized in that: The calculation of pedestrian scene structure modeling loss is specifically: For any positive pair sampled, the corresponding category index is represented as a,b. represents the set of all categories contained in all pairs of categories a or b. CGM represents the pedestrian scene feature vector. The total loss is the sum of all the pedestrian scene structure losses calculated directly: Loss gallery =∑l a,b 。 10. The pedestrian search method based on similarity hierarchical tree reordering and scene feature embedding according to claim 1, characterized in that: Step S5 specifically includes the following steps: Step S51: Calculate the overall network loss function Loss, the calculation formula is as follows Loss=Loss rpn +Loss det +Loss reid +Loss gallery The overall optimization goal consists of four parts: Loss rpn is the proposal box detection loss, Loss det Loss for pedestrian detection reid Loss is the pedestrian re-identification loss. gallery It is the pedestrian scene structure loss; Step S52: Perform iterative training according to the specified parameters, and continuously update the gradient according to the overall loss Loss calculated in step S51 to perform iterative training; Step S53: In the inference process, by combining the scene information predictions and making full use of the complementarity of information, the pedestrian search box and pedestrian category information are finally obtained; Step S54: During the training process, the model is verified at a certain iteration interval according to the verification interval flag, and the optimal model is continuously saved. When the number of iterations reaches the preset maximum iteration threshold, the training process ends and the optimal fine-grained prediction accuracy of the currently specified data set is returned.

Images