Pedestrian re-identification method and system based on cross-modal feature fusion, and medium

Abstract

The invention discloses a pedestrian re-identification method and system based on cross-modal feature fusion and a medium, and the method comprises the steps: carrying out the human body contour detection of a to-be-detected video stream obtained in real time, and obtaining a first continuous frame set and a second continuous frame set; respectively inputting the first continuous frame set and the second continuous frame set into a preset target recognition model so as to extract target gait features from the first continuous frame set through a first extraction branch and extract appearance features and clothes features from the second continuous frame set through a second extraction branch, mapping the target gait feature to a semantic subspace to obtain a gait identity vector, and performing feature fusion on the gait identity vector and the appearance feature to obtain a target feature vector; and performing similarity matching on the target feature vector to determine a pedestrian re-identification result. According to the invention, the accuracy and robustness of pedestrian re-identification can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video monitoring, in particular to a pedestrian re-identification method and system based on cross-modal feature fusion and a medium. BACKGROUND

[0002] Pedestrian re-identification and gait recognition are two important technologies in the field of video monitoring and intelligent security, aiming to identify the identity of target individuals across cameras or across scenes. In modern urban environments, with the widespread deployment of monitoring systems, the demand for management and monitoring of personnel flow is increasing, for example, in airports, subway stations, shopping centers and other public places, real-time and accurate tracking and identification of personnel identity is crucial to ensure public safety, optimize personnel management and improve user experience.

[0003] Existing pedestrian re-identification technology mainly relies on the modeling of appearance information, such as color, texture and local structure of the human body, etc. static features, however, this method will significantly decrease the recognition performance when facing the change of clothes, occlusion or illumination of pedestrians. On the other hand, although gait recognition technology can identify identity through the time series features of human motion, it relies on standardized data sets and is difficult to cover the diversified behavior patterns in real monitoring environments, resulting in insufficient generalization ability of the model in actual application. In addition, the existing gait recognition and pedestrian re-identification technologies are relatively independent in research path, lacking a unified modeling framework that effectively integrates the advantages of both, this separate modeling approach limits the research and practical landing of cross-modal feature fusion, and cannot fully utilize appearance and motion information to improve recognition accuracy and generalization ability. Therefore, in order to overcome the defects of the existing technology, a feature fusion method that can jointly model pedestrian time series behavior and appearance static features is needed to achieve more robust and efficient pedestrian re-identification. SUMMARY

[0004] The present application provides a pedestrian re-identification method, system and medium based on cross-modal feature fusion to improve the accuracy and robustness of pedestrian re-identification through cross-modal feature fusion.

[0005] An embodiment of the present application provides a pedestrian re-identification method based on cross-modal feature fusion, comprising:

[0006] performing human contour detection on the real-time acquired video stream to be detected to obtain a first continuous frame set and a second continuous frame set;

[0007] input the first continuous frame set and the second continuous frame set into a preset target recognition model, to extract target gait features from the first continuous frame set through a first extraction branch, to extract appearance features and clothing features from the second continuous frame set through a second extraction branch, to map the target gait features to a semantic subspace to obtain a gait identity vector, and to perform feature fusion on the gait identity vector and the appearance features to obtain a target feature vector, wherein the target recognition model comprises the first extraction branch and the second extraction branch, and the target recognition model is obtained by pre-training using a staged progressive optimization strategy, and the semantic subspace is constructed for the second extraction branch through a singular value decomposition method;

[0008] perform similarity matching on the target feature vector to determine a pedestrian re-identification result.

[0009] The embodiments of the present application can accurately and completely extract the contour information of pedestrians from video streams through human contour detection, reduce the interference of background noise and irrelevant information, make the feature extraction more focused on the pedestrians themselves, and thus improve the accuracy of pedestrian re-identification. By extracting target gait features and clothing features respectively, different feature information of pedestrians can be fully utilized, and pedestrians can be more comprehensively described, thus improving the accuracy of pedestrian re-identification. In the case where the appearance of pedestrians changes or the gait is disturbed, the model can better cope with these changes by extracting multiple features and performing fusion, thus enhancing the robustness. By mapping the target gait features to a semantic subspace, features of different modalities can be aligned to the same semantic space, which has better semantic consistency, thus realizing more effective feature fusion and further improving the accuracy of pedestrian re-identification. Through similarity matching, the most similar target to the current pedestrian can be retrieved from the database based on the fused target feature vector, thus improving the accuracy of pedestrian re-identification. Compared with the prior art, the present application can improve the accuracy and robustness of pedestrian re-identification through cross-modal feature fusion.

[0010] Further, the human contour detection is performed on the real-time acquired video stream to be detected to obtain the first continuous frame set and the second continuous frame set, specifically:

[0011] The video stream to be detected is acquired, and a human body detection is performed on each frame of image in the video stream to be detected through a target detection algorithm to identify the position area of the pedestrian;

[0012] A tracking algorithm is used to track the position area to obtain a pedestrian trajectory, and a continuous frame sequence containing the pedestrian is cropped from each video segment according to the pedestrian trajectory;

[0013] input the continuous frame sequence into a preset visual detection model to perform human contour detection on each continuous frame to obtain a contour mask set as the first continuous frame set, and extract a color image sequence containing only pedestrian parts from the continuous frame sequence in combination with the contour mask set to determine a second continuous frame set.

[0014] In this way, the contour information of the pedestrian can be accurately and completely extracted from the video stream through human contour detection, the interference of background noise and irrelevant information is reduced, feature extraction is more focused on the pedestrian itself, and thus the accuracy of pedestrian re-identification is improved.

[0015] Further, the target gait feature is extracted from the first continuous frame set by a first extraction branch, specifically:

[0016] The first extraction branch processes the input first continuous frame set according to a global branch and a local branch;

[0017] In the local branch, the first continuous frame set is horizontally divided into four parts, and each part is input into a three-dimensional convolutional neural network to obtain a local gait feature of the part, and the local gait features are spliced to obtain a target local gait feature;

[0018] In the global branch, the first continuous frame set is input into a three-dimensional convolutional neural network to extract a global gait feature;

[0019] The global gait feature and the target local gait feature are added to obtain the target gait feature.

[0020] In this way, the gait information of the pedestrian can be more comprehensively captured through the multi-scale and global combined feature extraction method, and thus the accuracy and robustness of pedestrian re-identification are improved.

[0021] Further, the appearance feature and the clothing feature are extracted from the second continuous frame set by a second extraction branch, specifically:

[0022] Each frame in the second continuous frame set is divided into a plurality of image units by the second extraction branch;

[0023] The splicing result of each image unit and an initial identity vector is input into a visual encoder to output an identity vector feature and a plurality of appearance features;

[0024] Each appearance feature is fused to obtain an aggregated image unit feature, and the aggregated image unit feature and the identity vector feature are decoupled to obtain a clothing feature.

[0025] In this way, by extracting the appearance feature and the clothing feature, the inherent identity feature of the pedestrian can be effectively distinguished from the variable clothing feature, so that the pedestrian re-identification model can still accurately extract and utilize the stable biological feature for identity recognition when facing the situation of the pedestrian changing clothes or the clothing being blocked, thereby significantly improving the accuracy and robustness of pedestrian re-identification.

[0026] Further, the appearance feature and the clothing feature are obtained by decoupling each of the image unit features, specifically:

[0027] Each of the image unit features is fused along the time dimension using one-dimensional convolution to obtain an aggregated image unit feature.

[0028] Based on the aggregated image unit feature and the identity vector feature, the clothing feature not containing the identity feature is determined.

[0029] In this way, the inherent identity feature of the pedestrian can be effectively distinguished from the variable clothing feature, so that the pedestrian re-identification model can still accurately extract and utilize the stable biological feature for identity recognition when facing the situation of the pedestrian changing clothes or the clothing being blocked, thereby significantly improving the accuracy and robustness of pedestrian re-identification.

[0030] Further, the target gait feature is mapped to a semantic subspace to obtain a gait identity vector, specifically:

[0031] The visual encoder semantic space matrix of the second extraction branch is singular value decomposed, and the maximum singular value of a preset dimension parameter and the corresponding basis vector in the singular value matrix are extracted to constitute a semantic subspace;

[0032] The target gait feature is mapped to the semantic subspace by linear transformation to obtain a mapping result, and the mapping result is input into a multi-layer perception machine to be converted into a gait identity feature vector.

[0033] In this way, by mapping the target gait feature to the semantic subspace and converting it into a gait identity feature vector, the gait feature and the visual feature are effectively aligned and fused, thereby enhancing the accuracy and robustness of pedestrian re-identification by utilizing stable gait information in the case of pedestrian appearance change or occlusion.

[0034] Further, the target feature vector is subjected to similarity matching to determine a pedestrian re-identification result, specifically:

[0035] The similarity between the target feature vector and a preset feature vector is calculated, and the similarity is compared with a preset similarity threshold;

[0036] If the similarity is higher than the preset similarity threshold, the matching is successful, and the pedestrian re-identification result is determined.

[0037] In this way, by similarity matching, the target most similar to the current pedestrian can be retrieved from the database based on the fused target feature vector, thereby improving the accuracy of pedestrian re-identification.

[0038] Further, the target recognition model further comprises a third extraction branch, and the training process of the target recognition model comprises:

[0039] perform human contour detection on the obtained original image sequence to obtain a training data set;

[0040] construct an initial recognition model, input the training data set into the initial recognition model, and independently train the first extraction branch and the second extraction branch by using a staged progressive optimization strategy; when the training period is greater than a preset period threshold, jointly train the first extraction branch, the second extraction branch and the third extraction branch until the multi-task loss function corresponding to the model prediction result is minimized, and determine the trained target recognition model according to the model parameters of the current iteration, wherein the model parameters of the recognition model are adjusted each time to update the outputs of the first extraction branch, the second extraction branch and the third extraction branch based on the updated model parameters.

[0041] In this way, by using the staged progressive optimization strategy, the extraction capability and fusion effect of the model on the features such as the appearance and gait of pedestrians can be gradually improved at different stages, and finally the accuracy and robustness of pedestrian re-identification are improved.

[0042] Another embodiment of the present application also provides a pedestrian re-identification system based on cross-modal feature fusion, comprising: an acquisition module, a recognition module and a matching module;

[0043] The acquisition module is configured to perform human contour detection on a real-time acquired video stream to be detected to obtain a first continuous frame set and a second continuous frame set;

[0044] The recognition module is configured to input the first continuous frame set and the second continuous frame set into a preset target recognition model respectively, extract target gait features from the first continuous frame set by a first extraction branch, extract appearance features and clothing features from the second continuous frame set by a second extraction branch, map the target gait features to a semantic subspace to obtain a gait identity vector, and perform feature fusion on the gait identity vector and the appearance features to obtain a target feature vector, wherein the target recognition model comprises the first extraction branch and the second extraction branch, and the target recognition model is obtained by pre-training by using a staged progressive optimization strategy, and the semantic subspace is a space constructed for the second extraction branch by a singular value decomposition method;

[0045] The matching module is configured to perform similarity matching on the target feature vector to determine a pedestrian re-identification result.

[0046] The embodiment of the present application can accurately and completely extract the contour information of pedestrians from the video stream through human contour detection, reduce the interference of background noise and irrelevant information, make the feature extraction more focused on the pedestrians themselves, and thus improve the accuracy of pedestrian re-identification; by extracting target gait features and clothing features respectively, different feature information of pedestrians can be fully utilized, and pedestrians can be more comprehensively described, thus improving the accuracy of pedestrian re-identification; in the case that the appearance of pedestrians changes or the gait is disturbed, the model can better cope with these changes by extracting multiple features and fusing them, thus enhancing the robustness; by mapping the target gait features to a semantic subspace, the features of different modalities can be aligned to the same semantic space, having better semantic consistency, thus realizing more effective feature fusion, and further improving the accuracy of pedestrian re-identification; through similarity matching, the most similar target to the current pedestrian can be retrieved from the database based on the fused target feature vector, thus improving the accuracy of pedestrian re-identification. Compared with the prior art, the present application can improve the accuracy and robustness of pedestrian re-identification through cross-modal feature fusion.

[0047] Another embodiment of the present application also provides a computer-readable storage medium item, comprising: a stored computer program, when the computer program runs, controls the device where the computer-readable storage medium is located to execute the steps of the pedestrian re-identification method based on cross-modal feature fusion provided by the present application. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0049] Figure 1 is a flowchart of an embodiment of the pedestrian re-identification method based on cross-modal feature fusion provided by the present application;

[0050] Figure 2 is a flowchart of an embodiment of steps S201 to S203 provided by the present application;

[0051] Figure 3 is a processing flowchart of the visual detection model provided by the present application;

[0052] Figure 4 is a training process diagram of the target recognition model provided by the present application;

[0053] Figure 5 is a schematic diagram of a processing flow of a third extraction branch provided by the present application;

[0054] Figure 6 is a schematic diagram of a processing flow of a first extraction branch provided by the present application;

[0055] Figure 7 is a schematic diagram of a processing flow of a second extraction branch provided by the present application;

[0056] Figure 8 is a schematic diagram of a processing flow of singular value decomposition provided by the present application;

[0057] Figure 9 is a schematic diagram of a structure of an embodiment of a pedestrian re-identification system based on cross-modal feature fusion provided by the present application. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms “include” and “have” and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover not exclusive inclusion.

[0060] In the description of the embodiments of the present application, the technical terms “first”, “second”, and the like are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of “a plurality of” is two or more, unless otherwise explicitly and specifically limited.

[0061] Reference herein to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0062] In the description of the embodiments of the present application, the term "and / or" is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship.

[0063] In the description of the embodiments of the present application, the term "a plurality of" refers to two or more (including two), and similarly, "a plurality of groups" refers to two or more groups (including two groups), and "a plurality of pieces" refers to two or more pieces (including two pieces).

[0064] In the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the technical terms "mounting", "connection", "connection", "fixing" and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanical connection, or it can be electrical connection; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0065] Pedestrian re-identification and gait recognition are important technologies in the field of video surveillance and intelligent security, used for identifying target individuals across cameras or scenes. In modern urban environments, there is an increasing demand for real-time, accurate tracking and identity verification of personnel flow in public places such as airports, subway stations and shopping centers, which is crucial for ensuring public safety, optimizing personnel management and improving user experience. However, existing pedestrian re-identification technology mainly relies on static appearance features (such as color, texture and local structure of the human body), and the recognition performance will decrease significantly when the pedestrian changes clothes, is blocked or the lighting changes. Although gait recognition technology can identify identity through the timing characteristics of human motion, it relies on standardized data sets and is difficult to adapt to diverse behavior patterns in real monitoring environments, with insufficient generalization ability. In addition, the research paths of the two are relatively independent, and lack of a unified modeling framework, limiting the research and practical landing of cross-modal feature fusion, and failing to fully utilize appearance and action information to improve recognition accuracy and generalization ability.

[0066] Referring to Figure 1 To improve the accuracy and robustness of pedestrian re-identification, an embodiment of the present application provides a pedestrian re-identification method based on cross-modal feature fusion, comprising steps S101 to S103.

[0067] Step S101, human body contour detection is performed on the real-time acquired video stream to be detected to obtain a first continuous frame set and a second continuous frame set;

[0068] In some embodiments, step S101 includes but is not limited to steps S201-S203, specifically:

[0069] Step S201, obtaining a to-be-detected video stream, and performing human body detection on each frame of image in the to-be-detected video stream by using a target detection algorithm to identify the position area of the pedestrian;

[0070] In some embodiments, first, the to-be-detected video stream is obtained from a monitoring camera or a network video recorder; then, each frame of image in the to-be-detected video stream is processed by using a target detection algorithm (such as YOLOv8) to quickly and accurately detect the human body target in the image, and a bounding box is assigned to each detected pedestrian, so as to identify the position area of the pedestrian.

[0071] Step S202, tracking the position area by using a tracking algorithm to obtain a pedestrian trajectory, and cutting out a continuous frame sequence containing the pedestrian from each video segment according to the pedestrian trajectory;

[0072] In some embodiments, the detected pedestrian position area is tracked by using a tracking algorithm (such as SORT algorithm) to confirm the pedestrian trajectory, and a continuous frame sequence containing the pedestrian is cut out from the video stream according to the pedestrian trajectory, and usually a certain number of continuous frames (for example, 8 frames) are retained to ensure that there is enough information for subsequent feature extraction.

[0073] It should be noted that the tracking algorithm assigns a unique tracking ID to each pedestrian according to the bounding box information of the pedestrian position area, and generates a trajectory buffer area of the pedestrian.

[0074] It should be noted that the cut-out area should reduce background information as much as possible to ensure that subsequent processing only focuses on the pedestrian itself.

[0075] Step S203, inputting the continuous frame sequence into a preset visual detection model to perform human body contour detection on each continuous frame to obtain a contour mask set as the first continuous frame set, and extracting a color image sequence containing only the pedestrian part from the continuous frame sequence in combination with the contour mask set to determine a second continuous frame set.

[0076] In some embodiments, the processing flow diagram of the visual detection model is as shown in Figure 3As shown, first, the cropped continuous frame sequence x containing pedestrians is input into a preset visual detection model (such as a DINOv2 model) to perform human contour detection on each continuous frame to generate human contour masks, which can accurately identify the position and shape of pedestrians in the image, form a contour mask set, and use the contour mask set as the first continuous frame set x1 for gait feature extraction; then, a color image sequence containing only the pedestrian part is extracted from the original continuous frame sequence using the generated contour mask set, and the specific operation is to apply the mask to the original image, remove the background part, and retain the pedestrian part to form a second continuous frame set x2.

[0077] It should be noted that before generating the second continuous frame set x2, the continuous frame sequence of the pedestrian also needs to be subjected to data enhancement processing (horizontal flip, color enhancement, etc.) to obtain an image sequence containing complete color and texture information. The specific data enhancement is not the focus of the present application, and therefore will not be expanded here.

[0078] It should be noted that the first continuous frame set is a binary image containing only pedestrian contour information, which is used for subsequent extraction of gait features; and the second continuous frame set is a color image sequence containing only the pedestrian part, which is a color image retaining the color and texture information of the pedestrian, and is used for subsequent extraction of appearance features.

[0079] In this way, through human contour detection, the contour information of the pedestrian can be accurately and completely extracted from the video stream, reducing the interference of background noise and irrelevant information, making the feature extraction more focused on the pedestrian itself, thereby improving the accuracy of pedestrian re-identification.

[0080] In step S102, the first continuous frame set and the second continuous frame set are respectively input into a preset target recognition model to extract target gait features from the first continuous frame set through a first extraction branch, extract appearance features and clothing features from the second continuous frame set through a second extraction branch, map the target gait features to a semantic subspace to obtain a gait identity vector, and perform feature fusion on the gait identity vector and the appearance features to obtain a target feature vector. The target recognition model includes the first extraction branch and the second extraction branch, and the target recognition model is obtained by pre-training using a staged progressive optimization strategy. The semantic subspace is constructed for the second extraction branch by a singular value decomposition method;

[0081] In some embodiments, the target recognition model further includes a third extraction branch, and the training process of the target recognition model is as follows: Figure 4, specifically: performing human contour detection on the obtained original image sequence x to obtain a training data set; constructing an initial recognition model, inputting the training data set into the initial recognition model, and using a staged progressive optimization strategy to independently train the first extraction branch and the second extraction branch; when the training period is greater than a preset period threshold, jointly training the first extraction branch, the second extraction branch, and the third extraction branch until the multi-task loss function corresponding to the model prediction result is minimized, and determining the trained target recognition model according to the current iteration model parameter, wherein, at each iteration, the model parameters of the recognition model are adjusted to update the outputs of the first extraction branch, the second extraction branch, and the third extraction branch based on the updated model parameters.

[0082] In some embodiments, the constructed initial recognition model includes a first extraction branch, a second extraction branch, a third extraction branch, and a feature fusion alignment mechanism, wherein the first extraction branch is responsible for extracting gait features (gait feature F) from the contour mask sequence, the second extraction branch is responsible for extracting appearance features (identity vector feature, clothing feature) from the color image sequence, the third extraction branch is responsible for extracting biological features (identity prototype vector) that do not contain clothing information from the contour mask sequence, and the feature fusion alignment mechanism is used to align features of different modalities to the same semantic space for fusion and recognition.

[0083] It should be noted that the processing flow diagram of the third extraction branch is as shown in Figure 5 The input contour mask sequence x1 is converted into n image units 1 (each frame is n, and the resolution size of each unit is 16x16), a random vector is initialized, which is an identity prototype vector used to learn the biological features of the pedestrian that do not contain clothing information, and is spliced at the beginning of the n image units 1. The n+1 units are input into the frozen visual encoder, and an identity prototype vector feature dominated by geometric features and n image unit 1 features are output. The image unit features are fused along the time dimension using one-dimensional convolution to obtain aggregated image unit 1 features, and a value matrix and a key matrix are generated through a fully connected layer. The identity prototype vector feature is used as a query matrix to calculate cross-attention to obtain the final identity prototype vector.

[0084] In some embodiments, first, a pre-trained DINOv2 model is used to perform human contour detection on an input original image sequence to generate an image mask as a contour mask sequence x1 for gait feature extraction, and a color image sequence x2 containing only the pedestrian part of the original image is extracted using the mask (retaining complete color, texture, etc. features), and the x2 is subjected to data enhancement processing (horizontal flip, color enhancement, etc.) to generate a diversified training sample set to increase the richness of the training data and the robustness of the model. Second, the training data set (including the contour mask sequence x1 and the color image sequence x2) is input into the initial recognition model, and in the first 20 epochs of training, the pedestrian re-identification loss function L reID The preliminary learning of visual appearance features and time-series gait dynamic features optimizes the SVD subspace projection layer in the feature fusion alignment module to ensure the stability of the principal component basis of the cross-modal semantic subspace S k After independent training, the first extraction branch and the second extraction branch initially have the ability to extract pedestrian appearance features. Then, the third extraction branch is added for joint training with the first extraction branch and the second extraction branch (a total of 60 epochs of training), and the orthogonal constraint loss function L ORTH and the instance-level loss function L INS are introduced to combine the orthogonal constraint loss function L ORTH , the instance-level loss function L INS , and the pedestrian re-identification loss function L reID to construct a multi-task loss function L. In this stage, the learning rate of the first extraction branch and the second extraction branch is reduced to 10% of the initial value every 20 epochs to slow down the parameter update speed and avoid damaging the established feature correlation. At the same time, the multi-task loss function L is used to jointly optimize the parameters of the whole model until the multi-task loss function L corresponding to the predicted result of the model is minimized. The trained target recognition model is determined according to the current iteration of the model parameters.

[0085] It should be noted that the calculation formula of the multi-task loss function L is: L = L reID + γL ORTH + δL INS , which combines the pedestrian re-identification loss L reID , the orthogonal constraint loss function L ORTH , and the instance-level loss function L INS . By adjusting the weight coefficients α, β, γ, and δ, the influence of different loss functions is balanced to achieve multi-task optimization of the model and ultimately improve the accuracy and robustness of pedestrian re-identification. The pedestrian re-identification loss function L reID is composed of the cross-entropy loss function L ce and the triplet loss function L tri , and the calculation formula is: L reID = αLce + βL tri , where α and β are preset weight coefficients, which comprehensively consider the cross-entropy loss function L ce and the triplet loss function L tri , are used to optimize the extraction ability of the model to the appearance features of pedestrians, the cross-entropy loss function L ce is used to measure the difference between the predicted value of the model and the true label, and the smaller the difference is, the more accurate the prediction is, and the triplet loss L tri is used to pull the distance between samples of the same class closer and push the distance between samples of different classes farther, thereby enhancing the distinguishing ability of the model.

[0086] It should be noted that the orthogonal constraint loss function L ORTH forces the identity vector feature and the clothing vector to be orthogonal and irrelevant to each other by calculating the Frobenius norm of the dot product of the identity vector feature ID_token and the clothing feature Cloth_token, thereby realizing the orthogonal decoupling of the features, and the related formula is: where F ID is the identity vector feature ID_token, F Cloth is the clothing feature Cloth_token, and the purpose is to push the feature relationship between the identity vector feature ID_token and the clothing feature Cloth_token farther away, thereby forcing the identity vector feature and the clothing vector to be orthogonal and irrelevant to each other.

[0087] It should be noted that the instance-level loss function L INS pulls the spatial distance between the identity vector features and the identity prototype vectors of the same ID closer and pushes the spatial distance between the identity vector features and the identity prototype vectors of different IDs farther away by calculating the similarity between the identity vector features and the identity prototype vectors and performing normalization, thereby enhancing the distinguishing ability of the model to the identities of pedestrians, wherein the identity prototype vectors are mapped to the query matrix Query, the identity vector features are mapped to the key matrix Key and the value matrix Value, the cross-attention A is calculated to update the identity prototype features online, and the instance-level loss function is calculated using the identity vector features and the updated identity prototype features: where i is the ID of the i th pedestrian, j is the ID of the j th pedestrian, F' is the identity vector feature, F is the identity prototype vector, θ is the similarity, and τ is a control temperature coefficient (such as 0.07).

[0088] It should be noted that the cross-entropy loss function L ce and the triplet loss function L tri can be calculated based on the gait features extracted by the first extraction branch; and the orthogonal constraint loss function L ORTH; and calculate an instance-level loss function L based on the biological features extracted by the third extraction branch INS based on the instance-level loss function L INS updating the identity feature vector of the second extraction branch, thereby updating the model parameters.

[0089] In this way, the progressive optimization strategy in different stages can gradually improve the extraction ability and fusion effect of the model on the appearance and gait features of pedestrians, and finally improve the accuracy and robustness of pedestrian re-identification.

[0090] In some embodiments, the target gait features are extracted from the first set of continuous frames by a first extraction branch, wherein the processing flow diagram of the first extraction branch is as shown in Figure 6 The first extraction branch processes the input first set of continuous frames in a global branch and a local branch; in the local branch, the first set of continuous frames is divided into four parts horizontally, and each part is input into a three-dimensional convolutional neural network to obtain the local gait features of each part, and the local gait features are spliced to obtain target local gait features; in the global branch, the first set of continuous frames is input into a three-dimensional convolutional neural network to extract global gait features; the global gait features and the target local gait features are added to obtain target gait features. Specifically, first, after the target recognition model is trained, the first set of continuous frames and the second set of continuous frames are input into the target recognition model, at which time the first extraction branch is processed in a global branch and a local branch. Second, in the local branch processing, the horizontal direction of each frame is divided into four parts, which are input into four parallel three-dimensional convolutional neural networks (3D CNN), each of which is responsible for extracting the local gait features of the corresponding part, and the four local gait features are spliced to form a comprehensive local feature representation, i.e., target local gait features, wherein the three-dimensional convolutional neural network is specially designed for processing time series data and can capture the spatio-temporal dynamic features in the video. Then, in the global branch processing, the entire frame set is directly input into a separate three-dimensional convolutional neural network to extract global gait features. Finally, the local gait features and the global gait features are added to obtain a target gait feature that combines local and global information.

[0091] In this way, the multi-scale and global combined feature extraction method can more comprehensively capture the gait information of pedestrians, and this fusion helps the model to capture both the local details and the overall motion pattern of the pedestrian gait, thereby improving the accuracy and robustness of pedestrian re-identification.

[0092] In some embodiments, the appearance feature and the clothing feature are extracted from the second continuous frame set by a second extraction branch, wherein a processing flowchart of the second extraction branch is as shown in Figure 7 Specifically, each frame in the second continuous frame set is divided into a plurality of image units by the second extraction branch; the spliced result of each image unit and an initial identity vector is input into a visual encoder to output an identity vector feature and a plurality of appearance features; each appearance feature is fused to obtain an aggregated image unit feature, and the aggregated image unit feature is decoupled with the identity vector feature to obtain a clothing feature. Specifically, first, each frame image in the second continuous frame set x2 is divided into a plurality of image units (n for each frame, and the resolution size of each unit is 16x16) by the second extraction branch. These image units can be fixed-size blocks or regions dynamically generated by an algorithm (such as a sliding window). Then, a random vector is initialized as an initial identity vector, which is used to learn the identity feature of the pedestrian, and each image unit is spliced with the identity vector to form an input feature vector, which is input into a trainable visual encoder to obtain an identity vector feature and a plurality of appearance features. Then, a one-dimensional convolutional neural network is used to fuse the appearance features along the time dimension to integrate the appearance information in different time frames to obtain a more comprehensive appearance feature representation, thereby obtaining an aggregated image unit feature. Finally, a specific loss function (such as an orthogonal constraint loss function LORTH) is designed to force the identity vector feature and the clothing feature to be perpendicular and irrelevant to each other, and the aggregated image unit feature is subtracted by the identity vector feature to obtain a clothing feature that does not contain identity features. The purpose of this step is to decouple the identity information and clothing information in the appearance feature, so that the model can focus more on the clothing feature.

[0093] It should be noted that the visual encoder can be a convolutional neural network (CNN) or other deep learning model for extracting features of image units.

[0094] It should be noted that the identity vector feature is used to represent the identity information of the pedestrian, and the appearance feature is used to represent the appearance information such as clothing, color, and texture of the pedestrian.

[0095] It should be noted that since the result extracted by the third extraction branch is only used to optimize the target recognition model in the model inference process, it does not participate in the subsequent model fusion and matching part, i.e., it has no direct impact on the final pedestrian re-identification result, so the processing flow of the third extraction branch is not described here.

[0096] In this way, by extracting the appearance feature and the clothing feature, the inherent identity feature of the pedestrian can be effectively distinguished from the variable clothing feature, so that the pedestrian re-identification model can still accurately extract and use the stable biological feature for identity recognition when facing the situation of the pedestrian changing clothes or clothing occlusion, thereby significantly improving the accuracy and robustness of pedestrian re-identification.

[0097] In some embodiments, the decoupling of each of the image unit features to obtain the appearance feature and the clothing feature specifically includes: fusing each of the image unit features along a time dimension using one-dimensional convolution to obtain an aggregated image unit feature; and determining the clothing feature not containing the identity feature based on the aggregated image unit feature and the identity vector feature. Specifically, first, the obtained image unit features are arranged along the time dimension (i.e., the order of video frames) to form a feature sequence, and a one-dimensional convolutional neural network (1D CNN) is used to process the feature sequence to capture local patterns in the time sequence, thereby extracting time-continuous features. Through one-dimensional convolution, each image unit feature can be weighted and summed to obtain an aggregated image unit feature. Then, the aggregated image unit feature and the identity vector feature are decoupled to separate the clothing feature not containing the identity feature, which is specifically achieved by calculating the difference between the aggregated image unit feature and the identity vector feature, thereby obtaining the clothing feature. The clothing feature only contains information related to the clothing of the pedestrian, and does not contain identity information.

[0098] In this way, the inherent identity feature of the pedestrian can be effectively distinguished from the variable clothing feature, so that the pedestrian re-identification model can still accurately extract and use the stable biological feature for identity recognition when facing the situation of the pedestrian changing clothes or clothing occlusion, thereby significantly improving the accuracy and robustness of pedestrian re-identification.

[0099] In some embodiments, the mapping of the target gait feature to a semantic subspace to obtain a gait identity vector specifically includes: performing singular value decomposition on a semantic space matrix of a visual encoder of the second extraction branch, and extracting a maximum singular value of a preset dimension parameter and a corresponding basis vector in a singular value matrix to constitute a semantic subspace; and mapping the target gait feature to the semantic subspace through linear transformation to obtain a mapping result, and inputting the mapping result into a multi-layer perception machine to convert to obtain a gait identity feature vector. Specifically, since the clothing feature and the target gait feature are not in the same semantic space, they cannot be aligned for semantic fusion and recognition. Therefore, a space needs to be constructed to align the two features, so as to realize the recognition of pedestrians using the gait feature and the clothing feature. First, according to the semantic space matrix E output by the visual encoder of the second extraction branch, singular value decomposition is performed on the semantic space matrix E to obtain E=UΛV T, i.e. to obtain three matrices of basis vector matrix U and V, and singular value matrix Λ; secondly, according to a preset dimension parameter k, the largest k singular values and the corresponding basis vectors in U and V are extracted from the singular value matrix Λ, and these basis vectors constitute a semantic subspace S k ; then, the target gait feature F is mapped into the semantic subspace S k through linear transformation, which is usually achieved by dot product of F and the corresponding basis vector in U or V (depending on the implementation of SVD) to obtain a mapping result; finally, the mapping result is input into a multi-layer perception (MLP) to further extract features through nonlinear transformation to obtain a gait identity feature vector.

[0100] It should be noted that the processing flowchart of singular value decomposition is shown in Figure 8 , wherein for any m x n real matrix E, the singular value decomposition can be expressed as E = UΛV T . Wherein U is an m x m orthogonal matrix, whose column vectors are called left singular vectors; Λ is an m x n diagonal matrix, and the non-negative real numbers on the diagonal are called singular values, which are usually arranged in descending order; V T is an n x n orthogonal matrix, whose column vectors are called right singular vectors. From a geometric point of view, singular value decomposition can be understood as a process of transforming vectors in one high-dimensional space to another high-dimensional or low-dimensional space by left and right singular vectors. Specifically, the linear transformation corresponding to any matrix E can be decomposed into the composition of three basic transformations: a rotation (V T ), a scaling (Λ) and another rotation (U). This transformation is similar to transforming a unit sphere into an ellipsoid, and the singular values represent the lengths of the principal axes. The non-negative real numbers on the diagonal of Λ are arranged in descending order, and the first k larger singular values reflect the main information of the matrix.

[0101] In this way, by mapping the target gait feature to the semantic subspace and converting it into a gait identity feature vector, the effective alignment and fusion of gait features and visual features are achieved, so that the accuracy and robustness of pedestrian re-identification are enhanced by using stable gait information in the case of pedestrian appearance changes or occlusion.

[0102] In some embodiments, the gait identity vector and the appearance feature are fused to obtain a target feature vector. Specifically, the gait identity vector and the appearance feature are spliced to form a fusion feature vector, and the fusion feature vector is further processed through a fully connected layer to obtain a target feature vector.

[0103] Step S103, performing similarity matching on the target feature vector to determine a pedestrian re-identification result.

[0104] In some embodiments, the step of performing similarity matching on the target feature vector to determine the pedestrian re-identification result specifically involves: calculating the similarity between the target feature vector and a preset feature vector, and comparing the similarity with a preset similarity threshold; if the similarity is higher than the preset similarity threshold, the match is successful, and the pedestrian re-identification result is determined. Specifically, firstly, a similarity calculation method (such as cosine similarity, Euclidean distance, etc.) is used to calculate the similarity between the target feature vector and the preset feature vector. Then, the calculated similarity score is compared with a preset similarity threshold. If the similarity score is higher than the preset similarity threshold, the match is considered successful. If the match is successful, the target most similar to the current pedestrian features is retrieved from the database, and the similarity and related metadata, such as pedestrian ID, camera ID, etc., are returned, thus determining the pedestrian re-identification result. If a pedestrian is identified as a target, the system will trigger the corresponding business decision module, such as personnel flow statistics, warnings, etc.

[0105] It should be noted that the Approximate Nearest Neighbor (ANN) algorithm can also be used to retrieve the target most similar to the current pedestrian's features from the database, and return the similarity and related metadata to determine the pedestrian re-identification result.

[0106] By using similarity matching, the system can retrieve the most similar target to the current pedestrian from the database based on the fused target feature vector, thereby improving the accuracy of pedestrian re-identification.

[0107] This application's embodiments utilize human contour detection to accurately and completely extract pedestrian contour information from video streams, reducing interference from background noise and irrelevant information. This allows feature extraction to focus more on the pedestrian themselves, thereby improving the accuracy of pedestrian re-identification. By extracting target gait features and clothing features separately, different feature information of pedestrians can be fully utilized, providing a more comprehensive description of the pedestrian and improving the accuracy of pedestrian re-identification. When pedestrian appearance changes or gait is disturbed, by extracting and fusing multiple features, the model can better cope with these changes, thus enhancing robustness. By mapping target gait features to a semantic subspace, features from different modalities can be aligned to the same semantic space, resulting in better semantic consistency and more effective feature fusion, further improving the accuracy of pedestrian re-identification. Through similarity matching, based on the fused target feature vector, the database can be retrieved to find the target most similar to the current pedestrian, thereby improving the accuracy of pedestrian re-identification. Compared with existing technologies, this application can improve the accuracy and robustness of pedestrian re-identification through cross-modal feature fusion.

[0108] like Figure 9 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided;

[0109] An embodiment of the present application provides a pedestrian re-identification system based on cross-modal feature fusion, comprising an acquisition module 100, an identification module 200 and a matching module 300.

[0110] The acquisition module 100 is used for performing human contour detection on a real-time acquired video stream to be detected to obtain a first continuous frame set and a second continuous frame set.

[0111] The identification module 200 is used for inputting the first continuous frame set and the second continuous frame set into a preset target identification model respectively, extracting target gait features from the first continuous frame set through a first extraction branch, extracting appearance features and clothing features from the second continuous frame set through a second extraction branch, mapping the target gait features to a semantic subspace to obtain a gait identity vector, and performing feature fusion on the gait identity vector and the appearance features to obtain a target feature vector, wherein the target identification model comprises the first extraction branch and the second extraction branch, and the target identification model is obtained by pre-training by using a staged progressive optimization strategy, and the semantic subspace is constructed by using a singular value decomposition method for a space of the second extraction branch.

[0112] The matching module 300 is used for performing similarity matching on the target feature vector to determine a pedestrian re-identification result.

[0113] It can be understood that the above-mentioned device item embodiment is corresponding to the method item embodiment of the present application, and can realize the pedestrian re-identification method based on cross-modal feature fusion provided by any one of the above-mentioned method item embodiments.

[0114] It should be noted that the device embodiments described above are only schematic, and part or all of the modules can be selected to achieve the purpose of the present embodiment. In addition, in the device embodiment provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0115] On the basis of the above-mentioned embodiment of the pedestrian re-identification method based on cross-modal feature fusion, another embodiment of the present application provides a terminal device, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, the pedestrian re-identification method based on cross-modal feature fusion of any one embodiment of the present application is realized.

[0116] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the terminal device.

[0117] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing devices. The terminal device can include, but is not limited to, a processor and a memory.

[0118] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and connects all parts of the terminal device through various interfaces and lines.

[0119] On the basis of the above-mentioned method embodiment, another embodiment of the present application provides a computer readable storage medium, including a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located executes the pedestrian re-identification method based on cross-modal feature fusion described in any one of the above-mentioned method embodiments of the present application.

[0120] The modules / units integrated in the device / terminal equipment, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0121] The above is the preferred embodiment of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A pedestrian re-identification method based on cross-modal feature fusion, characterized in that, The method comprises the following steps: human body contour detection is performed on a real-time acquired video stream to be detected to obtain a first continuous frame set and a second continuous frame set; the first continuous frame set and the second continuous frame set are respectively input into a preset target recognition model, target gait features are extracted from the first continuous frame set through a first extraction branch, appearance features and clothing features are extracted from the second continuous frame set through a second extraction branch, the target gait features are mapped to a semantic subspace to obtain gait identity vectors, the gait identity vectors are fused with the appearance features to obtain target feature vectors, wherein the target recognition model comprises the first extraction branch and the second extraction branch, and the target recognition model is obtained by pre-training using a staged progressive optimization strategy, and the semantic subspace is constructed for the second extraction branch through a singular value decomposition method; similarity matching is performed on the target feature vectors to determine a pedestrian re-identification result.

2. The cross-modal feature fusion based pedestrian re-identification method according to claim 1, characterized in that, The human body contour detection on the real-time acquired video stream to be detected to obtain the first continuous frame set and the second continuous frame set is specifically as follows: a video stream to be detected is acquired, and a human body detection is performed on each frame of image in the video stream to be detected through a target detection algorithm to identify a position region of a pedestrian; a tracking algorithm is used to track the position region to obtain a pedestrian trajectory, and a continuous frame sequence containing the pedestrian is cropped from each video segment according to the pedestrian trajectory; the continuous frame sequence is input into a preset visual detection model to perform human body contour detection on each continuous frame to obtain a contour mask set as the first continuous frame set, and a color image sequence containing only a pedestrian part is extracted from the continuous frame sequence in combination with the contour mask set to determine the second continuous frame set. 3.The cross-modal feature fusion based pedestrian re-identification method according to claim 1, characterized in that, The target gait features are extracted from the first continuous frame set through the first extraction branch, and the specific process is as follows: the first extraction branch processes the input first continuous frame set according to a global branch and a local branch; in the local branch, the first continuous frame set is horizontally divided into four parts, and each part is input into a three-dimensional convolutional neural network to obtain a local gait feature, and the local gait features are spliced to obtain target local gait features; in the global branch, the first continuous frame set is input into a three-dimensional convolutional neural network to extract a global gait feature; the global gait feature and the target local gait features are added to obtain the target gait features.

4. The cross-modal feature fusion based pedestrian re-identification method of claim 1, wherein, The appearance features and clothing features are extracted from the second continuous frame set through the second extraction branch, and the specific process is as follows: each frame in the second continuous frame set is divided into a plurality of image units through the second extraction branch; a splicing result of each image unit and an initial identity vector is input into a visual encoder to output an identity vector feature and a plurality of appearance features; each appearance feature is fused to obtain an aggregated image unit feature, and the aggregated image unit feature is decoupled with the identity vector feature to obtain a clothing feature.

5. The cross-modal feature fusion based pedestrian re-identification method according to claim 4, characterized in that, The appearance features and clothing features are obtained by decoupling each image unit feature, and the specific process is as follows: The image unit features are fused along a time dimension using one-dimensional convolution to obtain aggregated image unit features; Based on the aggregated image unit features and the identity vector features, the clothing features not containing identity features are determined.

6. The cross-modal feature fusion based pedestrian re-identification method of claim 1, wherein, The target gait features are mapped to a semantic subspace to obtain a gait identity vector, specifically: The singular value decomposition is performed on the visual encoder semantic space matrix of the second extraction branch, and the maximum singular value and the corresponding basis vector of a preset dimension parameter in the singular value matrix are extracted to form a semantic subspace; The target gait features are mapped to the semantic subspace through linear transformation to obtain a mapping result, and the mapping result is input into a multi-layer perception machine to be converted to obtain a gait identity feature vector.

7. The cross-modal feature fusion based pedestrian re-identification method of claim 1, wherein, The target feature vector is subjected to similarity matching to determine a pedestrian re-identification result, specifically: The similarity between the target feature vector and a preset feature vector is calculated, and the similarity is compared with a preset similarity threshold; If the similarity is higher than the preset similarity threshold, the matching is successful, and a pedestrian re-identification result is determined.

8. The cross-modal feature fusion based pedestrian re-identification method of claim 1, wherein, The target recognition model further includes a third extraction branch, and the training process of the target recognition model is specifically: Human contour detection is performed on the obtained original image sequence to obtain a training data set; An initial recognition model is constructed, the training data set is input into the initial recognition model, a staged progressive optimization strategy is adopted to independently train the first extraction branch and the second extraction branch, when a training period is greater than a preset period threshold, the first extraction branch, the second extraction branch and the third extraction branch are jointly trained until a multi-task loss function corresponding to a model prediction result is minimized, and a trained target recognition model is determined according to current iteration model parameters, wherein the model parameters of the recognition model are adjusted each time, and the outputs of the first extraction branch, the second extraction branch and the third extraction branch are updated based on the updated model parameters.

9. A pedestrian re-identification system based on cross-modal feature fusion, characterized in that, It includes: An acquisition module, a recognition module and a matching module; The acquisition module is configured to perform human contour detection on a real-time acquired video stream to be detected to obtain a first continuous frame set and a second continuous frame set; The recognition module is configured to input the first continuous frame set and the second continuous frame set into a preset target recognition model respectively, to extract target gait features from the first continuous frame set through a first extraction branch, to extract appearance features and clothing features from the second continuous frame set through a second extraction branch, to map the target gait features to a semantic subspace to obtain a gait identity vector, and to perform feature fusion on the gait identity vector and the appearance features to obtain a target feature vector, wherein the target recognition model includes the first extraction branch and the second extraction branch, and the target recognition model is trained by using a staged progressive optimization strategy in advance, and the semantic subspace is constructed by a singular value decomposition method for the second extraction branch; The matching module is configured to perform similarity matching on the target feature vector to determine a pedestrian re-identification result.

10. A computer-readable storage medium, characterized in that, It includes: A computer program product stored in the computer readable storage medium, wherein the steps of the pedestrian re-identification method based on cross-modal feature fusion as claimed in any one of claims 1-8 are performed by a device in which the computer readable storage medium is located when the computer program runs.

Images