Shielded pedestrian re-identification method, device and equipment and storage medium

Abstract

The invention discloses an occluded pedestrian re-identification method and device, equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: capturing the global context information of a pedestrian image through employing a first branch network of a preset dual-branch network, and obtaining a global feature, generating a private feature view set based on the human body key point information of the pedestrian image extracted by a posture estimator in the second branch network; obtaining a shared feature view set by using a multi-head attention mechanism of a preset decoder based on a posture heat map generated by the human body key point information, a preset semantic matrix and global features; using a Hungary algorithm to match the private feature view set and the shared feature view set to determine a joint feature set to train the initial pedestrian re-recognition model, and using a target loss function to optimize the initial pedestrian re-recognition model; and identifying the to-be-identified pedestrian image based on the obtained target pedestrian re-identification model to obtain an identification result. The accuracy of identifying pedestrians under the shielding condition is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for re-identifying an occluded pedestrian. Background Art

[0002] In modern banking environments, applications such as security monitoring, customer behavior analysis, and intelligent anti-fraud are increasingly relying on person re-identification technology. As densely populated and complex spaces, banks pose a significant challenge to person re-identification. Traditional person re-identification methods rely primarily on complete images for identity verification. However, in real-world applications, pedestrians are often obscured by other objects or people, making accurate identification difficult.

[0003] As can be seen from the above, how to improve the accuracy of pedestrian recognition under occlusion conditions is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for re-identifying pedestrians under occlusion, which can improve the accuracy of pedestrian recognition under occlusion. The specific solution is as follows:

[0005] In a first aspect, the present application provides a method for pedestrian re-identification under occlusion, comprising:

[0006] The first branch network of a preset two-branch network is used to capture global context information of a pedestrian image to obtain global features. Then, a posture estimator in a second branch network of the preset two-branch network is used to extract human body key point information from the pedestrian image, and the human body key point information is used to generate a private feature view set related to the pedestrian's posture; the posture estimator is a posture estimator determined based on a deep neural network;

[0007] Based on the posture heat map generated by the human body key point information, the preset semantic matrix and the global features, a shared feature view set is obtained by using the multi-head attention mechanism of the preset decoder;

[0008] Matching the private feature view set and the shared feature view set using the Hungarian algorithm, and determining a joint feature set based on the matching result;

[0009] An initial person re-identification model is trained based on the joint feature set, and the initial person re-identification model is optimized using a target loss function during the training process to obtain a target person re-identification model. The pedestrian image to be identified is identified based on the target pedestrian re-identification model to obtain a recognition result; the target loss function is a loss function determined based on a quaternion loss function.

[0010] Optionally, the method of capturing global context information of the pedestrian image using the first branch network of the preset dual-branch network to obtain global features includes:

[0011] Acquire a surveillance video image of a bank branch, and enhance the surveillance video image using image enhancement technology to obtain a processed video image;

[0012] Detecting pedestrians in the processed video image using a target detection algorithm, and cropping the area where the pedestrians are located in the processed video image based on the detection result to obtain a pedestrian image;

[0013] The pedestrian image is encoded to obtain an encoded pedestrian image, and then the encoded pedestrian image is input into a preset visual model of a preset dual-branch network, and the preset visual model is used to capture global context information of the pedestrian image to obtain global features.

[0014] Optionally, the posture estimator in the second branch network based on the preset dual-branch network extracts human body key point information of the pedestrian image, and uses the human body key point information to generate a private feature view set related to the pedestrian posture, including:

[0015] Extracting key points of the pedestrian image based on a posture estimator in the second branch network of the preset dual-branch network to obtain human body key point information; the posture estimator is a posture estimator determined based on a high-resolution network;

[0016] Extract features from the pedestrian image using a preset feature extractor to obtain a local feature set, generate a corresponding posture heat map based on the human body key point information, and perform a linear transformation on the posture heat map to obtain a transformed heat map;

[0017] determining a posture guidance feature set based on the transformed heat map and the local feature set;

[0018] Feature matching is performed on the posture guidance feature set and the local feature set, and then feature fusion is performed based on the feature matching results to generate private features related to the pedestrian posture, and a private feature view set is determined based on the private features.

[0019] Optionally, the posture heat map generated based on the human body key point information, the preset semantic matrix and the global features, and the shared feature view set obtained by using the multi-head attention mechanism of the preset decoder, include:

[0020] Performing average pooling on the posture heat map to obtain a pooled heat map, and inputting the global features and the human body key point information into a Transformer encoder to obtain a corresponding output feature representation using the Transformer encoder;

[0021] Combining the pooled heat map with the output feature representation to obtain decoder input information;

[0022] Defining a preset semantic matrix, and inputting the decoder input information and the preset semantic matrix into a Transformer decoder, so that the Transformer decoder linearly maps the decoder input information and the preset semantic matrix to obtain a query, a key, and a value;

[0023] Performing parallel processing of a weighted integration of the query, the key, and the value based on a multi-head attention mechanism in the Transformer decoder to obtain an output tensor dimension;

[0024] The fully connected layers in the Transformer decoder and the output tensor dimensions are used for nonlinear transformation and integration to obtain a shared feature view set.

[0025] Optionally, matching the private feature view set and the shared feature view set using the Hungarian algorithm, and determining a joint feature set based on the matching result, includes:

[0026] Constructing a similarity matrix based on the similarity between the private feature view set and the shared feature view set;

[0027] Determining an augmenting path of a private feature view element in the private feature view set using the Hungarian algorithm and the similarity matrix, and determining a corresponding relationship between the private feature view set and the shared feature view set based on the augmenting path;

[0028] The private feature view set and the shared feature view set are merged based on the corresponding relationship to obtain a joint feature set.

[0029] Optionally, before optimizing the initial person re-identification model using a target loss function during training to obtain a target person re-identification model, the method further includes:

[0030] Determining a batch of images based on the joint feature set, and selecting an image from the batch of images as an anchor point;

[0031] Determining the distance between the anchor point and the sample images in the batch of images, and determining the sample image with the largest distance as a positive sample, determining the sample image with the smallest distance as a first negative sample, and determining the sample image with the second smallest distance as a second negative sample;

[0032] Determine feature maps corresponding to the positive sample, the first negative sample, and the second negative sample, respectively, and determine a target loss function based on the feature maps, the positive sample, the first negative sample, and the second negative sample.

[0033] Optionally, the step of training an initial person re-identification model based on the joint feature set, optimizing the initial person re-identification model using a target loss function during the training process to obtain a target person re-identification model, and performing recognition on the image of the person to be recognized based on the target person re-identification model to obtain a recognition result includes:

[0034] constructing an initial person re-identification model, inputting the joint feature set into the initial person re-identification model, and determining a loss value using a target loss function during training;

[0035] Optimizing the initial person re-identification model using the loss value to obtain a target person re-identification model;

[0036] The pedestrian image to be identified is identified based on the target pedestrian re-identification model to obtain a target feature vector corresponding to the pedestrian image to be identified, and the similarity between the target feature vector and the pedestrian feature vectors in the preset library is determined, and the pedestrian corresponding to the pedestrian feature vector with the highest similarity is determined as the target pedestrian.

[0037] In a second aspect, the present application provides a device for re-identifying pedestrians under occlusion, comprising:

[0038] A private view set determination module is configured to capture global context information of a pedestrian image using a first branch network of a preset two-branch network to obtain global features, then extract human body key point information from the pedestrian image based on a posture estimator in a second branch network of the preset two-branch network, and generate a private feature view set related to the pedestrian's posture using the human body key point information; the posture estimator is a posture estimator determined based on a deep neural network;

[0039] A shared view set determination module is configured to obtain a shared feature view set based on the posture heat map generated by the human body key point information, the preset semantic matrix, and the global features, and using the multi-head attention mechanism of the preset decoder;

[0040] a view matching module, configured to match the private feature view set and the shared feature view set using a Hungarian algorithm, and determine a joint feature set based on the matching result;

[0041] The recognition model training module is used to train the initial pedestrian re-identification model based on the joint feature set, and optimize the initial pedestrian re-identification model using the target loss function during the training process to obtain a target pedestrian re-identification model, and recognize the pedestrian image to be recognized based on the target pedestrian re-identification model to obtain a recognition result; the target loss function is a loss function determined based on the quaternion loss function.

[0042] In a third aspect, the present application provides an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] The processor is configured to execute the computer program to implement the aforementioned method for re-identifying an occluded pedestrian.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned obscured pedestrian re-identification method.

[0046] This application uses the first branch network of a preset two-branch network to capture the global context information of the pedestrian image to obtain global features, and then extracts the human key point information of the pedestrian image based on the posture estimator in the second branch network of the preset two-branch network, and uses the human key point information to generate a private feature view set related to the pedestrian posture; the posture estimator is a posture estimator determined based on a deep neural network; based on the posture heat map generated by the human key point information, the preset semantic matrix and the global features, and the multi-head attention mechanism of the preset decoder, a shared feature view set is obtained; the private feature view set and the shared feature view set are matched using the Hungarian algorithm, and a joint feature set is determined based on the matching results; the initial pedestrian re-identification model is trained based on the joint feature set, and the initial pedestrian re-identification model is optimized using the target loss function during the training process to obtain a target pedestrian re-identification model, and the pedestrian image to be identified is identified based on the target pedestrian re-identification model to obtain a recognition result; the target loss function is a loss function determined based on the quaternion loss function.

[0047] As can be seen from the above, the present application adopts a dual-branch network structure, extracting the global context information of the pedestrian image through the first branch network, and extracting the human key point information of the pedestrian image using the second branch network, and constructing a private feature view set related to the pedestrian posture based on the human key point information, and then determining the shared feature view set through the posture heat map generated by the human key point information, the preset semantic matrix and the global features, and then using the Hungarian algorithm to perform maximum matching of the private feature view set and the shared feature view set, so that each feature is effectively paired, solving the problem that shared and private features are difficult to effectively combine in traditional methods; then the target loss function is determined by the quaternary loss function to optimize the model, avoiding the noise and outlier problems that may occur in the traditional ternary loss function. This optimization makes the model more robust and can handle more complex situations in real scenes. In this way, the target pedestrian re-identification model is used to identify the pedestrian image to be identified, which greatly improves the accuracy of occluded pedestrian re-identification and reduces safety risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 This is a flow chart of a method for re-identifying an occluded pedestrian disclosed in this application;

[0050] Figure 2 A schematic diagram of a maximum matching process provided by this application;

[0051] Figure 3 This is a flowchart of a specific method for re-identifying an occluded pedestrian disclosed in this application;

[0052] Figure 4 This is a schematic structural diagram of a device for re-identifying an obscured pedestrian disclosed in this application;

[0053] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Currently, modern banking environments are increasingly relying on person re-identification technology for security monitoring, customer behavior analysis, and intelligent anti-fraud applications. Traditional person re-identification methods rely primarily on complete images for identity confirmation. However, in practical applications, pedestrians are often difficult to accurately identify due to occlusion by other objects or people. To address this, this application provides an occluded person re-identification method that utilizes a target person re-identification model to identify the image of the pedestrian to be identified, significantly improving the accuracy of occluded person re-identification and reducing security risks.

[0056] See also Figure 1 As shown, an embodiment of the present invention discloses a method for re-identifying an occluded pedestrian, comprising:

[0057] Step S11: Use the first branch network of a preset dual-branch network to capture the global context information of the pedestrian image to obtain global features, and then extract the human body key point information of the pedestrian image based on the posture estimator in the second branch network of the preset dual-branch network, and use the human body key point information to generate a private feature view set related to the pedestrian posture; the posture estimator is a posture estimator determined based on a deep neural network.

[0058] In this embodiment, a surveillance video image of a bank branch is obtained, and then the surveillance video image is enhanced and preprocessed using image enhancement technology to obtain a processed video image, and an image detection algorithm is used to detect pedestrians in the processed video image, and the area where the pedestrians are located in the processed video image is cropped based on the detection result to obtain a pedestrian image, and the pedestrian image is encoded using a Transformer (i.e., a deep learning model) encoder to obtain an encoded pedestrian image, and the encoded pedestrian image is input into a preset visual model of a preset dual-branch network, and the preset visual model is used to capture the global context information of the pedestrian image to obtain global features; the preset visual model can be db-ViT (i.e., a visual model based on a dual-branch architecture) or other models that can capture the global context information of pedestrian images.

[0059] Specifically, the first branch network of the preset dual-branch network is used to capture the global context information of the pedestrian image to obtain global features, including: obtaining the surveillance video image of the bank branch, and using image enhancement technology to enhance the surveillance video image to obtain a processed video image; using a target detection algorithm to detect pedestrians in the processed video image, and based on the detection result, cropping the area where the pedestrians are located in the processed video image to obtain a pedestrian image; encoding the pedestrian image to obtain an encoded pedestrian image, and then inputting the encoded pedestrian image into the preset visual model of the preset dual-branch network, and using the preset visual model to capture the global context information of the pedestrian image to obtain global features.

[0060] It can be understood that after obtaining the global features, a posture estimator is determined based on a high-resolution network (HRNet, i.e., High-Resolution Network), and the posture estimator in the second branch network of the preset dual-branch network is used to extract the key points of the pedestrian image to obtain human body key point information, and the key points may include but are not limited to the head key points, shoulder key points, knee key points and elbow key points of the pedestrian image; then a corresponding posture heat map is generated based on the human body key point information, each posture heat map corresponds to one key point, and the maximum corresponding point of each posture heat map corresponds to the position of the key point, and then the preset feature extractor is used to extract features from the pedestrian image to obtain a local feature set, and the posture heat map is linearly transformed based on the fully connected layer of the preset dual-branch network and the dimension of the local features in the local feature set, so that the dimension of the transformed heat map is the same as that of the local features, and then the product of the transformed heat map and the local features is used to obtain a posture guidance feature set. Then, feature matching is performed on the posture guidance feature set and the local feature set, that is, for each posture guidance feature of the posture guidance feature set , find the index position k of the target local feature that is most similar to it in the local feature set, and the corresponding formula is as follows:

[0061] ;

[0062] in, for and The inner product of is the first posture guidance features; is the first local features; express The norm of .

[0063] Further, after finding the most similar target local feature in the local feature set, Add it to the corresponding target local feature to get the corresponding private feature view element. The corresponding formula is as follows:

[0064] ;

[0065] in, To find the local feature set The most similar target local features; is the first After obtaining the private features related to the pedestrian posture, a private feature view set is determined based on the private features.

[0066] Specifically, the posture estimator in the second branch network based on the preset dual-branch network extracts the human body key point information of the pedestrian image, and uses the human body key point information to generate a private feature view set related to the pedestrian posture, including: the posture estimator in the second branch network based on the preset dual-branch network extracts the key points of the pedestrian image to obtain human body key point information; the posture estimator is a posture estimator determined based on a high-resolution network; the pedestrian image is feature extracted using a preset feature extractor to obtain a local feature set, and a corresponding posture heat map is generated based on the human body key point information, and the posture heat map is linearly transformed to obtain a transformed heat map; a posture guidance feature set is determined based on the transformed heat map and the local feature set; feature matching is performed on the posture guidance feature set and the local feature set, and then feature fusion is performed based on the feature matching results to generate private features related to the pedestrian posture, and a private feature view set is determined based on the private features.

[0067] Step S12: Based on the posture heat map generated by the human body key point information, the preset semantic matrix and the global features, a shared feature view set is obtained using the multi-head attention mechanism of the preset decoder.

[0068] In this embodiment, after obtaining the private feature view set, the posture heat map is average pooled to obtain a pooled heat map, and the global features and the human body key point information are input into the Transformer encoder to obtain the corresponding output feature representation using the Transformer encoder. , combine the pooled heat map and the output feature representation to obtain the decoder input information. Then define a preset semantic matrix , the decoder input information and the preset semantic matrix are input to the Transformer decoder, so that the Transformer decoder linearly maps the decoder input information and the preset semantic matrix to obtain the query, key and value. The corresponding formula is as follows:

[0069] ;

[0070] ;

[0071] ;

[0072] in, For the The vector corresponding to the query; is the first OK; is the linear mapping matrix corresponding to the query; For the A vector corresponding to each key; The output feature represents the OK; is the linear mapping matrix corresponding to the key; No. The vector corresponding to the values; is the linear mapping matrix corresponding to the value, and is the dimension before linear mapping; 、 as well as is the dimension after linear mapping.

[0073] It can be understood that after obtaining the query, the key and the value, the query, the key and the value are weighted and integrated in parallel based on the multi-head attention mechanism in the Transformer decoder to obtain the output tensor dimension, and then the fully connected layer in the Transformer decoder and the output tensor dimension are used to perform nonlinear transformation and integration to obtain a shared feature view set.

[0074] Specifically, the posture heat map, the preset semantic matrix and the global features generated based on the human key point information, and the shared feature view set obtained by using the multi-head attention mechanism of the preset decoder, include: performing average pooling processing on the posture heat map to obtain a pooled heat map, and inputting the global features and the human key point information into the Transformer encoder to obtain the corresponding output feature representation using the Transformer encoder; combining the pooled heat map and the output feature representation to obtain decoder input information; defining a preset semantic matrix, and inputting the decoder input information and the preset semantic matrix into the Transformer decoder, so that the Transformer decoder linearly maps the decoder input information and the preset semantic matrix to obtain queries, keys and values; performing weighted integration and parallel processing on the queries, keys and values based on the multi-head attention mechanism in the Transformer decoder to obtain the output tensor dimension; and performing nonlinear transformation and integration using the fully connected layer in the Transformer decoder and the output tensor dimension to obtain a shared feature view set.

[0075] Step S13: Use the Hungarian algorithm to match the private feature view set and the shared feature view set, and determine a joint feature set based on the matching result.

[0076] In this embodiment, after obtaining the shared feature view set, a joint feature set is determined based on the similarity between the private feature view set and the shared feature view set using the Kuhn-Munkres algorithm. The corresponding formula is as follows:

[0077] ;

[0078] ;

[0079] in, is the first A private feature view element; is the first shared feature view elements; In order to make and The index position of the target private feature view element with the greatest similarity between them. For example, if there is a shared feature view element in the shared feature view set and five private feature view elements in the private feature view set 、 、 、 、 By calculating the cosine similarity of a shared feature view element and five private feature view elements, we get 0.7, 0.6, 0.9, 0.5, and 0.8 respectively. Therefore, k=3. Since the largest similarity is 0.9, it can be considered that the private feature view element For the shared feature view element The best matching index of the private feature view set is then connected with each best matching index in the shared feature view set to obtain a joint feature set .

[0080] It can be understood that the joint feature set is obtained based on the maximum matching between the private feature view set and the shared feature view set. Figure 2 This embodiment provides a schematic diagram of a maximum matching process. The maximum matching is found in the constructed subgraph (i.e., similarity matrix). If a conflict occurs during the search for the maximum matching and the conflict cannot be reconciled, the search range is expanded, i.e., more edges and elements are considered to find a new match, or the maximum matching requirement is lowered. In a specific embodiment, the similarity matrix is first constructed using the similarity between the private feature view set and the shared feature view set. For example, the first digit in the similarity matrix is 0. Rank The elements of the column are the first Shared Feature View Elements With the private feature view set Private feature view elements Prepare an empty matching set M to store the final matching pairs, and try to find an augmenting path for each private feature view element in the private feature view set. Specifically, for the current private feature view element Initialize an empty augmenting path P and Add to P, and then define an empty set visited={} to record the vertices that have been visited to avoid repeated visits. Start looking for Neighbors of ,Will Add to visited and judge Is it already in the current path P? If it is already in the current path P, it means that an odd loop has appeared, that is, an abnormal situation has occurred, and you need to find The next neighbor of . If it is not in the current path P, then judge Is it matched? If not matched, Add to P and mark it as a potential endpoint; if is matched, then If the matching node w is added to the path P, the preset recursive function is used to find the augmenting path starting from w. If the preset recursive function returns a non-empty path, the path is determined as the augmenting path, and then the matching relationship is updated by reversing the matching edges on the augmenting path; that is, for each pair of adjacent nodes (x, y) on the augmenting path, if there is a matching edge between x and y, the matching direction is reversed from x matching y to y matching x to update the matching set M. If all neighbors of are found but no augmenting path is found, null is returned.

[0081] Specifically, the Hungarian algorithm is used to match the private feature view set and the shared feature view set, and a joint feature set is determined based on the matching result, including: constructing a similarity matrix based on the similarity between the private feature view set and the shared feature view set; using the Hungarian algorithm and the similarity matrix to determine the augmenting path of the private feature view elements in the private feature view set, and determining the correspondence between the private feature view set and the shared feature view set based on the augmenting path; and merging the private feature view set and the shared feature view set based on the correspondence to obtain a joint feature set.

[0082] Step S14: train an initial person re-identification model based on the joint feature set, and optimize the initial person re-identification model using a target loss function during the training process to obtain a target person re-identification model, and identify the pedestrian image to be identified based on the target pedestrian re-identification model to obtain a recognition result; the target loss function is a loss function determined based on a quaternion loss function.

[0083] In this embodiment, after obtaining the joint feature set, the joint feature set is used as a batch of images, and an image is randomly selected from the batch of images as an anchor point. Based on the distance between the anchor point and each sample image in the batch of images, the sample image with the largest distance is determined as a positive sample, the sample image with the smallest distance is determined as a first negative sample, and the sample image with the second smallest distance is determined as a second negative sample. Then, based on the positive sample, the first negative sample, and the second negative sample, a corresponding feature map is determined using a convolutional neural network model. The target loss function is determined based on the feature map, the positive sample, the first negative sample, and the second negative sample. The corresponding formula is as follows:

[0084] ;

[0085] in, is the target loss function; is the size of the feature map; is the anchor point; is the positive sample; is the first negative sample; is the second negative sample; is the feature map of the anchor point; is the preset boundary value.

[0086] Specifically, the method of using the target loss function to optimize the initial person re-identification model during training to obtain the target person re-identification model also includes: determining a batch of images based on the joint feature set, and selecting an image from the batch of images as an anchor point; determining the distance between the anchor point and the sample image in the batch of images, and determining the sample image with the largest distance as the positive sample, determining the sample image with the smallest distance as the first negative sample, and determining the sample image with the second smallest distance as the second negative sample; determining the feature maps corresponding to the positive sample, the first negative sample and the second negative sample respectively, and determining the target loss function based on the feature maps, the positive sample, the first negative sample and the second negative sample.

[0087] It can be understood that after obtaining the target loss function, an initial pedestrian re-identification model is constructed, and then the joint feature set is input into the initial pedestrian re-identification model, and the target loss function is used to determine the corresponding loss value during the training process, and the initial pedestrian re-identification model is optimized using the loss value to obtain a target pedestrian re-identification model. The pedestrian image to be identified is identified based on the target pedestrian re-identification model to obtain a target feature vector corresponding to the pedestrian image to be identified, and the similarity between the target feature vector and each pedestrian feature vector in the preset gallery is calculated, and the pedestrian corresponding to the pedestrian feature vector with the highest similarity is determined as the target pedestrian.

[0088] Specifically, the initial pedestrian re-identification model is trained based on the joint feature set, and the initial pedestrian re-identification model is optimized using the target loss function during the training process to obtain a target pedestrian re-identification model, and the pedestrian image to be identified is identified based on the target pedestrian re-identification model to obtain a recognition result, including: constructing an initial pedestrian re-identification model, inputting the joint feature set into the initial pedestrian re-identification model, and determining the loss value using the target loss function during the training process; optimizing the initial pedestrian re-identification model using the loss value to obtain a target pedestrian re-identification model; identifying the pedestrian image to be identified based on the target pedestrian re-identification model to obtain a target feature vector corresponding to the pedestrian image to be identified, and determining the similarity between the target feature vector and each pedestrian feature vector in a preset gallery, and determining the pedestrian corresponding to the pedestrian feature vector with the highest similarity as the target pedestrian.

[0089] As can be seen from the above, the present application adopts a dual-branch network structure, extracting the global context information of the pedestrian image through the first branch network, and extracting the human key point information of the pedestrian image using the second branch network, and constructing a private feature view set related to the pedestrian posture based on the human key point information, and then determining the shared feature view set through the posture heat map generated by the human key point information, the preset semantic matrix and the global features, and then using the Hungarian algorithm to perform maximum matching of the private feature view set and the shared feature view set, so that each feature is effectively paired, solving the problem that shared and private features are difficult to effectively combine in traditional methods; then the target loss function is determined by the quaternary loss function to optimize the model, avoiding the noise and outlier problems that may occur in the traditional ternary loss function. This optimization makes the model more robust and can handle more complex situations in real scenes. In this way, the target pedestrian re-identification model is used to identify the pedestrian image to be identified, which greatly improves the accuracy of occluded pedestrian re-identification and reduces safety risks.

[0090] It can be seen from the above embodiments that the present application trains the out-of-line sliding pedestrian re-identification model based on the joint feature set determined by the dual-branch network to improve the accuracy of occluded pedestrian re-identification. Therefore, the process of training the out-of-line sliding pedestrian re-identification model based on the joint feature set determined by the dual-branch network is described.

[0091] See also Figure 3 As shown, first, a surveillance video image of a bank branch is obtained, and the surveillance video image is preprocessed such as image enhancement to obtain a processed video image. The encoded processed video image is subjected to global feature extraction using a preset visual model of a preset dual-branch network to obtain global features. The posture estimator in the second branch network of the preset dual-branch network is used to extract human key point information in the processed video image, and the preset feature extractor is used to extract local features in the processed video image to obtain a local feature set. Then, a posture heat map is generated based on the human key point information, and a linear transformation is performed on the posture heat map to obtain a transformed heat map. Feature matching and feature fusion are performed based on the transformed heat map and the local feature set to generate private features related to pedestrian posture to obtain a private feature view set.

[0092] The global features and the human key point information are then input into the Transformer encoder to obtain an output feature representation. The posture heat map is then average pooled, the pooled heat map obtained is combined with the output feature representation, and a shared feature view set is generated through a multi-head attention mechanism. A similarity matrix is constructed based on the similarity between the private feature view set and the shared feature view set, and the Hungarian algorithm is used to determine the augmentation path of the private feature view elements in the private feature view set to find the best match to obtain a joint feature set. The initial pedestrian re-identification model is trained using the joint feature set, and the trained pedestrian re-identification model is optimized using an improved quaternion loss function and a target loss function determined by combining the male sample mining idea. Specifically, a batch of images is determined based on the joint feature set, and anchor points are selected from the batch of images. The positive sample, the first negative sample, and the second negative sample are determined based on the distance between the anchor point and the sample image in the batch of images.

[0093] After obtaining the sample, a loss value is determined based on the feature maps corresponding to the positive sample, the first negative sample, and the second negative sample, respectively, and the target loss function is used to optimize the trained person re-identification model based on the loss value to obtain a target person re-identification model. An image of a person to be identified is obtained, and the target person re-identification model is used to identify the image of the person to be identified to obtain a target feature vector corresponding to the image of the person to be identified. The target feature vector is then compared with the feature vectors of each person in a preset image library to determine the target person.

[0094] As can be seen above, this application uses a preset dual-branch network to extract global features and private features from pedestrian images, and uses Transformer to generate shared features. The Hungarian algorithm is then used to match the two feature sets. Finally, the resulting joint feature set is used to train the initial pedestrian re-identification model to achieve accurate recognition of occluded pedestrians. In this way, occluded pedestrians are recognized through the global context information and posture-related information of the pedestrian image, effectively improving the accuracy and robustness of pedestrian re-identification.

[0095] Accordingly, see Figure 4 As shown, the present application also provides a device for re-identifying pedestrians under occlusion, comprising:

[0096] The private view set determination module 11 is configured to capture global context information of a pedestrian image using a first branch network of a preset two-branch network to obtain global features, then extract human body key point information of the pedestrian image based on a posture estimator in a second branch network of the preset two-branch network, and generate a private feature view set related to the pedestrian posture using the human body key point information; the posture estimator is a posture estimator determined based on a deep neural network;

[0097] A shared view set determination module 12 is configured to obtain a shared feature view set based on the posture heat map generated by the human body key point information, the preset semantic matrix, and the global features, and using a multi-head attention mechanism of a preset decoder;

[0098] a view matching module 13, configured to match the private feature view set and the shared feature view set using the Hungarian algorithm, and determine a joint feature set based on the matching result;

[0099] The recognition model training module 14 is used to train the initial pedestrian re-identification model based on the joint feature set, and optimize the initial pedestrian re-identification model using the target loss function during the training process to obtain a target pedestrian re-identification model, and recognize the pedestrian image to be recognized based on the target pedestrian re-identification model to obtain a recognition result; the target loss function is a loss function determined based on the quaternion loss function.

[0100] As can be seen from the above, the present application adopts a dual-branch network structure, extracting the global context information of the pedestrian image through the first branch network, and extracting the human key point information of the pedestrian image using the second branch network, and constructing a private feature view set related to the pedestrian posture based on the human key point information, and then determining the shared feature view set through the posture heat map generated by the human key point information, the preset semantic matrix and the global features, and then using the Hungarian algorithm to perform maximum matching of the private feature view set and the shared feature view set, so that each feature is effectively paired, solving the problem that shared and private features are difficult to effectively combine in traditional methods; then the target loss function is determined by the quaternary loss function to optimize the model, avoiding the noise and outlier problems that may occur in the traditional ternary loss function. This optimization makes the model more robust and can handle more complex situations in real scenes. In this way, the target pedestrian re-identification model is used to identify the pedestrian image to be identified, which greatly improves the accuracy of occluded pedestrian re-identification and reduces safety risks.

[0101] In some specific implementations, the private view set determining module 11 may specifically include:

[0102] An image enhancement processing unit, configured to acquire a surveillance video image of a bank branch and perform enhancement processing on the surveillance video image using image enhancement technology to obtain a processed video image;

[0103] a pedestrian detection unit, configured to detect pedestrians in the processed video image using a target detection algorithm, and crop the area of the processed video image where the pedestrians are located based on the detection result to obtain a pedestrian image;

[0104] A global feature determination unit is used to encode the pedestrian image to obtain an encoded pedestrian image, and then input the encoded pedestrian image into a preset visual model of a preset dual-branch network, and use the preset visual model to capture the global context information of the pedestrian image to obtain global features.

[0105] In some specific implementations, the private view set determining module 11 may specifically include:

[0106] A key point extraction unit, configured to extract key points of the pedestrian image based on a posture estimator in the second branch network of the preset dual-branch network to obtain human body key point information; the posture estimator is a posture estimator determined based on a high-resolution network;

[0107] a heat map conversion unit, configured to extract features from the pedestrian image using a preset feature extractor to obtain a local feature set, generate a corresponding posture heat map based on the human body key point information, and perform a linear transformation on the posture heat map to obtain a transformed heat map;

[0108] a feature set determining unit, configured to determine a posture guidance feature set based on the transformed heat map and the local feature set;

[0109] A private feature generation unit is used to perform feature matching on the posture guidance feature set and the local feature set, and then perform feature fusion based on the feature matching results to generate private features related to the pedestrian posture, and determine a private feature view set based on the private features.

[0110] In some specific implementations, the shared view set determining module 12 may specifically include:

[0111] an output feature representation unit, configured to perform average pooling on the posture heat map to obtain a pooled heat map, and input the global features and the human body key point information into a Transformer encoder to obtain a corresponding output feature representation using the Transformer encoder;

[0112] an input information determining unit, configured to combine the pooled heat map and the output feature representation to obtain decoder input information;

[0113] a matrix mapping unit, configured to define a preset semantic matrix and input the decoder input information and the preset semantic matrix into a Transformer decoder, so that the Transformer decoder linearly maps the decoder input information and the preset semantic matrix to obtain a query, a key, and a value;

[0114] A tensor dimension determination unit, configured to perform weighted integration and parallel processing of the query, the key, and the value based on a multi-head attention mechanism in the Transformer decoder to obtain an output tensor dimension;

[0115] A tensor dimension integration unit is used to perform nonlinear transformation and integration using the fully connected layer in the Transformer decoder and the output tensor dimension to obtain a shared feature view set.

[0116] In some specific implementations, the view matching module 13 may specifically include:

[0117] a similarity matrix construction unit, configured to construct a similarity matrix based on the similarity between the private feature view set and the shared feature view set;

[0118] an augmenting path determining unit, configured to determine an augmenting path of the private feature view elements in the private feature view set by using the Hungarian algorithm and the similarity matrix, and determine a correspondence between the private feature view set and the shared feature view set based on the augmenting path;

[0119] The view set merging unit is configured to merge the private feature view set and the shared feature view set based on the corresponding relationship to obtain a joint feature set.

[0120] In some specific implementations, the obstructed pedestrian re-identification device may further include:

[0121] an anchor point determination unit, configured to determine a batch of images based on the joint feature set, and select an image from the batch of images as an anchor point;

[0122] a positive sample determining unit, configured to determine a distance between the anchor point and a sample image in the batch of images, and determine the sample image with the largest distance as a positive sample, the sample image with the smallest distance as a first negative sample, and the sample image with the second smallest distance as a second negative sample;

[0123] A target loss function determining unit is used to determine feature maps corresponding to the positive sample, the first negative sample, and the second negative sample, respectively, and determine a target loss function based on the feature maps, the positive sample, the first negative sample, and the second negative sample.

[0124] In some specific implementations, the recognition model training module 14 may specifically include:

[0125] a loss value determining unit, configured to construct an initial person re-identification model, input the joint feature set into the initial person re-identification model, and determine a loss value using a target loss function during training;

[0126] a model optimization unit, configured to optimize the initial person re-identification model using the loss value to obtain a target person re-identification model;

[0127] The target pedestrian determination unit is used to identify the pedestrian image to be identified based on the target pedestrian re-identification model to obtain a target feature vector corresponding to the pedestrian image to be identified, and determine the similarity between the target feature vector and the pedestrian feature vectors in the preset library, and determine the pedestrian corresponding to the pedestrian feature vector with the highest similarity as the target pedestrian.

[0128] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the obscured pedestrian re-identification method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0129] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0130] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0131] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20. It can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the obscured pedestrian re-identification method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0132] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for re-identifying pedestrians with obstructions. The specific steps of this method can be found in the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0134] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0136] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0137] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for re-identifying pedestrians under occlusion, characterized in that: include: The first branch network of a preset two-branch network is used to capture global context information of a pedestrian image to obtain global features. Then, a posture estimator in a second branch network of the preset two-branch network is used to extract human body key point information from the pedestrian image, and the human body key point information is used to generate a private feature view set related to the pedestrian's posture; the posture estimator is a posture estimator determined based on a deep neural network; Based on the posture heat map generated by the human body key point information, the preset semantic matrix and the global features, a shared feature view set is obtained by using the multi-head attention mechanism of the preset decoder; Matching the private feature view set and the shared feature view set using the Hungarian algorithm, and determining a joint feature set based on the matching result; An initial person re-identification model is trained based on the joint feature set, and the initial person re-identification model is optimized using a target loss function during the training process to obtain a target person re-identification model. The pedestrian image to be identified is identified based on the target pedestrian re-identification model to obtain a recognition result; the target loss function is a loss function determined based on a quaternion loss function.

2. The method for re-identifying pedestrians under occlusion according to claim 1, characterized in that: The method of using the first branch network of the preset dual-branch network to capture global context information of the pedestrian image to obtain global features includes: Acquire a surveillance video image of a bank branch, and enhance the surveillance video image using image enhancement technology to obtain a processed video image; Detecting pedestrians in the processed video image using a target detection algorithm, and cropping the area where the pedestrians are located in the processed video image based on the detection result to obtain a pedestrian image; The pedestrian image is encoded to obtain an encoded pedestrian image, and then the encoded pedestrian image is input into a preset visual model of a preset dual-branch network, and the preset visual model is used to capture global context information of the pedestrian image to obtain global features.

3. The method for re-identifying pedestrians under occlusion according to claim 1, characterized in that: The posture estimator in the second branch network based on the preset dual-branch network extracts human body key point information of the pedestrian image, and uses the human body key point information to generate a private feature view set related to the pedestrian posture, including: Extracting key points of the pedestrian image based on a posture estimator in the second branch network of the preset dual-branch network to obtain human body key point information; the posture estimator is a posture estimator determined based on a high-resolution network; Extract features from the pedestrian image using a preset feature extractor to obtain a local feature set, generate a corresponding posture heat map based on the human body key point information, and perform a linear transformation on the posture heat map to obtain a transformed heat map; determining a posture guidance feature set based on the transformed heat map and the local feature set; Feature matching is performed on the posture guidance feature set and the local feature set, and then feature fusion is performed based on the feature matching results to generate private features related to the pedestrian posture, and a private feature view set is determined based on the private features.

4. The method for re-identifying pedestrians under occlusion according to claim 3, characterized in that: The posture heat map generated based on the human body key point information, the preset semantic matrix and the global features, and the shared feature view set obtained by using the multi-head attention mechanism of the preset decoder, include: Performing average pooling on the posture heat map to obtain a pooled heat map, and inputting the global features and the human body key point information into a Transformer encoder to obtain a corresponding output feature representation using the Transformer encoder; Combining the pooled heat map with the output feature representation to obtain decoder input information; Defining a preset semantic matrix, and inputting the decoder input information and the preset semantic matrix into a Transformer decoder, so that the Transformer decoder linearly maps the decoder input information and the preset semantic matrix to obtain a query, a key, and a value; Performing parallel processing of a weighted integration of the query, the key, and the value based on a multi-head attention mechanism in the Transformer decoder to obtain an output tensor dimension; The fully connected layers in the Transformer decoder and the output tensor dimensions are used for nonlinear transformation and integration to obtain a shared feature view set.

5. The method for re-identifying pedestrians under occlusion according to claim 1, characterized in that: The matching of the private feature view set and the shared feature view set by using the Hungarian algorithm and determining a joint feature set based on the matching result includes: Constructing a similarity matrix based on the similarity between the private feature view set and the shared feature view set; Determining an augmenting path of a private feature view element in the private feature view set using the Hungarian algorithm and the similarity matrix, and determining a corresponding relationship between the private feature view set and the shared feature view set based on the augmenting path; The private feature view set and the shared feature view set are merged based on the corresponding relationship to obtain a joint feature set.

6. The method for re-identifying pedestrians under occlusion according to claim 1, characterized in that: Before optimizing the initial person re-identification model using the target loss function during the training process to obtain the target person re-identification model, the method further includes: Determining a batch of images based on the joint feature set, and selecting an image from the batch of images as an anchor point; Determining the distance between the anchor point and the sample images in the batch of images, and determining the sample image with the largest distance as a positive sample, determining the sample image with the smallest distance as a first negative sample, and determining the sample image with the second smallest distance as a second negative sample; Determine feature maps corresponding to the positive sample, the first negative sample, and the second negative sample, respectively, and determine a target loss function based on the feature maps, the positive sample, the first negative sample, and the second negative sample.

7. The method for re-identifying pedestrians under occlusion according to any one of claims 1 to 6, characterized in that: The initial person re-identification model is trained based on the joint feature set, and the initial person re-identification model is optimized using a target loss function during the training process to obtain a target person re-identification model. The image of the person to be identified is recognized based on the target person re-identification model to obtain a recognition result, including: constructing an initial person re-identification model, inputting the joint feature set into the initial person re-identification model, and determining a loss value using a target loss function during training; Optimizing the initial person re-identification model using the loss value to obtain a target person re-identification model; The pedestrian image to be identified is identified based on the target pedestrian re-identification model to obtain a target feature vector corresponding to the pedestrian image to be identified, and the similarity between the target feature vector and the pedestrian feature vectors in the preset library is determined, and the pedestrian corresponding to the pedestrian feature vector with the highest similarity is determined as the target pedestrian.

8. A device for re-identifying pedestrians under occlusion, characterized in that: include: A private view set determination module is configured to capture global context information of a pedestrian image using a first branch network of a preset two-branch network to obtain global features, then extract human body key point information from the pedestrian image based on a posture estimator in a second branch network of the preset two-branch network, and generate a private feature view set related to the pedestrian's posture using the human body key point information; the posture estimator is a posture estimator determined based on a deep neural network; A shared view set determination module is configured to obtain a shared feature view set based on the posture heat map generated by the human body key point information, the preset semantic matrix, and the global features, and using the multi-head attention mechanism of the preset decoder; a view matching module, configured to match the private feature view set and the shared feature view set using a Hungarian algorithm, and determine a joint feature set based on the matching result; The recognition model training module is used to train the initial pedestrian re-identification model based on the joint feature set, and optimize the initial pedestrian re-identification model using the target loss function during the training process to obtain a target pedestrian re-identification model, and recognize the pedestrian image to be recognized based on the target pedestrian re-identification model to obtain a recognition result; the target loss function is a loss function determined based on the quaternion loss function.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the method for re-identifying an occluded pedestrian according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the method for re-identifying an occluded pedestrian according to any one of claims 1 to 7 is implemented.

Images