Person re-identification method and system under incomplete observation conditions based on graph convolution

Abstract

The present invention belongs to the field of person re-identification technology, and in particular relates to a method and system for person re-identification under incomplete observation conditions based on graph convolution, comprising: obtaining a pedestrian image and preprocessing it; performing local feature extraction, global feature extraction, and attribute feature extraction on the preprocessed pedestrian image to obtain key point representation, global visual features, and text attribute features of the pedestrian image; inputting the global visual features and key point representation into an adaptive threshold-guided masked graph convolutional network to obtain visually guided key points; based on the text attribute features and key point representation, utilizing a cyclic heterogeneous graph convolutional network to obtain attribute-enhanced key points and visually enhanced attributes; and performing weighted fusion of the visually guided key points, attribute-enhanced key points, and visually enhanced attributes to obtain a final pedestrian representation. The cyclic heterogeneous graph convolutional network of the present invention effectively fuses structured visual and textual pedestrian information through intra-graph and inter-graph interactions.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian re-identification, and in particular relates to a method and system for pedestrian re-identification under incomplete observation conditions based on graph convolution. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Person re-identification (Person Re-identification), also known as pedestrian re-identification, or ReID for short, is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence; in other words, pedestrian re-identification refers to identifying the target pedestrian in a video sequence with existing possible sources and non-overlapping camera fields of view. It is widely considered to be a sub-problem of image retrieval. Given a surveillance pedestrian image, retrieve images of the pedestrian across devices. In surveillance videos, due to camera resolution and shooting angles, it is usually impossible to obtain very high-quality facial images. When face recognition fails, ReID becomes a very important alternative technology. A very important feature of ReID is that it is cross-camera. By retrieving images of the same pedestrian under different cameras, it can determine whether a specific pedestrian exists, which greatly improves the accuracy of pedestrian re-identification.

[0004] However, there are also certain difficulties in studying the task of pedestrian re-identification. The specific reasons are as follows:

[0005] 1) Due to differences in images captured by cameras at different times and in different spaces, some cameras may experience spatial misalignment due to occlusion or out-of-field visual field. In certain real-world scenarios, such as when a pedestrian walks out of the camera's field of view or is obscured by obstacles such as walls, cars, or other people, the person may not be fully captured. This results in insufficient global and local features in the acquired visual features, reducing the degree of matching between global and local features, which in turn affects model training performance and reduces the accuracy of person re-identification. This is a challenge faced by existing person re-identification algorithms.

[0006] 2) Pedestrian images taken at different times vary significantly. For pedestrian images taken during the same time period, such as at noon, the pedestrian features and outlines displayed are relatively clear, and there are fewer negative shadow areas in the pedestrian images. Simply relying on existing classic deep learning models can achieve good results. However, these pedestrian images taken at fixed times are not generalizable. For pedestrian images in actual application scenarios, the shooting time is often highly random, and pedestrian images have very high requirements for the brightness of the shooting environment. Different brightness levels will lead to significant differences in the outline features of pedestrian images. The influence of the shooting environment and brightness will also produce different background noises, which will bring great difficulties to the pedestrian re-identification task.

[0007] 3) Pedestrian images captured in different spatial domains vary significantly. In real-world applications, pedestrian images are collected from a variety of locations, and different cameras capture images from different angles. This results in a large number of highly diverse pedestrian images. When performing person re-identification tasks on these images, the model must not only be able to correctly identify pedestrians captured in different environments, but also have high recognition accuracy for images captured from different angles. This requires the model to be highly resistant to noise.

[0008] 4) A single captured image contains multiple pedestrians. Due to different image acquisition environments, multiple people may pass through the acquisition point. This raises another issue: how to recognize multiple people in images. Such images often contain more complex features, requiring a high level of noise immunity from the model. However, the performance of current person re-identification models for such images still needs to be improved.

[0009] To address these challenges, a masked pedestrian image classification method was proposed. This method randomly masks pixels in a subset of collected pedestrian images during model training, successfully improving person re-identification performance. However, due to the complexity and diversity of collected pedestrian images, this method generally has limitations. Furthermore, its re-identification performance for multiple pedestrian images falls short of expectations, demonstrating the model's poor noise immunity and its inability to accurately identify complex pedestrian images. Summary of the Invention

[0010] To overcome the above-mentioned deficiencies of the prior art, the present invention provides a method and system for pedestrian re-identification under incomplete observation conditions based on graph convolution, which improves the accuracy of pedestrian re-identification in different scenarios to a certain extent.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] A first aspect of the present invention provides a method for person re-identification under incomplete observation conditions based on graph convolution, comprising:

[0013] Acquire pedestrian images and preprocess the acquired images;

[0014] Perform local feature extraction, global feature extraction, and attribute feature extraction on the preprocessed pedestrian image to obtain key point representation, global visual features, and text attribute features of the pedestrian image;

[0015] The global visual features and key point representations are input into the adaptive threshold-guided mask graph convolutional network to obtain visually guided key points;

[0016] Based on text attribute features and key point representation, a cyclic heterogeneous graph convolutional network is used to capture cross-modal interaction information, obtaining attribute-enhanced key points and visually enhanced attributes;

[0017] The visually guided keypoints, attribute-enhanced keypoints and visually enhanced attributes are weightedly fused to obtain the final pedestrian representation.

[0018] A second aspect of the present invention provides a person re-identification system under incomplete observation conditions based on graph convolution, comprising:

[0019] The preprocessing module is configured to: acquire pedestrian images and preprocess the acquired images;

[0020] The feature extraction module is configured to perform local feature extraction, global feature extraction, and attribute feature extraction on the preprocessed pedestrian image, respectively, to obtain key points, global visual features, and text attribute features of the pedestrian image;

[0021] A visually guided keypoint acquisition module is configured to: input global visual features and keypoint representations into an adaptive threshold-guided mask graph convolutional network to obtain visually guided keypoints;

[0022] The key point and text attribute feature interaction module is configured to: capture cross-modal interaction information based on text attribute features and key point representations using a recurrent heterogeneous graph convolutional network to obtain attribute-enhanced key points and visually enhanced attributes;

[0023] The output module is configured to perform weighted fusion of the visually guided keypoints, attribute-enhanced keypoints, and visually enhanced attributes to obtain the final pedestrian representation.

[0024] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a pedestrian re-identification method under incomplete observation conditions based on graph convolution as described in the first aspect of the present invention.

[0025] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, the steps of a method for pedestrian re-identification under incomplete observation conditions based on graph convolution as described in the first aspect of the present invention are implemented.

[0026] One or more of the above technical solutions have the following beneficial effects:

[0027] (1) To address the problem of multi-scene person re-identification, this paper proposes a pedestrian re-identification method under incomplete observation conditions based on graph convolution, which integrates noisy keypoint constraints, structured multimodal representation aggregation and robust pedestrian representation learning into a unified framework.

[0028] (2) To address problems such as image occlusion, a cyclic heterogeneous graph convolutional network is used to effectively fuse structured visual and textual pedestrian information through intra-graph and inter-graph interactions, thereby enhancing the robustness and recognition ability of pedestrian representation.

[0029] (3) To address the spatiotemporal differences in pedestrian images, we designed an adaptive threshold-guided mask graph convolutional network to suppress the influence of noise keypoints, and designed a partial similarity constraint to promote the representation of information keypoints. Compared with previous methods, this method has significantly improved the accuracy of pedestrian re-identification tasks, and its universality and noise resistance have been improved.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 This is a flow chart of the method of the first embodiment.

[0033] Figure 2 Flowchart of obtaining key points for visual guidance according to the first embodiment.

[0034] Figure 3 This is a data processing flowchart based on a cyclic heterogeneous graph convolutional network in the first embodiment.

[0035] Figure 4 This is the construction and propagation of graphs in the cyclic heterogeneous graph convolutional network of the first embodiment. DETAILED DESCRIPTION

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment discloses a pedestrian re-identification method under incomplete observation conditions based on graph convolution, including:

[0038] Step 1: Obtain pedestrian images and preprocess the obtained images;

[0039] Step 2: Perform local feature extraction, global feature extraction, and attribute feature extraction on the preprocessed pedestrian image to obtain key point representation, global visual features, and text attribute features of the pedestrian image;

[0040] Step 3: Input the global visual features and key point representation into the adaptive threshold-guided mask graph convolutional network to obtain the visually guided key points;

[0041] Step 4: Based on the text attribute features and key point representation, a recurrent heterogeneous graph convolutional network is used to capture cross-modal interaction information, obtaining attribute-enhanced key points and visually enhanced attributes;

[0042] Step 5: Perform weighted fusion of the visually guided key points, attribute-enhanced key points, and visually enhanced attributes to obtain the final pedestrian representation.

[0043] In step 1, the pedestrian image includes a local query image and a reference image;

[0044] First, a local query image x is given, such as a pedestrian half-length photo; the reference image set {y1,y2,…y n}, where n represents the number of images in the reference image set, and the images in the reference are full-body photos.

[0045] The purpose of this invention is to design a method for extracting query image x and reference image set {y1, y2, ... n}feature method, and then use the distance metric function (for example, calculating the cosine similarity of x and y to obtain the final similarity score) to calculate the similarity score; taking the query image x as an example, the feature extraction process is explained.

[0046] In step 2, feature extraction includes:

[0047] Step 201: Local visual feature extraction

[0048] Step 2011, use ResNet-50 as the backbone to extract the feature map of the pedestrian image x, and use Indicates, where C, H, and W represent channel, height, and width respectively.

[0049] Step 212: Use the key point extraction model HR-Net to obtain the key point heat map of the local query image x, denoted as Where M is the number of key points, H′ and W′ are the height and width of the heat map respectively.

[0050] Step 2013: In order to obtain the feature representation of each key point, first align the heat map with the corresponding feature map using bilinear interpolation to obtain

[0051] For each keypoint heatmap, such as Expand and copy the dimensions in sequence, that is

[0052] Finally, yes and Perform the Hadamard product and then sum it over the H and W dimensions to obtain the feature representation of the corresponding key points. The specific formula is as follows:

[0053]

[0054] Where ⊙ is the Hadamard product, is the feature representation of the i-th key point.

[0055] Step 202: Global visual feature extraction

[0056] Use Transformer to extract the global visual representation of x, and use To represent this, the input image is first converted into a pixel matrix. A two-dimensional convolutional layer is then applied to break the pixel matrix into smaller blocks, each representing a vector, similar to a text word embedding. These blocks are then fed into the Swin-Transformer framework, where a learnable variable "[CLS]" is used to generate global visual features for the image x.

[0057] Specifically, when an image is input into the Swin-Transformer framework, it is divided into small blocks and a "[CLS] block" is added. After passing through the Swin-Transformer, the feature code corresponding to [CLS] contains information from other blocks. Usually, the feature code corresponding to [CLS] is directly taken as the global feature code of the image.

[0058] Step 203: Text attribute feature extraction

[0059] Leverage a publicly available structured human attribute model to extract N attributes and corresponding confidence levels from a pedestrian image x. These attributes consist of key-value structured information, such as {upper body color: red}, {upper body clothing: long sleeves}, and {lower body clothing: pants}.

[0060] Among them, the human attribute structured model is the Pedestrian Detect Attribute model tool opened by Open-AI, which mainly extracts image attributes and corresponding confidence levels.

[0061] To obtain attribute embeddings, we use the special token "[CLS]" from the last layer of the pre-trained BERT model as the word embedding. The embedded feature encoding representation for the text information (i.e., the i-th attribute) is equivalent to adding [CLS] to the attribute text information. The resulting encoding for [CLS] is used as the feature encoding for this attribute.

[0062] Formally, for the i-th attribute, its embedding It can be defined as:

[0063]

[0064] The key Sum Represents the key embedding and value embedding of the i-th attribute respectively. i and val i is the key and value of the i-th attribute. Symbol f a denotes a fully connected layer, and avg(·) denotes the average function. It is worth noting that BERT uses underscores instead of the default space as a tokenization method for attribute text, which means that, for example, “upper_color” is tokenized into two separate words, “upper” and “color”.

[0065] Step 3: Adaptive Threshold-Guided Mask Graph Convolutional Network

[0066] It is proposed to enhance the representation of each keypoint by aggregating the information of neighboring keypoints. Specifically, the local keypoints are organized into a frame graph where ξ is a node composed of key points, The edge set corresponding to the framework connection relationship provided in the COCO (Common Objects in Context) dataset, such as shoulder-elbow, elbow-wrist, hip-knee, etc. In order to facilitate the propagation of information between adjacent local nodes, graph convolution operations such as GAT can be directly used. These operations show great potential in transmitting information.

[0067] MS COCO is the Microsoft COCO dataset, funded and annotated by Microsoft in 2014. Like the ImageNet competition, it is considered one of the most popular and authoritative competitions in the field of computer vision. The COCO dataset is a large and rich dataset for object detection, segmentation, and captioning.

[0068] In addition, a simple and effective masking mechanism is designed, which uses an adaptive threshold to mask low-confidence key points and restricts information propagation only between information nodes, thus avoiding the interference of noise information. Specifically, by utilizing the heat map Obtain the confidence of all key points by performing the maximum operation on H×W elements

[0069] Take the i-th key point Taking the heat map of as an example, the confidence of the i-th keypoint is determined as the maximum value among the H×W elements. Nodes (i.e., keypoints) with confidence values ​​below the threshold ω are considered untrustworthy, and therefore their outgoing edges are untrustworthy. These untrustworthy edges are removed to reduce the outward propagation of noise information, while retaining all incoming edges to introduce valid information from trusted nodes, as follows:

[0070]

[0071] Where, Indicates the confidence of the i-th node, out i Represents the output degree of the i-th node. Adaptive threshold Used to determine whether a node is trustworthy. By applying this threshold, we can obtain the updated node connection relationship after deleting untrustworthy nodes and exit edges.

[0072] In addition, due to the different importance of nodes in information dissemination, the confidence of each node is Reweight as follows:

[0073]

[0074] in, represents the credible key point representation of the i-th node. In order to obtain a comprehensive representation of each node, the global visual information e g Incorporated into the node representation. It can be expressed as:

[0075]

[0076] where f v represents the fully connected layer, is the global enhanced representation of the i-th node, and || represents the connection operation. By this method, the enhanced node can be obtained, which is recorded as New framework diagram

[0077] After the noise suppression step, continue to update the skeleton graph Specifically, the GAT network is used to propagate information between trusted nodes. The lth layer of the i-th node graph convolution can be expressed as:

[0078]

[0079] Among them, γ l and are parameters that need to be learned. Functions ψ and φ represent LeakyReLU and Exponential Linear Unit (ELU) respectively. Parameter K represents the number of heads in the multi-head attention mechanism. Represents the normalized attention weight of the K-th head of the j-th node in the l-th layer. The vector represents the updated feature of the i-th node, which contains global visual information and local neighborhood information. By using this method, the visually guided key point representation is obtained.

[0080] Step 4: Circular Heterogeneous Graph Convolutional Network

[0081] To address the difficulty and inconsistent granularity in effectively capturing information from both modalities, this paper proposes a recurrent heterogeneous graph convolutional network, which consists of three main steps:

[0082] Step 401: In-graph aggregation:

[0083] Based on text attribute features and key point representation, a hierarchical aggregation method is used to aggregate text attributes and key points to obtain semantic isomorphism graph and key point isomorphism graph respectively;

[0084] Keypoint isomorphism graph It is a three-layer structure. The leaf nodes in the first layer are composed of key points, including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left knee, right knee, left ankle, right ankle;

[0085] The macro nodes in the second layer include: head, left arm, torso, right arm, left leg and right leg;

[0086] The macro nodes in the third layer include: head, upper body, and lower body;

[0087] Property Graph It is a two-layer structure. The leaf nodes in the attribute graph are specific things, such as (top color: red); while the macro nodes are more comprehensive representations.

[0088] The leaf nodes in the first layer are composed of attribute nodes including: handbag, shoulder bag, backpack, lower body color, lower body clothing, upper body color, upper body clothing, hat, glasses;

[0089] The macro nodes in the second layer include: handbag, shoulder bag, backpack, upper body, lower body, hat, glasses;

[0090] The nodes in the outermost layer of the graph represent concepts at a finer level of granularity, while the nodes in the other layers are relatively more comprehensive. Therefore, leaf nodes are defined as micro nodes, and the other nodes are defined as macro nodes. In this embodiment, nodes in the graph that cannot be further subdivided are called leaf nodes (i.e., the nodes in the first layer mentioned above), and nodes that can be further subdivided are called other nodes.

[0091] This paper proposes a hierarchical aggregation method that combines the modal information of micro-nodes into macro-nodes. Specifically, micro-keypoints (such as shoulders, elbows, and wrists) are first aggregated according to their compositional relationship in the body element, thus creating macro-nodes in the second layer (such as arms). Subsequently, the nodes in the second layer (such as left and right legs) are further aggregated to obtain macro-nodes in the third layer (such as the lower body). The new node representation can be obtained using the following formula:

[0092]

[0093] Where, e o,3 represents the features generated by the oth macro node in the third layer, represents the set of nodes in the second layer connected to the oth node in the third layer. ,e j,1 represents the key points (i.e. micro nodes) of the first layer, e j,2 represents the jth macro node in the second layer; represents the set of first-layer nodes connected to the j-th node in the second layer.

[0094] Following the same steps, we can also derive the semantic graph constructed based on text attributes. The node representation of a Represents a set of attribute nodes (such as upper body, upper body clothing, upper body color), Represents the set of edges connecting these nodes (e.g., upper body - upper body clothing).

[0095] Finally, we get the isomorphic graph after information aggregation processing and Node representation of .

[0096] Step 402: Interaction between graphs:

[0097] In order to achieve cross-modal information interaction, a heterogeneous graph is first constructed. in It is composed of the third-layer nodes of the structural graph and the second-layer nodes (i.e. the innermost nodes) of the semantic graph. Represents the set of edges connecting these nodes. Using this heterogeneous graph GAT is applied to capture cross-modal interaction information and obtain enhanced macro-node representations.

[0098] Step 403: In-graph propagation:

[0099] In order to integrate text modality information into visual key points, the structure graph It is divided into two bipartite graphs and adopts a step-by-step propagation mechanism. First, GAT is used to propagate information from the third-layer macro nodes to the second-layer macro nodes. Then, the attribute-enhanced node representation of the second layer is converted into micro nodes. Compared to applying this propagation on the image, iterative subgraph convolution can better control the information flow, resulting in a more robust visual representation. Finally, the attribute-enhanced keypoint representation is obtained. Similarly, the gradual propagation mechanism can be used to obtain the visually enhanced attribute representation

[0100] like Figure 4 As shown in Figure 2, the dotted lines represent the splitting of two isomorphic graphs. Arrows indicate the direction of information flow, and different arrows indicate corresponding relationships. The figure only shows the information flow from attribute nodes to keypoint nodes, while the information flow from keypoint nodes to attribute nodes is exactly the opposite.

[0101] Step 5: Output module

[0102] In order to effectively fuse the key point features of visual enhancement and attribute enhancement, a gated neural network is used to balance the information from these two sources. Specifically, a dynamic gate mechanism is used to select the key point features of the global visual feature e g , the key point features of visual enhancement are The key point features of attribute enhancement are Its expression is:

[0103]

[0104] in, is a learnable parameter, and σ is a sigmoid function. The fused key point representation can be obtained by the following formula:

[0105]

[0106] where ⊙ represents element-wise multiplication, and Is the final representation of the i-th key point. Normalize them and use them as weights to combine the above key point features to obtain the final pedestrian representation.

[0107] The present invention uses the weighted sum of multiple loss functions as the total loss of the method, and the losses involved are as follows:

[0108] [1] Classification loss:

[0109] Considering the weak differentiability of a single attribute, we choose to use the entire attribute feature for optimization. To achieve this, we first calculate the global attribute feature a using the following formula g :

[0110]

[0111] in, represents the normalized confidence score of the i-th attribute generated by the attribute structured model, represents the visual enhancement attribute feature. Next, the cross entropy loss is used to calculate the global visual feature e of each identity. g , local visual features and global attribute feature a g The optimization formula is as follows:

[0112]

[0113] where p i represents the predicted probability of the corresponding feature generated by the classifier implemented by the fully connected layer, and q represents the true value label. M represents the predicted probability represented by M key points, p M+1 and p M+2 Represent the probabilities of global visual features and attribute features respectively.

[0114] [2] Contrastive loss:

[0115] In order to regularize the semantic consistency between global visual features and global attribute features of the same identity, the following contrastive loss is adopted:

[0116]

[0117] Where τ represents a margin hyperparameter, d(a,b)=a T b. Symbol and denote the global visual features of the t-th identity and the global attribute features of the b-th identity, respectively. It is important to note that the global visual representations of the same identity are treated similarly.

[0118] [3] Ternary loss:

[0119] Different identities are considered to be far apart. Therefore, the following ternary loss is introduced:

[0120]

[0121] where λ is a margin hyperparameter. The global visual features corresponding to the anchor sample, positive sample, and negative sample, respectively. For each anchor sample, the triplet selection is based on the global distance between samples. Specifically, the most dissimilar sample of the same identity and the most similar sample of different identities are selected to form a triplet. This triplet construction process ensures that the anchor sample is compared with samples from a different range, allowing the model to learn meaningful embeddings for discrimination; [] + If the result is positive, it is taken as normal; if it is negative, it is taken as 0.

[0122] [4] Similarity loss:

[0123] In order to further constrain the learning of local keypoint representations, a similarity constraint considering the symmetry of the human body is proposed, which is expressed as:

[0124]

[0125] Where, and Denote symmetrical keypoint features, and Ω represents the index set of symmetrical parts of the human body. The symbol δ represents a dynamic coefficient that controls the weight of features related to symmetrical parts of the human body. Its value is determined based on the confidence level of the corresponding symmetrical parts. Its main goal is to prioritize the use of reliable symmetrical features as masks while reducing the influence of noise information. Specifically, reliable symmetrical features are given a larger weight, while other components are given a smaller weight, as follows:

[0126]

[0127] in, and are the confidences of symmetric key points respectively. abs(·) represents the absolute value function.

[0128] Finally, the final objective function is:

[0129]

[0130] where μ1, μ2, and μ3 represent non-negative hyperparameters that control the importance of the corresponding regularization.

[0131] Example 2

[0132] This embodiment discloses a person re-identification system under incomplete observation conditions based on graph convolution, including:

[0133] The preprocessing module is configured to: acquire pedestrian images and preprocess the acquired images;

[0134] The feature extraction module is configured to perform local feature extraction, global feature extraction, and attribute feature extraction on the preprocessed pedestrian image, respectively, to obtain key points, global visual features, and text attribute features of the pedestrian image;

[0135] A visually guided keypoint acquisition module is configured to: input global visual features and keypoint representations into an adaptive threshold-guided mask graph convolutional network to obtain visually guided keypoints;

[0136] The key point and text attribute feature interaction module is configured to: capture cross-modal interaction information based on text attribute features and key point representations using a recurrent heterogeneous graph convolutional network to obtain attribute-enhanced key points and visually enhanced attributes;

[0137] The output module is configured to perform weighted fusion of the visually guided keypoints, attribute-enhanced keypoints, and visually enhanced attributes to obtain the final pedestrian representation.

[0138] Example 3

[0139] The purpose of this embodiment is to provide a computer-readable storage medium.

[0140] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for pedestrian re-identification under incomplete observation conditions based on graph convolution as described in Example 1 of the present disclosure.

[0141] Example 4

[0142] The purpose of this embodiment is to provide an electronic device.

[0143] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of a method for pedestrian re-identification under incomplete observation conditions based on graph convolution as described in Example 1 of the present disclosure are implemented.

[0144] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0145] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0146] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A person re-identification method under incomplete observation conditions based on graph convolution, characterized by: include: Acquire pedestrian images and preprocess the acquired images; Perform local feature extraction, global feature extraction, and attribute feature extraction on the preprocessed pedestrian image to obtain key point representation, global visual features, and text attribute features of the pedestrian image; The global visual features and key point representations are input into the adaptive threshold-guided mask graph convolutional network to obtain the visually guided key points. The specific steps are as follows: The key points are regarded as nodes, and the connection relationship between the key points is regarded as edges to form a framework graph; Calculate the confidence of all nodes, mask low-confidence nodes in the framework graph based on a masking mechanism guided by an adaptive threshold, filter out trusted nodes, limit information propagation only between trusted nodes, and update node connection relationships; The confidence of each node is re-weighted to obtain the trusted node representation. The global visual features and the trusted node representation are spliced ​​through the fully connected layer to obtain the visually enhanced node. An updated framework graph is obtained based on the visually enhanced nodes and the updated node connection relationships; information is propagated on the updated framework graph to obtain a visually guided key point representation; Based on text attribute features and key point representation, a cyclic heterogeneous graph convolutional network is used to capture cross-modal interaction information, obtain attribute-enhanced key points and visually enhanced attributes, specifically: Based on text attribute features and key point representation, the text attributes and key points are hierarchically aggregated to obtain semantic isomorphism graph and key point isomorphism graph respectively; Construct a heterogeneous graph based on the top nodes of the semantic isomorphic graph and the key point isomorphic graph; Based on heterogeneous graphs, we use graph convolutional networks to capture cross-modal interaction information and obtain key points of attribute enhancement and visual enhancement attributes, specifically: Adopting a step-by-step propagation mechanism, we first use a graph convolutional network to propagate attribute information from the third-layer macro nodes to the second-layer macro nodes. Then, we convert the attribute enhancement node representation of the second layer into micro nodes, and finally obtain the key points of attribute enhancement. The key point isomorphism graph has a three-layer structure. The key points in the first layer are aggregated according to their compositional relationships in the body, and macro nodes are created in the second layer. The macro nodes in the second layer are aggregated to obtain macro nodes in the third layer. The semantic isomorphism graph is a two-layer structure, and the macro nodes in the second layer are obtained by aggregating the leaf nodes in the first layer; The visually guided keypoints, attribute-enhanced keypoints and visually enhanced attributes are weightedly fused to obtain the final pedestrian representation.

2. The method for person re-identification under incomplete observation conditions based on graph convolution according to claim 1, characterized in that: The local features of the pre-processed pedestrian image are extracted to obtain the key point representation of the pedestrian image, including the following steps: Use ResNet-50 convolutional neural network to extract feature maps of pedestrian images; Use the key point extraction model HR-Net to extract the key point heat map of pedestrian images; The key point heat map and the corresponding feature map are processed using bilinear interpolation to obtain the corresponding key point representation.

3. The method for person re-identification under incomplete observation conditions based on graph convolution as claimed in claim 1, characterized in that: Performing global feature extraction on the preprocessed pedestrian image to obtain the global visual features of the pedestrian image includes the following steps: The pedestrian image is converted into a pixel matrix; a two-dimensional convolutional layer is used to divide the pixel matrix block into small blocks, where each small block represents a vector; each small block is input into the Swin-Transformer framework, and the learnable variable "[CLS]" is used to generate the global visual features of the pedestrian image.

4. The method for person re-identification under incomplete observation conditions based on graph convolution according to claim 1, characterized in that: Attribute feature extraction is performed on the preprocessed pedestrian image to obtain text attribute features of the pedestrian image, including the following steps: Utilize the human attribute structured model to extract attributes of pedestrian images and the confidence level of the corresponding attributes; Using the representation of the special token "[CLS]" from the last layer of the pre-trained BERT model as word embedding, formally, for the attributes, which are embedded It can be defined as: The key Sum Respectively represent Key embedding and value embedding of each attribute; and It is The key and value of each property; the symbol represents the fully connected layer, (·) represents the average function.

5. A person re-identification system under incomplete observation conditions based on graph convolution, characterized by: A method for person re-identification under incomplete observation conditions based on graph convolution as claimed in any one of claims 1 to 4 is adopted, comprising: The preprocessing module is configured to: acquire pedestrian images and preprocess the acquired images; The feature extraction module is configured to perform local feature extraction, global feature extraction, and attribute feature extraction on the preprocessed pedestrian image, respectively, to obtain key points, global visual features, and text attribute features of the pedestrian image; A visually guided keypoint acquisition module is configured to: input global visual features and keypoint representations into an adaptive threshold-guided mask graph convolutional network to obtain visually guided keypoints; The key point and text attribute feature interaction module is configured to: capture cross-modal interaction information based on text attribute features and key point representations using a recurrent heterogeneous graph convolutional network to obtain attribute-enhanced key points and visually enhanced attributes; The output module is configured to perform weighted fusion of the visually guided keypoints, attribute-enhanced keypoints, and visually enhanced attributes to obtain the final pedestrian representation.

6. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for pedestrian re-identification under incomplete observation conditions based on graph convolution are implemented.

7. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for pedestrian re-identification under incomplete observation conditions based on graph convolution are implemented as described in any one of claims 1 to 4.

Images